Door control device, vehicle, door system, and program

Abstract

To determine whether a person enters or not on the basis of imaged data by machine learning.SOLUTION: A door control device 10 comprises a control unit 11 for controlling opening and closing of a door 6 on the basis of imaged data. The control unit recognizes a user region including a user in a frame image included in the imaged data and extracts an overlapping portion between an entry determination region adjacent to the freely opening and closing door and the user region and, when an overlapping degree of the overlapping portion to the entry determination region is equal to or higher than a prescribed value, calculates depth difference between the entry determination area and the overlapping portion in the frame image and, when the depth difference is equal to or larger than a threshold, determines that a user has entered the entry determination region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a door control device for controlling the opening and closing of a door, a vehicle, a door system, and a program. [Background technology]

[0002] Patent Document 1 discloses a surveillance system that detects the distance to an object based on image data captured by a television camera. This technology compares image information obtained by the television camera capturing the object with background image information captured in advance to detect the presence or absence of an intruding object. If an intruding object is detected, the depth of field is shallowed to perform focusing, and each time focus is achieved, the distance to the intruding object is detected based on information about the focus position. This technology complements multiple focus positions with shallow depth of field to detect the outline and recognize the intruding object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 09-098412 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology described in Patent Document 1, it is not possible to identify whether an intruder is a person unless the entire intruder is within the camera's field of view, which means that it may be difficult to determine whether a person has intruded when only part of the person has entered the imaging area.

[0005] The present invention aims to provide a door control device, a vehicle, a door system, and a program that can determine whether or not a person has entered the vehicle based on image data using machine learning. [Means for solving the problem]

[0006] One aspect of the present invention is a door control device that includes a control unit that controls the opening and closing of a door based on image data, wherein the control unit recognizes a user area including a user in a frame image included in the image data, extracts an overlapping portion between the user area and an intrusion determination area adjacent to a door that can be opened and closed, calculates a depth difference between the intrusion determination area and the overlapping portion in the frame image when the overlapping degree of the overlapping portion with respect to the intrusion determination area is greater than or equal to a predetermined value, and determines that the user has entered the intrusion determination area when the depth difference is greater than or equal to a threshold value. [Effects of the Invention]

[0007] According to the present invention, machine learning can be used to determine whether or not a person has intruded based on imaging data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a vehicle according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an imaging range of a camera. [Figure 3] FIG. 10 is a diagram illustrating a method for recognizing a user based on image data captured by an internal camera. [Figure 4] 10 is a diagram showing an overlapping portion between a user area and an intrusion determination area based on image data captured by an internal camera. FIG. [Figure 5] FIG. 10 is a diagram illustrating a method for recognizing a user based on image data captured by an external camera. [Figure 6] 10 is a diagram showing an overlapping portion between a user area and an intrusion determination area based on image data captured by an external camera. FIG. [Figure 7] 3 is a flowchart showing a process flow of a door control method executed in the door control device. DETAILED DESCRIPTION OF THE INVENTION

[0009] As shown in Figures 1 and 2, vehicle 1 is equipped with a door device 4 that can be opened and closed automatically. Vehicle 1 is equipped with a door control device 10 that controls the door device 4. Vehicle 1 is equipped with a detection unit 2 that detects a detection value for controlling the door device 4. Vehicle 1 is, for example, a bus vehicle that transports passengers. Vehicle 1 may be an autonomous vehicle or a manually operated vehicle. The detection unit 2, door device 4, and door control device 10 constitute a door system.

[0010] The detection unit 2 is composed of a camera that captures images of the interior and exterior of the vehicle and generates image data. The detection unit 2 outputs the image data to the door control device 10. The detection unit 2 is equipped with, for example, an internal camera 2A that captures images of the interior of the vehicle. The internal camera 2A captures images of, for example, the interior of the vehicle 1. The internal camera 2A is provided in a position within the vehicle 1 that is capable of capturing images of the intrusion determination area T1 adjacent to the door device 4. The internal camera 2A is provided, for example, in a position within the vehicle 1 that is higher than the user and that overlooks the intrusion determination area T1 adjacent to the door 6.

[0011] The detection unit 2 includes, for example, an external camera 2B that captures images outside the vehicle 1. The external camera 2B captures images outside the vehicle 1. The external camera 2B is provided at a position outside the vehicle 1 that allows it to capture an image of the intrusion determination area T2 adjacent to the door device 4. The external camera 2B is provided at a position outside the vehicle 1 that is higher than the user and that overlooks the intrusion determination area T2 adjacent to the door 6, for example.

[0012] The door device 4 includes, for example, a door 6 that can be opened and closed freely. The door 6 opens and closes based on, for example, a slide-type opening and closing mechanism. The door 6 may also open and close based on a folding door-type opening and closing mechanism. The door 6 is driven to open and close by a drive unit 5. The drive unit 5 is an actuator having a mechanism for opening and closing the door 6. The drive unit 5 is controlled by a door control device 10.

[0013] The door control device 10 includes a control unit 11 that controls the opening and closing of the door 6 based on the detection value detected by the detection unit 2. The control unit 11 is configured with at least one hardware processor such as a CPU (Central Processing Unit). The door control device 10 includes a storage unit 12 that stores data and programs. The storage unit 12 is configured with a non-transitory storage medium such as a hard disk drive (HDD) or a solid state disk (SSD). The storage unit 12 stores computer programs and data required for controlling the opening and closing of the door 6.

[0014] The control unit 11 is configured to be able to recognize the content of the image data by, for example, previously executing machine learning such as deep learning using the image data as training data. The control unit 11 is configured to be able to recognize the environment near the door 6 and a user present near the door 6 based on the image data captured by the detection unit 2. The control unit 11 is configured to control the opening and closing of the door 6 when it recognizes that a user is approaching the door 6.

[0015] 3 shows a frame image F1 included in the imaging data captured by the internal camera 2A. The imaging data is video data having a plurality of frame images within a predetermined unit time based on a predetermined frame rate. The frame image F1 includes a predetermined imaging range of the interior of the vehicle 1. For example, the frame image F1 captures an image of the door 6 and the passenger's riding space including the door 6.

[0016] The control unit 11 performs image analysis on the frame image F1 and recognizes a predetermined area including the door 6. When the control unit 11 recognizes the door 6, it sets an intrusion determination area T1 in an area adjacent to the door 6. The intrusion determination area T1 is a virtually set area for determining the intrusion of a user P. The intrusion determination area T1 is a rectangular area having a predetermined area, and may include a part of the area of ​​the door 6 or the entire area of ​​the door 6. The intrusion determination area T1 may be changed as appropriate depending on the imaging range based on the installation position of the internal camera 2A. The shape and range of the intrusion determination area T1 may be changed as appropriate as long as it is possible to determine the intrusion of a user P.

[0017] The control unit 11 recognizes the frame image F1 in a state where user P is not present as a reference frame. The control unit 11 stores an image including at least the intrusion determination area T1 in a state where user P is not present as a reference frame in the memory unit 12. The control unit 11 recognizes user P present in the frame image F1 based on the comparison result between the reference frame and the frame image F1. When the control unit 11 recognizes user P, it sets a user area G1 including user P. The user area G1 is a rectangular area that virtually surrounds user P. The user area G1 is used in calculations to determine whether user P has entered the intrusion determination area T1. The user area G1 may be set to another shape as long as it is possible to determine user P.

[0018] As shown in FIG. 4, when the control unit 11 recognizes a user area G1 including a user in a frame image F1 included in the imaging data, it extracts an overlapping portion H1 between the intrusion determination area T1 and the user area G1. The control unit 11 calculates the overlapping degree of the rectangular overlapping portion H1 where the rectangular user area G1 including the user P overlaps with the rectangular intrusion determination area T1. The overlapping degree is calculated, for example, from the area ratio between the intrusion determination area T1 and the overlapping portion H1. The control unit 11 compares the calculated overlapping degree with a threshold. If the overlapping degree is equal to or greater than a predetermined value, the control unit 11 calculates the depth of the overlapping portion H1 in the frame image F1.

[0019] Depth is the distance from internal camera 2A to the object. Even if the overlapping degree is equal to or greater than a predetermined value, it is not necessarily the case that a part of the body of user P is included in overlapping portion H1. Therefore, when the overlapping degree is equal to or greater than a predetermined value, control unit 11 calculates the depth of overlapping portion H1 to determine whether a part of the body of user P is present in overlapping portion H1.

[0020] The control unit 11, for example, analyzes the frame image F1 and estimates the depth of an object captured in the image. The control unit 11 uses the frame image F1, in which the intrusion determination area T1 is captured without any overlapping area, as a reference frame. The control unit 11 calculates a depth difference based on the comparison result between the intrusion determination area T1 in the reference frame and the overlapping area H1. The control unit 11 performs depth estimation of the overlapping area H1 using, for example, a general monocular depth estimation method based on the RGB values ​​of the pixels in the image.

[0021] Any depth estimation method may be used as long as it can be calculated based on an image. Depth estimation may be calculated based on not only monocular depth estimation but also panoramic images. Depth estimation may be calculated based not only on captured images but also on measurement data measuring the distance to an object.

[0022] The control unit 11 estimates the depth of an object contained in the image, for example, based on the RGB values ​​of pixels in the frame image F1, determining that the larger the R value of the pixel, the closer the object is, and determining that the smaller the R value, the farther the object is. The control unit 11 estimates the depth of the intrusion determination area T1 in the reference frame, for example. The control unit 11 estimates the depth of the overlapping portion H1 in the frame image F1. The control unit 11 compares the depth of the overlapping portion H1 with the depth of the intrusion determination area T1 in the reference frame, and if the depth difference between the depth of the overlapping portion H1 and the depth of the intrusion determination area T1 in the reference frame is equal to or greater than a threshold, determines that the user P has intruded into the intrusion determination area T1.

[0023] By comparing the depth of the overlapping portion H1 with the depth of the intrusion determination area T1 in the reference frame based on the frame image F1, the accuracy of determining the intrusion of user P can be improved compared to a method of simply determining the intrusion of user P based on the degree of overlap between the intrusion determination area T1 and the user area G1.

[0024] When the control unit 11 determines that the user P has entered the intrusion determination area T1, it controls the drive unit 5 that drives the door 6 to restrict the opening and closing operation of the door 6. For example, when the control unit 11 is performing a closing operation of the door 6 and determines that the user P has entered the intrusion determination area T1, the control unit 11 slows down or stops the closing operation of the door 6 and changes it to an opening operation, thereby opening the door 6. For example, when the control unit 11 is performing an opening operation of the door 6 and determines that the user P has entered the intrusion determination area T1, the control unit 11 slows down or stops the opening operation of the door 6. The control method of the door device 4 described above may be applied to a user P who is outside the vehicle 1.

[0025] 5 shows a frame image F2 included in the image data captured by the external camera 2B. In the following explanation, redundant explanations of configurations with the same names as those in the control method based on the internal camera 2A will be omitted as appropriate. In the frame image F2, for example, a door 6 and the external space including the door 6 are captured.

[0026] The control unit 11 performs image analysis on the frame image F2 and recognizes a predetermined area including the door 6. When the control unit 11 recognizes the door 6, it sets an intrusion determination area T2 in an area adjacent to the door 6. The intrusion determination area T2 may be changed as appropriate depending on the imaging range based on the mounting position of the external camera 2B. The shape and range of the intrusion determination area T2 may be changed as appropriate as long as it is possible to determine the intrusion of the user P.

[0027] The control unit 11 recognizes the frame image F2 in which the user P is not present as the reference frame. The control unit 11 recognizes the user P present in the frame image F2 based on the comparison result between the reference frame and the frame image F2. When the control unit 11 recognizes the user P, it sets a user area G2 that includes the user P. The user area G2 may be set to another shape as long as the user P can be determined.

[0028] As shown in FIG. 6, when the control unit 11 recognizes a user area G2 including a user in a frame image F2 included in the imaging data, it extracts an overlapping portion H2 between the intrusion determination area T2 and the user area G2. The control unit 11 calculates the overlapping degree of the rectangular overlapping portion H2 where the rectangular user area G2 including the user P overlaps with the rectangular intrusion determination area T2. The overlapping degree is calculated, for example, from the area ratio between the intrusion determination area T2 and the overlapping portion H2. The control unit 11 compares the calculated overlapping degree with a threshold. If the overlapping degree is equal to or greater than a predetermined value, the control unit 11 calculates the depth of the overlapping portion H2 in the frame image F2.

[0029] The control unit 11, for example, analyzes the frame image F2 and estimates the depth of the object captured in the image. The control unit 11 uses the frame image F2, in which the intrusion determination area T2 is captured without any overlapping area, as a reference frame. The control unit 11 calculates the depth difference based on the comparison result between the intrusion determination area T2 in the reference frame and the overlapping area H2.

[0030] The control unit 11 estimates the depth of an object contained in the image, for example, based on the RGB values ​​of pixels in frame image F2, determining that the larger the R value of the pixel, the closer the object is, and determining that the smaller the R value, the farther the object is. The control unit 11 estimates the depth of the intrusion determination area T2 in the reference frame, for example. The control unit 11 estimates the depth of the overlapping portion H2 in frame image F2. The control unit 11 compares the depth of the overlapping portion H2 with the depth of the intrusion determination area T2 in the reference frame, and if the depth difference between the depth of the overlapping portion H2 and the depth of the intrusion determination area T2 in the reference frame is equal to or greater than a threshold, determines that the user P has intruded into the intrusion determination area T2.

[0031] When the control unit 11 determines that the user P has entered the intrusion determination area T2, it controls the drive unit 5 that drives the door 6 to restrict the opening and closing operation of the door 6. For example, when the control unit 11 is performing a closing operation of the door 6 and determines that the user P has entered the intrusion determination area T2, the control unit 11 slows down or stops the closing operation of the door 6 and changes it to an opening operation, thereby opening the door 6. For example, when the control unit 11 is performing an opening operation of the door 6 and determines that the user P has entered the intrusion determination area T2, the control unit 11 slows down or stops the opening operation of the door 6.

[0032] 7 shows the processing flow of a door control method executed in the door control device 10. The door control method is executed based on a computer program installed in a computer mounted on the door control device 10. The computer program causes the door control device 10 to execute the following processes.

[0033] The control unit 11 recognizes the user area G1 including the user P and the intrusion determination area T1 from the frame image based on the image data acquired by the internal camera 2A (step S100). The control unit 11 extracts an overlapping portion between the intrusion determination area T1 and the user area G1 (step S102). The control unit 11 calculates the overlapping degree of the overlapping portion with respect to the intrusion determination area T1 and determines whether the overlapping degree is equal to or greater than a predetermined value (step S104).

[0034] If the degree of overlap is less than a predetermined value, the control unit 11 calculates the depth of the intrusion determination area T1 and saves a frame image including the intrusion determination area T1 as a reference frame (step S106). The control unit 11 determines that the user P has not entered the intrusion determination area T1 (step S108). The control unit 11 executes control of the door 6 without restriction (step S110). If the degree of overlap is greater than or equal to a predetermined value in step S104, the control unit 11 calculates the depth difference between the depth of the overlapping portion H1 and the depth of the intrusion determination area T1 in the reference frame (step S112).

[0035] The control unit 11 determines whether the depth difference is equal to or greater than a threshold value (step S114). If the depth difference is equal to or greater than the threshold value, the control unit 11 determines that the user P has entered the intrusion determination area T1 (step S116). The control unit 11 restricts and executes the opening / closing control of the door 6 (step S118). The control unit 11 executes each of the above processes for the number of recognized users P. If the control unit 11 determines that at least one user P has entered the intrusion determination area T1, it restricts the opening / closing control of the door 6. Each of the above processes may also be executed for a user P who is outside the vehicle 1, based on image data captured by the external camera 2B.

[0036] As described above, the door control device 10 can use machine learning to determine whether or not a user P has entered the intrusion determination areas T1, T2 adjacent to the door 6 based on image data. The door control device 10 can improve the accuracy of determining whether or not a user P has entered the intrusion determination areas T1, T2 by calculating not only the degree of overlap of the overlapping areas H1, H2 between the user areas G1, G2 and the intrusion determination areas T1, T2 but also the depth difference between the overlapping areas H1, H2 and the intrusion determination areas T1, T2. The door control device 10 can improve safety for the user P in controlling the opening and closing of the door 6.

[0037] In the above-described embodiment, the computer program executed by each component of the door control device 10 may be provided in a form recorded on a portable computer-readable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. The present invention is not limited to the above-described embodiment and may be modified as appropriate without departing from the spirit of the present invention. For example, the door control device 10 may be applied to control doors of not only the vehicle 1 but also other vehicles. [Explanation of symbols]

[0038] 1 vehicle 2. Detection unit 2A Internal Camera 2B External Camera 4 door device 5 Drive unit 6 doors 7. Radar equipment 10 Door control device 11 Control section 12 Storage section F1, F2 frame images G1, G2 user area H1, H2 overlap part P User T1, T2 Intrusion detection area

Claims

1. A control unit is provided that controls opening and closing of the door based on the imaging data, The control unit Recognizing a user area including a user in a frame image included in the imaging data; extracting an overlapping area between an intrusion determination area adjacent to an openable door and the user area; When the overlapping degree of the overlapping portion with respect to the intrusion determination area is equal to or greater than a predetermined value, a depth difference between the intrusion determination area and the overlapping portion in the frame image is calculated; If the depth difference is equal to or greater than a threshold, it is determined that the user has entered the intrusion determination area. Door control device.

2. The control unit When it is determined that the user has entered the intrusion determination area, a drive unit that drives the door to limit the opening and closing operation of the door; The door control device according to claim 1 .

3. The control unit The frame image in which the intrusion determination area is captured without the overlapping portion is used as a reference frame, calculating the depth difference based on a comparison result between the overlapping portion having the overlap degree equal to or greater than a predetermined value and the reference frame; The door control device according to claim 1 .

4. The control unit calculating the overlapping degree of the overlapping portion of the rectangle where the user region of the rectangle including the user and the intrusion determination region set in the rectangle overlap; The door control device according to claim 1 .

5. With openable doors and a camera that captures an image of an area adjacent to the door and generates image data; a door control device that controls opening and closing of the door based on the imaging data; The door control device includes: Recognizing a user area including a user in a frame image included in the imaging data; extracting an overlapping area between the intrusion determination area adjacent to the door and the user area; When the overlapping degree of the overlapping portion with respect to the intrusion determination area is equal to or greater than a predetermined value, a depth difference between the intrusion determination area and the overlapping portion in the frame image is calculated; If the depth difference is equal to or greater than a threshold, it is determined that the user has entered the intrusion determination area. vehicle.

6. With openable doors and a camera that captures an image of an area adjacent to the door and generates image data; a door control device that controls opening and closing of the door based on the imaging data; The door control device includes: Recognizing a user area including a user in a frame image included in the imaging data; extracting an overlapping area between the intrusion determination area adjacent to the door and the user area; When the overlapping degree of the overlapping portion with respect to the intrusion determination area is equal to or greater than a predetermined value, a depth difference between the intrusion determination area and the overlapping portion in the frame image is calculated; If the depth difference is equal to or greater than a threshold, it is determined that the user has entered the intrusion determination area. Door system.

7. A program installed in a computer mounted on a door control device that controls the opening and closing of a door based on imaging data, Recognizing a user area including a user in a frame image included in the imaging data; extracting an overlapping area between an intrusion determination area adjacent to an openable door and the user area; When the overlapping degree of the overlapping portion with respect to the intrusion determination area is equal to or greater than a predetermined value, a depth difference between the intrusion determination area and the overlapping portion in the frame image is calculated; causing the computer to execute a process of determining that the user has entered the intrusion determination area when the depth difference is equal to or greater than a threshold value; program.

Images