Door control device, vehicle, door system, and program product

Abstract

The invention provides a door control device, a vehicle, a door system, and a program product. The door control device includes an electronic control unit that controls opening and closing of a door based on imaging data. The electronic control unit is configured to recognize a user region including a user in a frame image included in the captured data, extract an overlap portion between an intrusion determination region adjacent to the openable and closable door and the user region, and determine whether the overlap portion has a degree of repetition with respect to the intrusion determination region is equal to or greater than a predetermined degree of repetition. The depth difference between the intrusion determination region and the overlapping portion in the frame image is calculated, and if the depth difference is equal to or greater than a threshold value, it is determined that the user has intruded into the intrusion determination region.

Description

Technical Field

[0001] This invention relates to door control devices, vehicles, door systems, and program products for controlling the opening and closing of doors. Background Technology

[0002] Japanese Patent Application Publication No. 09-098412 discloses a surveillance system for detecting the distance of objects based on image data captured by a television camera. According to this technology, image information obtained from a television camera capturing the object is compared with pre-captured background image information to detect the presence or absence of an intruder. If an intruder is detected, the depth of field is reduced to focus. Each time focus is achieved, the distance of the intruder is detected based on the focus position information. This technology further supplements multiple focus positions with a shallow depth of field to detect contours and identify intruders.

[0003] According to the technology described in Japanese Patent Application Publication No. 09-098412, since it is impossible to determine whether the intruder is a person if the entire intruder does not enter the camera's field of view, it may be difficult to determine whether a person has intruded even if only part of a person has entered the shooting area. Summary of the Invention

[0004] This invention provides door control devices, vehicles, door systems, and software products that can use machine learning to determine the presence or absence of human intrusion based on captured data.

[0005] The first aspect of the present invention relates to a door control device equipped with an electronic control unit (ECU) for controlling the opening and closing of a door based on captured data. The ECU is configured to identify a user area including the user in a frame image included in the captured data. The ECU is configured to extract a repetitive portion of an intrusion detection area adjacent to the freely opening and closing door and the user area. The ECU is configured to calculate a depth difference between the intrusion detection area and the repetitive portion in the frame image if the repetition degree of the repetitive portion relative to the intrusion detection area is a predetermined value or higher. Furthermore, the ECU is configured to determine that the user has intruded into the intrusion detection area if the depth difference is a threshold value or higher.

[0006] In the door control device of the first aspect of the present invention, the electronic control unit may be configured to control the drive unit that drives the door to restrict the opening and closing operation of the door when it is determined that the user has intruded into the intrusion determination area.

[0007] In the door control device of the first aspect of the present invention, the electronic control unit may be configured to use the frame image of the intrusion determination area in which the state of the intrusion determination area where the repetition portion does not exist as a reference frame, and calculate the depth difference based on the comparison result between the repetition portion with the repetition degree of a certain or higher and the reference frame.

[0008] In the door control device of the first aspect of the present invention, the electronic control unit may be configured to calculate the degree of repetition of the user area, which includes the user, and the repetition of the repetition portion of the rectangle that is set as a rectangle and the intrusion determination area.

[0009] The second aspect of the present invention relates to a vehicle equipped with a door, a camera, and a door control device. The door is configured to open and close freely. The camera is configured to generate image data capturing an area adjacent to the door. The door control device is configured to control the opening and closing of the door based on the image data. Furthermore, the door control device is configured to identify a user area including the user in a frame image included in the image data, extract a duplicate portion of an intrusion determination area adjacent to the door and the user area, calculate a depth difference between the intrusion determination area and the duplicate portion in the frame image if the degree of duplication of the duplicate portion relative to the intrusion determination area is a predetermined value or higher, and determine that the user has intruded into the intrusion determination area if the depth difference is a threshold value or higher.

[0010] In the vehicle of the second aspect of the present invention, the aforementioned intrusion determination area can be appropriately changed according to the shooting range based on the installation position of the aforementioned camera.

[0011] A third aspect of the present invention relates to a door system comprising a door, a camera, and a door control device. The door is configured to open and close freely. The camera is configured to generate image data capturing an area adjacent to the door. The door control device is configured to control the opening and closing of the door based on the image data. The door control device is configured to identify a user area including the user in a frame image included in the image data, extract a duplicate portion of an intrusion determination area adjacent to the door and the user area, calculate a depth difference between the intrusion determination area and the duplicate portion in the frame image if the degree of duplication of the duplicate portion relative to the intrusion determination area is a predetermined value or higher, and determine that the user has intruded into the intrusion determination area if the depth difference is a threshold value or higher.

[0012] A fourth aspect of the present invention relates to a program product installed on a computer mounted on a door control device that controls the opening and closing of a door based on captured data. The program product causes the computer to perform the following processing: (i) identifying a user region including the user in frame images included in the captured data; (ii) extracting a duplicate portion of an intrusion determination region adjacent to the freely opening and closing door and the user region; (iii) calculating a depth difference between the intrusion determination region and the duplicate portion in the frame image if the degree of duplication of the duplicate portion relative to the intrusion determination region is a predetermined value or higher; and (iv) determining that the user has intruded into the intrusion determination region if the depth difference is a threshold value or higher.

[0013] According to the door control device, vehicle, door system, and program product of the present invention, it is possible to determine the presence or absence of human intrusion based on captured data through machine learning. Attached Figure Description

[0014] Hereinafter, the features, advantages, technical and industrial importance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, in which the same reference numerals denote the same constituent elements, wherein:

[0015] Figure 1 This is a block diagram illustrating the configuration of a vehicle according to an embodiment of the present invention.

[0016] Figure 2 Yes Figure 1 The diagram illustrates the shooting range of the internal and external cameras.

[0017] Figure 3 This diagram illustrates a method for identifying users based on data captured by the aforementioned internal camera.

[0018] Figure 4 This is a diagram showing the overlapping portion of the user area and the intrusion determination area based on the data captured by the aforementioned internal camera.

[0019] Figure 5 This diagram illustrates a method for identifying users based on data captured by the aforementioned external camera.

[0020] Figure 6 This is a diagram showing the overlapping portion of the user area and the intrusion determination area based on the data captured by the aforementioned external camera.

[0021] Figure 7 It means in Figure 1 The flowchart shows the processing flow of the door control method executed in the aforementioned door control device. Detailed Implementation

[0022] like Figure 1 as well as Figure 2 As shown, vehicle 1 is equipped with a door device 4 that can open and close automatically. Vehicle 1 is equipped with a door control device 10 that controls the door device 4. Vehicle 1 is equipped with a detection unit 2 that detects the detection value used to control the door device 4. Vehicle 1 is, for example, a bus that transports users. Vehicle 1 can be an autonomous vehicle or a manually driven vehicle. The door system is constituted by the detection unit 2, the door device 4, and the door control device 10.

[0023] The detection unit 2 consists of an internal camera 2A that generates images of the interior of the vehicle and an external camera 2B that generates images of the exterior of the vehicle. The detection unit 2 outputs the captured data to the door control device 10. The internal camera 2A is positioned inside the vehicle 1 at a location capable of capturing images of the intrusion detection area T1 adjacent to the door device 4. For example, the internal camera 2A is positioned inside the vehicle 1 at a height higher than the user and overlooking the intrusion detection area T1 adjacent to the door 6.

[0024] The detection unit 2, for example, includes an external camera 2B that captures images of the exterior of the vehicle 1. The external camera 2B is positioned outside the vehicle 1 at a location capable of capturing images of the intrusion detection area T2 adjacent to the door device 4. For example, the external camera 2B is positioned outside the vehicle 1 at a height higher than the user and overlooking the intrusion detection area T2 adjacent to the door 6.

[0025] The door device 4 includes, for example, a door 6 that can be opened and closed freely. The door 6 is opened and closed, for example, based on a sliding opening and closing mechanism. The door 6 can also be a door based on a folding door opening and closing mechanism. The door 6 is driven to open and close by a drive unit 5. The drive unit 5 is an actuator having a mechanism for opening and closing the door 6. The drive unit 5 is controlled by the door control device 10.

[0026] The door control device 10 includes an electronic control unit 11 that controls the opening and closing of the door 6 based on the detection value detected by the detection unit 2. The electronic control unit 11 is composed of at least one hardware processor such as a CPU (Central Processing Unit). The door control device 10 also includes a storage unit 12 for storing data and programs. The storage unit 12 is composed of a non-transitory storage medium such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage unit 12 stores the computer program and data required for controlling the opening and closing of the door 6.

[0027] The electronic control unit 11 is configured to perform machine learning, such as deep learning, which uses image data as teaching data, in advance, and is capable of recognizing the content of the image data. The electronic control unit 11 is configured to identify the environment near the door 6 and the user present near the door 6 based on the image data captured by the detection unit 2. The electronic control unit 11 is configured to control the opening and closing of the door 6 when it detects that a user is approaching the door 6.

[0028] Figure 3 The image shows frame F1, which is part of the image data captured by the internal camera 2A. The image data is dynamic image data consisting of multiple frames within a specified unit of time, based on a specified frame rate. Frame F1 includes a specified shooting range of the interior of the vehicle 1. For example, frame F1 may capture a door 6 and the passenger space for the user, including the door 6.

[0029] The electronic control unit 11 performs image analysis on frame image F1 and identifies a defined area including door 6. Upon identifying door 6, the electronic control unit 11 sets an intrusion determination area T1 in the area adjacent to door 6. The intrusion determination area T1 is a virtually defined area for determining intrusion by user P. The intrusion determination area T1 is a rectangular area with a defined area, which may include a portion of door 6 or the entire area of ​​door 6. The intrusion determination area T1 can be appropriately changed based on the shooting range of the internal camera 2A based on its installation position. The shape and range of the intrusion determination area T1 can be appropriately changed as long as it can determine intrusion by user P.

[0030] The electronic control unit 11 identifies a frame image F1 in a state where user P is absent as a reference frame. The electronic control unit 11 stores an image in the storage unit 12 that includes at least the intrusion determination region T1 in a state where user P is absent as a reference frame. Based on the comparison result between the reference frame and the frame image F1, the electronic control unit 11 identifies user P present within the frame image F1. When user P is identified, the electronic control unit 11 sets a user region G1 that includes user P. User region G1 is a rectangular region that virtually surrounds user P. User region G1 is used in the calculation for determining whether user P has intruded into the intrusion determination region T1. User region G1 can be set to any shape as long as user P can be identified.

[0031] like Figure 4As shown, when a user region G1 including the user is identified in the frame image F1 included in the captured data, the electronic control unit 11 extracts the overlapping portion H1 of the intrusion detection region T1 and the user region G1. The electronic control unit 11 calculates the repetition degree of the overlapping portion H1 of the rectangle of the user region G1 including the user P and the rectangle that overlaps with the rectangle of the intrusion detection region T1. For example, the repetition degree is calculated based on the area ratio of the intrusion detection region T1 to the overlapping portion H1. The electronic control unit 11 compares the calculated repetition degree with a threshold. If the repetition degree is above a certain threshold, the electronic control unit 11 calculates the depth of the overlapping portion H1 in the frame image F1.

[0032] Depth is the distance from the internal camera 2A to the object. Even when the repeatability is above a certain limit, the repeated portion H1 may not necessarily include a part of the user P's body. Therefore, when the repeatability is above a certain limit, the electronic control unit 11 determines whether a part of the user P's body exists in the repeated portion H1 by calculating the depth of the repeated portion H1.

[0033] The electronic control unit 11, for example, analyzes frame image F1 to infer the depth of an object captured within the image. The electronic control unit 11 uses frame image F1, which captures an intrusion detection region T1 in a state where no repeating portion exists, as a reference frame. The electronic control unit 11 calculates the depth difference based on a comparison between the intrusion detection region T1 and the repeating portion H1 in the reference frame. The electronic control unit 11, for example, uses a general monocular depth inference method based on the RGB values ​​of the image pixels to perform depth inference of the repeating portion H1.

[0034] Depth inference methods can be any approach that calculates depth based on an image. It can be monocular depth inference or depth calculation based on panoramic images. Furthermore, depth inference can be calculated not only based on captured images but also on measurements of distances to objects.

[0035] The electronic control unit 11, for example, infers the depth of objects within the image based on the RGB values ​​of pixels in the frame image F1, where a larger R value indicates a closer object and a smaller R value indicates a farther object. The electronic control unit 11, for example, infers the depth of the intrusion detection region T1 in the reference frame. The electronic control unit 11 infers the depth of the repeating portion H1 in the frame image F1. The electronic control unit 11 compares the depth of the repeating portion H1 with the depth of the intrusion detection region T1 in the reference frame, and if the difference between the depth of the repeating portion H1 and the depth of the intrusion detection region T1 in the reference frame is greater than or equal to a threshold, it determines that the user P has intruded into the intrusion detection region T1.

[0036] By comparing the depth of the repeated portion H1 based on frame image F1 with the depth of the intrusion detection region T1 in the reference frame, the accuracy of intrusion detection of user P can be improved compared with the method of simply judging the intrusion of user P based on the repetition of intrusion detection region T1 and user region G1.

[0037] If it is determined that user P has intruded into the intrusion determination area T1, the electronic control unit 11 controls the drive unit 5 of the drive door 6 to limit the opening and closing action of the door 6. For example, if it is determined that user P has intruded into the intrusion determination area T1 while the door 6 is being closed, the electronic control unit 11 slows down or stops the closing action of the door 6 and changes it to an opening action to open the door 6. For example, if it is determined that user P has intruded into the intrusion determination area T1 while the door 6 is being opened, the electronic control unit 11 slows down or stops the opening action of the door 6. The above-described control method of the door device 4 can be applied to a user P who is located outside the vehicle 1.

[0038] Figure 5 The image shown is a frame image F2 containing the captured data from the external camera 2B. In the following description, repeated descriptions of configurations with the same names as the control method based on the internal camera 2A are appropriately omitted. Frame image F2, for example, captures a door 6 and the external space including the door 6.

[0039] The electronic control unit 11 performs image analysis on frame image F2 to identify a defined area including door 6. Upon identification of door 6, the electronic control unit 11 sets an intrusion detection area T2 in the region adjacent to door 6. The intrusion detection area T2 can be appropriately changed based on the shooting range of the external camera 2B's installation position. The intrusion detection area T2 only needs to be able to determine the intrusion of user P, and its shape and range can be appropriately changed.

[0040] The electronic control unit 11 identifies frame image F2, where user P is absent, as a reference frame. Based on a comparison between the reference frame and frame image F2, the electronic control unit 11 identifies user P within frame image F2. If user P is identified, the electronic control unit 11 defines a user region G2 that includes user P. The user region G2 can be configured to include user P, but can also be configured to have other shapes.

[0041] like Figure 6As shown, when a user region G2 including the user is identified in the frame image F2 included in the captured data, the electronic control unit 11 extracts the overlapping portion H2 of the intrusion detection region T2 and the user region G2. The electronic control unit 11 calculates the repetition degree of the overlapping portion H2 of the rectangle of the user region G2 including the user P and the rectangle that overlaps with the rectangle of the intrusion detection region T2. ​​For example, the repetition degree is calculated based on the area ratio of the intrusion detection region T2 to the overlapping portion H2. The electronic control unit 11 compares the calculated repetition degree with a threshold. If the repetition degree is above a certain threshold, the electronic control unit 11 calculates the depth of the overlapping portion H2 in the frame image F2.

[0042] The electronic control unit 11, for example, analyzes frame image F2 to infer the depth of an object captured within the image. The electronic control unit 11 uses frame image F2, which captures an intrusion detection region T2 without any repeating portions, as a reference frame. The electronic control unit 11 calculates the depth difference based on a comparison between the intrusion detection region T2 and the repeating portion H2 in the reference frame.

[0043] The electronic control unit 11, for example, infers the depth of objects within the image based on the RGB values ​​of pixels in frame image F2, where a larger R value indicates a closer object and a smaller R value indicates a farther object. The electronic control unit 11, for example, infers the depth of the intrusion detection region T2 in the reference frame. The electronic control unit 11 also infers the depth of the repeating portion H2 in frame image F2. The electronic control unit 11 compares the depth of the repeating portion H2 with the depth of the intrusion detection region T2 in the reference frame. If the depth difference between the repeating portion H2 and the intrusion detection region T2 in the reference frame is greater than or equal to a threshold, it determines that user P has intruded into the intrusion detection region T2.

[0044] If it is determined that user P has intruded into the intrusion determination area T2, the electronic control unit 11 controls the drive unit 5 of the drive door 6 to restrict the opening and closing action of the door 6. For example, if it is determined that user P has intruded into the intrusion determination area T2 while the door 6 is being closed, the electronic control unit 11 slows down or stops the closing action of the door 6 and changes it to an opening action to open the door 6. For example, if it is determined that user P has intruded into the intrusion determination area T2 while the door 6 is being opened, the electronic control unit 11 slows down or stops the opening action of the door 6.

[0045] Figure 7 The diagram illustrates the processing flow of the door control method executed in the door control device 10. The door control method is executed based on a computer program installed in the computer mounted on the door control device 10. The computer program causes the door control device 10 to perform the following processes.

[0046] The electronic control unit 11 identifies the user region G1, including the user P, and the intrusion detection region T1 from the frame image based on the shooting data acquired by the internal camera 2A (step S100). The electronic control unit 11 extracts the overlapping portion between the intrusion detection region T1 and the user region G1 (step S102). The electronic control unit 11 calculates the repetition degree of the overlapping portion relative to the intrusion detection region T1 and determines whether the repetition degree is above a certain limit (step S104).

[0047] If the repeatability is less than a specified value, the electronic control unit 11 calculates the depth of the intrusion determination region T1 and saves the frame image including the intrusion determination region T1 as a reference frame (step S106). The electronic control unit 11 determines that the user P has not intruded into the intrusion determination region T1 (step S108). The electronic control unit 11 executes the control of the door 6 without restriction (step S110). If the repeatability is greater than or equal to a specified value in step S104, the electronic control unit 11 calculates the depth difference between the depth of the repeated portion H1 and the depth of the intrusion determination region T1 in the reference frame (step S112).

[0048] The electronic control unit 11 determines whether the depth difference is above a threshold (step S114). If the depth difference is above the threshold, the electronic control unit 11 determines that user P has intruded into the intrusion determination area T1 (step S116). The electronic control unit 11 then restricts the opening and closing control of door 6 (step S118). The electronic control unit 11 performs the above processes according to the number of users P identified. If it is determined that at least one user P has intruded into the intrusion determination area T1, the electronic control unit 11 restricts the opening and closing control of door 6. Based on the shooting data captured by the external camera 2B, the above processes are also performed on users P existing outside the vehicle 1.

[0049] As described above, according to the door control device 10, it is possible to determine whether user P has intruded into the intrusion detection areas T1 and T2 adjacent to door 6 based on captured data through machine learning. According to the door control device 10, by calculating not only the repetition degree of the overlapping portions H1 and H2 between user areas G1 and G2 and intrusion detection areas T1 and T2, but also the depth difference between the overlapping portions H1 and H2 and intrusion detection areas T1 and T2, the accuracy of intrusion detection of user P into intrusion detection areas T1 and T2 can be improved. According to the door control device 10, the security for user P in the opening and closing control of door 6 can be improved.

[0050] In the above embodiments, the computer programs executed in each component of the door control device 10 are provided in the form of a computer-readable, portable recording medium such as a semiconductor memory, magnetic recording medium, or optical recording medium. The present invention is not limited to the one embodiment described above and can be appropriately modified without departing from its spirit. For example, the door control device 10 can be applied not only to vehicle 1 but also to the control of doors other than those in vehicle 1.

Claims

1. A door control device, characterized in that, This includes electronic control units that control the opening and closing of doors based on captured data. in, The electronic control unit is configured to identify a user region, including the user, in the frame images included in the captured data. The electronic control unit is configured to extract overlapping portions of the intrusion detection area adjacent to the freely opening and closing door and the user area. The electronic control unit is configured to calculate the depth difference between the intrusion detection region and the repeated portion in the frame image when the repetition degree of the repeated portion relative to the intrusion detection region is greater than a certain limit. The electronic control unit is configured to determine that the user has intruded into the intrusion determination area when the depth difference is above a threshold.

2. The door control device according to claim 1, characterized in that, The electronic control unit is configured to control the drive unit that drives the door to restrict the opening and closing of the door when it is determined that the user has intruded into the intrusion determination area.

3. The door control device according to claim 1, characterized in that, The electronic control unit is configured to use a frame image of the intrusion determination area where the repeated portion is absent as a reference frame, and calculate the depth difference based on a comparison between the repeated portion with a repeatability of a specified degree and the reference frame.

4. The door control device according to claim 1, characterized in that, The electronic control unit is configured to calculate the degree of repetition of the user region, which includes the user, and the overlapping portion of the rectangle that is set as a rectangle and overlaps with the intrusion determination region.

5. A vehicle, characterized in that, include: The door is designed to open and close freely; A camera configured to generate image data showing that it has captured images of the area adjacent to the door; and The door control device is configured to control the opening and closing of the door based on the captured data. in, The gate control device is configured to identify a user region, including the user, in the frame images included in the captured data. The door control device is configured to extract the overlapping portion of the intrusion detection area adjacent to the door and the user area. The gate control device is configured to calculate the depth difference between the intrusion determination region and the repeated portion in the frame image when the repetition degree of the repeated portion relative to the intrusion determination region is greater than a certain limit. The door control device is configured to determine that the user has intruded into the intrusion determination area when the depth difference is above a threshold.

6. The vehicle according to claim 5, characterized in that, The intrusion detection area can be appropriately changed based on the shooting range based on the camera's installation location.

7. A gate system, characterized in that, include: The door is designed to open and close freely; A camera configured to generate image data showing that it has captured images of the area adjacent to the door; and The door control device is configured to control the opening and closing of the door based on the captured data. in, The gate control device is configured to identify a user region, including the user, in the frame images included in the captured data. The door control device is configured to extract the overlapping portion of the intrusion detection area adjacent to the door and the user area. The gate control device is configured to calculate the depth difference between the intrusion determination region and the repeated portion in the frame image when the repetition degree of the repeated portion relative to the intrusion determination region is greater than a certain limit. The door control device is configured to determine that the user has intruded into the intrusion determination area when the depth difference is above a threshold.

8. A program product installed on a computer mounted on a door control device, the door control device controlling the opening and closing of a door based on captured data, characterized in that... The computer is instructed to perform the following processing: Identify the user region, including the user, in the frame images included in the captured data; Extract the overlapping portions of the intrusion detection area and the user area adjacent to the freely opening and closing door; If the degree of repetition of the repeated portion relative to the intrusion determination region is greater than a certain limit, the depth difference between the intrusion determination region and the repeated portion in the frame image is calculated. as well as If the depth difference is above a threshold, it is determined that the user has intruded into the intrusion determination area.

Images