Door control device, vehicle control system, door control method, and door control program

Abstract

The present invention provides a door control device, a vehicle control system, a door control method, and a door control program that can appropriately control the opening and closing of vehicle doors. [Solution] The door control device 100 includes a control unit 190 that controls the opening and closing of a door 72 provided in the opening based on photographic data of the external area of ​​the vehicle 300. The control unit 190 determines whether or not a moving body intends to board the vehicle 300 based on the time-series data of the photographic data, based on the positional relationship between the direction of movement of the moving body and the opening. The control unit 190 changes the opening and closing timing of the door 72 based on the determination result of whether or not the moving body intends to board the vehicle 300.

Description

Technical Field

[0001] The present disclosure relates to a vehicle door control device, a vehicle control system, a vehicle door control method, and a door control program.

Background Art

[0002] As described in Patent Document 1, an elevator boarding detection system is known that estimates the presence or absence of a user's boarding intention based on the time-series change in the position of the elevator user and the speed of the user's movement in the door direction, and controls the opening and closing operation of the door based on the estimation result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is conceivable to estimate the presence or absence of the boarding intention of a person existing on the road facing the vehicle door in order to control the opening and closing operation of the vehicle door. However, even when a person on the road is moving in the direction approaching the vehicle, that person does not necessarily approach the vehicle and board it, and may pass by even though approaching the vehicle. Therefore, unlike the determination of the presence or absence of the boarding intention of a person existing in front of an elevator, it is difficult to determine the presence or absence of the boarding intention of a person moving in the direction approaching the vehicle. It is required to appropriately control the opening and closing operation of the door.

[0005] In view of such circumstances, an object of the present disclosure is to appropriately control the opening and closing of a vehicle door.

Means for Solving the Problems

[0006] A door control device according to one embodiment of the present disclosure includes a control unit that controls the opening and closing of a door provided in an opening of a vehicle based on image data of a shooting range in which one or more moving bodies exist in an external area relative to the opening of the vehicle. The control unit determines from the time-series data of the image data whether or not the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The control unit changes the opening and closing timing of the door based on the determination result of whether or not the one or more moving bodies intend to board the vehicle.

[0007] A vehicle control system according to one embodiment of the present disclosure comprises the door control device, the vehicle, and a camera for capturing the captured data.

[0008] A door control method according to one embodiment of the present disclosure is a method for controlling the opening and closing of a door provided in an opening of a vehicle based on photographic data of a shooting range in which one or more moving bodies exist in an external area relative to the opening of the vehicle. The door control method includes determining whether or not the one or more moving bodies intend to board the vehicle, based on the time-series data of the photographic data and the positional relationship between the direction of movement of the one or more moving bodies and the opening. The door control method includes changing the opening and closing timing of the door based on the determination result of whether or not the body intends to board the vehicle.

[0009] A door control program according to one embodiment of the present disclosure is a program that controls the opening and closing of a door provided in an opening of a vehicle based on image data of a shooting range in which one or more moving bodies exist in an external area relative to the opening of the vehicle. The door control program causes a processor to determine whether or not the one or more moving bodies intend to board the vehicle, based on the time-series data of the image data and the positional relationship between the direction of movement of the one or more moving bodies and the opening. The door control program causes a processor to change the opening and closing timing of the door based on the result of the determination of whether or not the one or more moving bodies intend to board the vehicle. [Effects of the Invention]

[0010] According to a door control device, vehicle control system, door control method, and door control program according to one embodiment of the present disclosure, the opening and closing of vehicle doors are appropriately controlled. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram illustrating an example of the configuration of a vehicle equipped with a controllable door as described in this disclosure. [Figure 2] This is a block diagram showing an example configuration of a vehicle control system related to this disclosure. [Figure 3] This figure shows an example of an image captured by a camera. [Figure 4] This is a schematic diagram explaining the determination of moving block. [Figure 5] This is a side view showing an example of the shooting range where a person is present. [Figure 6] This is a schematic diagram showing the corresponding captured image in Figure 5. [Figure 7] This flowchart shows an example procedure for the door control method related to this disclosure. [Figure 8] This flowchart shows an example of the procedure for determining whether a passerby is excluded. [Modes for carrying out the invention]

[0012] As shown in Figure 1, when the vehicle 300 is equipped with a door 72 at the opening, passengers of the vehicle 300 board the vehicle 300 from the external area of ​​the vehicle 300 through the door 72. The door 72 may be replaced with various other configurations that can be opened and closed.

[0013] The vehicle control system 1 (see Figure 2) uses the camera 10 to photograph the area outside the vehicle 300 around the door 72, detects passengers present within the shooting range 10A as moving objects based on the captured images, determines the presence or absence of passengers, and controls the opening and closing of the door 72.

[0014] In addition, passengers may rush into the vehicle 300. Around the vehicle 300, there are not only passengers who intend to rush into the vehicle 300 but also persons other than passengers such as pedestrians. In order to improve the accuracy of detecting rushing into the vehicle 300, that is, to reduce false detection of rushing into the vehicle, it is necessary to appropriately execute determination based on the moving direction of the person.

[0015] The vehicle control system 1 determines whether the moving object is a passenger or a person other than a passenger such as a pedestrian based on the positional relationship between the moving direction of a predetermined area including the detected moving object and an opening such as the door 72 of the vehicle 300 from the time-series data of the captured image. The vehicle control system 1 determines whether a moving object approaching the vehicle 300 has the intention of boarding the vehicle 300 based on the determination result as to whether the moving object is a passenger. The vehicle control system 1 controls the opening and closing of an opening such as the door 72 based on the determination result of the intention of boarding the vehicle 300.

[0016] Hereinafter, an example of an embodiment of the vehicle control system 1 according to the present disclosure will be described.

[0017] (Configuration example of vehicle control system 1) As shown in FIG. 2, the vehicle control system 1 according to the present disclosure includes a driving control device 100 and an object recognition device 16.

[0018] <Driving control device 100> The driving control device 100 includes an external recognition unit 110, a host vehicle position recognition unit 120, an operation detection unit 130, a travel control unit 140, and a contact possibility determination unit 150. The driving control device 100 includes a driver state recognition unit 160, a driving support control unit 170, a motion vector calculator 180, a motion vector corrector 182, and a door control unit 190. The driving support control unit 170 includes a braking force control unit 172 and a stop control unit 174.

[0019] The external environment recognition unit 110 recognizes the external environment of the vehicle 300 based on the recognition result of the object recognition device 16 described later. The vehicle position recognition unit 120 recognizes the position of the vehicle 300 based on satellite positioning data or the like. The operation detection unit 130 detects the driving operation of the vehicle 300. The driving control unit 140 controls the driving of the vehicle 300. The driving control unit 140 may control the vehicle 300 to travel in autonomous driving. The contact possibility determination unit 150 determines the possibility that the vehicle 300 contacts an object existing in the vicinity based on information such as the camera 10. The driver state recognition unit 160 recognizes the state of the driver of the vehicle 300 based on the data detected by the in-vehicle camera 50 or the biometric information detection sensor 60 described later. The driving support control unit 170 supports the driving of the vehicle 300 based on the recognition result of the state of the vehicle 300 or the state of the driver of the vehicle 300. The braking force control unit 172 automatically applies the brakes of the vehicle 300 as necessary. The stop control unit 174 automatically stops the running of the vehicle 300 as necessary.

[0020] The motion vector calculator 180 calculates a motion vector representing the moving direction of a moving object detected around the vehicle 300. The motion vector corrector 182 corrects the motion vector.

[0021] The door control unit 190 controls the opening and closing of the door 72 of the vehicle 300.

[0022] The driving control device 100 may include one or more processors or dedicated circuits to realize the functions of each component. In the present embodiment, the processor may be a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited thereto. The dedicated circuit may include, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0023] The operation control device 100 may include a storage unit. The storage unit may include, but is not limited to, semiconductor memory, magnetic memory, or optical memory. The storage unit may function as, for example, a main memory, auxiliary memory, or cache memory. The storage unit may include an electromagnetic storage medium such as a magnetic disk. The storage unit may include a non-temporary computer-readable medium. The storage unit stores any information or programs used for the operation of the operation control device 100. The storage unit may store, for example, a system program or an application program. The storage unit may be included in a processor or dedicated circuit.

[0024] The driving control device 100 may be configured to include an interface for communicating information or data with other components of the vehicle control system 1 or with external devices.

[0025] The interface may include a communication module configured to communicate with other components or external devices via a network. The communication module may support mobile communication standards such as 4G (4th Generation) or 5G (5th Generation). The communication module may also support communication standards such as LAN (Local Area Network). The communication module may support wired or wireless communication standards. The communication module is not limited to these and may support various communication standards. The interface may be configured to connect to the communication module.

[0026] The interface may include terminals that conform to standards such as RS-232C or RS-485, allowing for direct connection to other components or external devices.

[0027] The driving control device 100 may be configured to include an input device that receives information or data from a user of the vehicle control system 1. The input device may include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may also include physical keys. The input device may also include an audio input device such as a microphone. The driving control device 100 may be configured to be connectable to an external input device. The driving control device 100 may be configured to be able to acquire information or data input to an external input device from that external input device.

[0028] The operation control device 100 may be configured to include an output device that outputs information or data to the user. The output device may include, for example, a display device that outputs visual information such as images, characters, or graphics. The display device may include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, an inorganic EL display, or a PDP (Plasma Display Panel). The display device is not limited to these displays and may include various other types of displays. The display device may include a light-emitting device such as an LED (Light Emitting Diode) or an LD (Laser Diode). The display device may include various other devices. The output device may include, for example, an audio output device such as a speaker that outputs auditory information such as voice. The output device is not limited to these examples and may include various other devices. The operation control device 100 may be configured to be connectable to an external output device. The operation control device 100 may be configured to output information or data to an external output device.

[0029] The driving control device 100 may be configured to include one or a plurality of server devices that can communicate with each other. The driving control device 100 may be implemented as a cloud server. The driving control device 100 may be mounted on the vehicle 300. The driving control device 100 does not have to be mounted on the vehicle 300. At least a part of the components of the driving control device 100 may be mounted on the vehicle 300. At least a part of the components of the driving control device 100 does not have to be mounted on the vehicle 300.

[0030] <Object recognition device 16> The object recognition device 16 comprises a person detection unit 162, a person detection ReID (Re-Identification) unit 164, and a person area extraction unit 166. The object recognition device 16 acquires data from the camera 10, radar 12, and finder 14 (described later) detecting objects around the vehicle 300, and recognizes people present around the vehicle 300. The person detection unit 162 detects people present around the vehicle 300. The person detection ReID unit 164 tracks the movement of the detected people. The person area extraction unit 166 extracts the area around the vehicle 300 where people are present.

[0031] The object recognition device 16 may be configured to include one or more processors or dedicated circuits in order to realize the functions of each component. The processor or dedicated circuit may be configured in the same way as the processor or dedicated circuit of the driving control device 100. The object recognition device 16 may include a storage unit. The storage unit may be configured in the same way as the storage unit of the driving control device 100. The driving control device 100 may be configured to include an interface for communicating information or data with other components of the vehicle control system 1 or with external devices. The interface may be configured in the same way as the interface of the driving control device 100. The object recognition device 16 may be configured as part of the driving control device 100.

[0032] The object recognition device 16 may be configured to include one or a plurality of server devices that can communicate with each other. The object recognition device 16 may be implemented as a cloud server. The object recognition device 16 may be mounted on the vehicle 300. The object recognition device 16 may not be mounted on the vehicle 300. At least a portion of the components of the object recognition device 16 may be mounted on the vehicle 300. At least a portion of the components of the object recognition device 16 may not be mounted on the vehicle 300.

[0033] <Vehicle 300> The vehicle control system 1 may further include various components of the vehicle 300. The vehicle 300 includes a camera 10. The vehicle 300 is also equipped with, though not required, a radar 12, a finder 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, an in-cabin camera 50, and a biometric information detection sensor 60. The vehicle 300 is also equipped with a driver control unit 80, a driving control device 230, and a boarding / alighting control device 70.

[0034] Camera 10 photographs the area around vehicle 300. The area photographed by camera 10 is also called the shooting range 10A (see Figure 1). Radar 12 and finder 14 may detect objects around vehicle 300 in a manner other than images, such as point cloud data. The information about the area around vehicle 300 detected by camera 10, radar 12, and finder 14 is output to object recognition device 16.

[0035] The communication device 20 connects the remote center 2, which monitors the status of the vehicle 300, and the driving control device 100 in a communicative manner. The HMI 30 functions as an interface between the driver of the vehicle 300 and the driving control device 100.

[0036] The vehicle sensor 40 detects various states of the vehicle 300 and outputs the detection results to the driving control device 100. The vehicle sensor 40 may be controlled by the driving control device 100.

[0037] The in-vehicle camera 50 photographs the interior of the vehicle 300. The in-vehicle camera 50 may also photograph the driver or passengers of the vehicle 300. The in-vehicle camera 50 outputs the captured images to the driving control device 100. The in-vehicle camera 50 may be controlled by the driving control device 100.

[0038] The biometric information detection sensor 60 detects biometric information of the driver of the vehicle 300. The biometric information may include, for example, heart rate or blood pressure. The biometric information may also include the driver's state of alertness.

[0039] The driver control unit 80 includes an accelerator pedal 82, a brake pedal 84, and a steering wheel 86. The driver control unit 80 is operated by the driver of the vehicle 300. If the vehicle 300 is controlled by automated driving, it does not need to be equipped with the driver control unit 80.

[0040] The driving control device 230 includes a driving force output device 200, a brake device 210, and a steering device 220. The driving control device 230 may operate in response to the operation of the driver control device 80. The driving control device 230 may also operate in response to instructions from the driving control unit 140 or the driver assistance control unit 170 of the driver control device 100.

[0041] The passenger boarding / alighting control device 70 is equipped with doors 72 of the vehicle 300. The opening and closing of the doors 72 are controlled by the door control unit 190. The passenger boarding / alighting control device 70 is not required to include a display unit 74 and a speaker 76. The display unit 74 is installed outside the vehicle 300 and displays destination information or departure time of the vehicle 300 to passengers outside the vehicle 300. The speaker 76 is installed outside the vehicle 300 and outputs destination information or departure time of the vehicle 300 as audio information to passengers outside the vehicle 300.

[0042] (Example of operation of vehicle control system 1) The vehicle control system 1 according to this embodiment detects passengers who intend to rush into the vehicle 300 and controls the opening and closing of the door 72. The driving control device 100 or object recognition device 16 of the vehicle control system 1 is a device used to control the opening and closing of the door 72 and is also called a door control device. An example of the operation of the vehicle control system 1 will be described below.

[0043] The object recognition device 16 acquires images of the shooting range 10A from the camera 10. As illustrated in Figure 3, the captured images include images of objects located within the shooting range 10A surrounding the vehicle 300. In this example, it is assumed that moving objects 3 and 4, which are present around the vehicle 300, are captured in the images. Moving object 3 is assumed to be a person who intends to board the vehicle 300. Moving object 4 is assumed to be a person who does not intend to board the vehicle 300.

[0044] The object recognition device 16 acquires images captured by the camera 10 at each of several time points. Specifically, the camera captures the shooting range 10A at both the first and second time points. The second time point is defined as a predetermined time elapsed from the first time point. The predetermined time may be the frame rate at which the camera 10 records video. The image captured at the first time point is also referred to as the first captured image. The image captured at the second time point is also referred to as the second captured image.

[0045] The object recognition device 16 recognizes a person, such as a moving object 3 or 4, in the captured image using the person detection unit 162, and tracks the person using the person detection ReID unit 164 to calculate the person's movement. The object recognition device 16 outputs the recognition result of the person, such as a moving object 3 or 4, in the captured image, and the calculation result of their movement to the operation control device 100.

[0046] The object recognition device 16 may determine that any person it detects is a pedestrian. A pedestrian is a person who does not intend to board the vehicle 300, that is, a moving object that does not intend to board the vehicle 300. In other words, the object recognition device 16 determines whether a moving object intends to board the vehicle 300 and determines that a moving object that does not intend to board the vehicle 300 is a pedestrian. Based on the orientation of the face of the detected person, the object recognition device 16 may determine that the face of the person is not facing the door 72 and determine that the person whose face is not facing the door 72 is a pedestrian. To improve the accuracy of face orientation determination, the object recognition device 16 may acquire multiple images of the person whose face orientation is to be determined and determine the face orientation in each image. The object recognition device 16 may determine that a person is not facing the door 72 of the vehicle 300 based on an angle threshold in relation to the time-series trajectory of the detected person and determine that the person is not facing the door 72 is a pedestrian. The angle threshold is a value that can be determined as appropriate.

[0047] The object recognition device 16 may be configured not to output the detection result of a person identified as a pedestrian and the calculation result of that person's movement to the driving control device 100. This eliminates the need to calculate the movement vector for pedestrians, as described later. As a result, the processing load on the vehicle control system 1 is reduced.

[0048] The motion vector calculator 180 of the driving control device 100 calculates motion vectors as feature quantities of an object, for example, a moving object 3 or 4, that is captured in the image. As preparation for calculating motion vectors, the motion vector calculator 180 divides the captured image into multiple blocks 5. In this example of operation, the blocks 5 are assumed to be divided in a grid pattern. The manner in which the blocks 5 are divided is not limited to this. The motion vector calculator 180 calculates motion vectors in each of the divided blocks 5.

[0049] The motion vector of an object represents the direction and distance the object moved from its position in the first image to its position in the second image. The value obtained by dividing the distance the object moved by a predetermined time corresponds to the speed of the object. The motion vector calculator 180 may calculate the motion vector in each block 5 using a model that outputs a motion vector for block 5 units when the first image and the second image are input. The model may be, for example, a model based on Dense Opticalflow or Deep Learning.

[0050] For example, as shown in Figure 4, the motion vector calculator 180 can calculate a motion vector 5V representing the direction and distance of movement of a part of the moving object 3 in each of the blocks 5 in which the moving object 3 (a person) is pictured.

[0051] The motion vector corrector 182 of the driving control device 100 converts the motion vectors of objects in each block 5, such as moving bodies 3 and 4, calculated by the motion vector calculator 180, into scalar values ​​of the object's motion. The scalar value of an object's motion represents the magnitude of the object's motion, regardless of the direction of its motion. The scalar value of an object's motion is calculated as the length, i.e., the absolute value, of the object's motion vector. In the example in Figure 4, the motion vector corrector 182 can calculate a scalar value 5S converted from vector 5V, which is the motion vector of the part of the moving body 3, in each of the blocks 5 in which the moving body 3 is visible.

[0052] The motion vector corrector 182 may extract motion vectors in units of block 5 where the absolute value of the angular difference between the direction in which the detected object is moving toward the door 72 of the vehicle 300 and the direction of the motion vector is within an angular threshold, and convert the extracted motion vectors into scalar values ​​of the object's motion. The angular threshold is a value that can be determined as appropriate. By limiting the blocks 5 in which the conversion from motion vector to scalar value is performed using the angular threshold, objects moving in directions other than toward the door 72 of the vehicle 300, i.e., objects unrelated to the opening and closing control of the door 72, are excluded from recognition. As a result, the processing load on the vehicle control system 1 is reduced.

[0053] The motion vector corrector 182 determines that blocks 5 whose converted scalar value is equal to or greater than a threshold value are dynamic blocks. In this example, let's assume the threshold value for the scalar value is set to 7. In this case, in the example in Figure 4, the motion vector corrector 182 determines that the seven blocks 5 whose scalar value 5S is 7 or greater are dynamic blocks 6. The seven blocks 5 determined to be dynamic blocks 6 are enclosed by thick lines in the example in Figure 4. On the other hand, blocks 5 whose scalar value 5S is 6 are not included in dynamic blocks 6.

[0054] The motion vector corrector 182 removes motion blocks that are considered noise from the determined motion blocks. For example, the motion vector corrector 182 may remove isolated motion blocks 6 from among the motion blocks 6 within the multiple blocks 5 obtained by dividing the captured image as noise. Even when multiple motion blocks 6 are adjacent, the motion vector corrector 182 may remove multiple adjacent motion blocks 6 as noise if the area of ​​the region connecting the adjacent motion blocks 6 is smaller than the area expected when a person is captured.

[0055] The motion vector corrector 182 may exclude the motion block 6 from the motion block 6 by setting the magnitude of the motion vector of the motion block 6 to 0 to remove it as noise.

[0056] The motion vector corrector 182 may remove motion blocks 6 corresponding to people recognized as pedestrians by the object recognition device 16 as noise. As illustrated in Figure 3, if there are multiple pedestrian blocks 8 lined up, the motion vector corrector 182 may determine that the multiple pedestrians 8 are moving as a group and may connect the blocks of the multiple pedestrians 8 and consider them as one pedestrian area 9. Even if there are other blocks between each of the blocks of the multiple pedestrians 8, if there is no room for other people to enter between the multiple pedestrians 8, the object recognition device 16 may connect the blocks of the multiple pedestrians 8 and the blocks in between them and consider them as one pedestrian area 9. In other words, if the object recognition device 16 determines that a plurality of moving objects that have been determined not to intend to board the vehicle 300 are less than a predetermined value, it may set the area including the area between the area of

[0057] The motion vector corrector 182 determines the motion block 6 corresponding to the moving object from the motion block 6 remaining after noise reduction. Specifically, the motion vector corrector 182 groups adjacent motion block 6 together and determines the grouped motion block 6 as the motion block 6 corresponding to the moving object. In the example in Figure 4, seven motion block 6 are grouped together as the motion block 6 corresponding to the moving object 3.

[0058] Here, in the captured image, objects located near the camera 10 and objects located far from the camera 10 appear to be of different sizes. Specifically, objects located far from the camera 10 appear smaller in the captured image than objects located near the camera 10. As illustrated in Figure 3, assume that a grid 7 extending horizontally in the captured image is set at equal intervals vertically. The spacing of these grids 7 widens as you move further away from the camera 10 in the actual shooting range 10A, as shown in Figure 5. As shown in Figure 5, even if moving objects 3 and 4 are people of the same height, as shown in Figure 6, moving object 4 located far from the camera 10 appears smaller in the captured image than moving object 3 located near the camera 10. The grids 7 in Figures 5 and 6 are corresponding by the symbols P0 to P5.

[0059] From the above, the magnitude of motion of an object that is positioned higher in the captured image appears smaller than the magnitude of motion of an object that is positioned lower in the captured image. Therefore, the motion vector corrector 182 corrects the scalar value according to the position in the captured image in order to reduce the influence of the determination based on the distance from the camera 10 to the object.

[0060] The motion vector corrector 182 determines the grounding position of the moving body as preparation for correcting the scalar value. Specifically, for each group of moving body blocks 6, the motion vector corrector 182 identifies the moving body block 6 located at the bottom edge of the captured image. As illustrated in Figure 5, when the shooting range 10A by the camera 10 is viewed from the side, the grounding position 3A of the moving body 3 closest to the camera 10 and the grounding position 4A of the moving body 4 closest to the camera 10 are defined. Grounding position 3A is the moving body block 6 located at the bottom edge of the group of moving body blocks 6 corresponding to the moving body 3. Grounding position 4A is the moving body block 6 located at the bottom edge of the group of moving body blocks 6 corresponding to the moving body 4.

[0061] The motion vector corrector 182 corrects the scalar value according to the grounding position of the moving object. The object recognition device 16 corrects the scalar value by increasing it as the grounding position of the moving object is closer to the top edge of the captured image, assuming that the moving object is located farther from the camera 10 and appears smaller in the image. The motion vector corrector 182 may correct the scalar value by multiplying it by a scalar value correction coefficient listed in the correction table shown in Table 1, for example.

[0062] [Table 1]

[0063] In the correction table in Table 1, B0 to B7 are codes corresponding to each row when the captured image is divided vertically into eight blocks 5. B0 is the region located at the bottom edge of the captured image and is closest to the camera 10. B7 is the region located at the top edge of the captured image and is furthest from the camera 10. P0 to P3 correspond to the grid 7 illustrated in Figures 3, 5, and 6.

[0064] Specifically, the motion vector corrector 182 corrects the scalar values ​​using the correction table in Table 1 as described below. When the bottom edge of a group of motion blocks 6 is located on the grid represented by P0, the motion vector corrector 182 multiplies the scalar value of the motion block 6 in row B0, which is at the same height as P0, by 1, multiplies the scalar value of the motion block 6 in row B1 by 1.1, multiplies the scalar value of the motion block in any row from B2 to B6 by 1.3, and multiplies the scalar value of the motion block 6 in row B7 by 1.5. Furthermore, if the bottom edge of the group of motion blocks 6 is located on the grid represented by P1, the motion vector corrector 182 multiplies the scalar value of the motion block 6 in row B1, which is at the same height as P1, by 2, multiplies the scalar value of the motion block in any row from B2 to B6 by 2.2, and multiplies the scalar value of the motion block 6 in row B7 by 2.6. Also, if the bottom edge of the group of motion blocks 6 is located on the grid represented by P2, the motion vector corrector 182 multiplies the scalar value of the motion block 6 in any row from B2 to B6, which is at the same height as P2, by 3, multiplies the scalar value of the motion block in any row from B2 to B6 by 3, and multiplies the scalar value of the motion block 6 in row B7 by 3.4. Furthermore, if the bottom edge of a group of motion block 6 is located on the grid represented by P3, the motion vector corrector 182 multiplies the scalar value of the motion block 6 in row B7 that is at the same height as P3 by 4.

[0065] The correction table in Table 1 above is just one example. The motion vector corrector 182 may create a correction table as appropriate based on the relationship between the size and position of objects captured in the image taken by the camera 10 and the actual size and position of objects in the shooting range 10A.

[0066] The motion vector corrector 182 outputs the corrected scalar value, as described above, to the door control unit 190 of the driving control device 100. The door control unit 190 integrates the corrected scalar values ​​of each moving block 6 included in each group of moving block 6. The integrated corrected scalar value represents the amount of movement of the object corresponding to the group of moving block 6. For example, the door control unit 190 can calculate the amount of movement of the moving body 3 by correcting the scalar values ​​of the seven moving block 6 corresponding to the moving body 3 shown in Figure 4, and then integrating the corrected scalar values.

[0067] The door control unit 190 determines that an object, i.e., a person, intends to rush onto the vehicle 300 if the amount of movement of the object exceeds a threshold. If the door control unit 190 determines that one or more people in the captured image intend to rush onto the vehicle 300, it detects the rush onto the vehicle 300.

[0068] If the door control unit 190 does not detect someone rushing onto the vehicle 300, it leaves the door 72 open. If the door control unit 190 detects someone rushing onto the vehicle 300, it closes the door 72.

[0069] The door control unit 190 may keep the door 72 open when it detects someone rushing onto the vehicle 300, and may close the door 72 if the length of time during which no rushing onto the vehicle 300 is detected exceeds a threshold.

[0070] <Example of a door control procedure> The vehicle control system 1 according to this embodiment may execute a door control method, including the steps of the flowchart illustrated in Figure 7, to determine whether a moving object surrounding the vehicle 300 intends to board the vehicle 300 and to control the opening and closing of the door 72. The door control method may be implemented as a door control program to be executed by a processor such as the driving control device 100 or object recognition device 16 of the vehicle control system 1. The door control program may be stored in a non-temporary computer-readable medium.

[0071] The object recognition device 16 receives an image input from the camera 10 capturing the shooting range 10A around the vehicle 300 (S1). The object recognition device 16 may also receive input of information about objects around the vehicle 300 from the radar 12 or the finder 14.

[0072] The object recognition device 16 calculates feature quantities of objects in the captured image (S2). The object recognition device 16 divides the captured image into multiple blocks and calculates the motion vector of the object in each block as a feature quantity. Based on a comparison of the image captured at the first time step and the image captured at the second time step, which is a predetermined time after the first time step, the object recognition device 16 calculates the direction in which the object moved and the distance the object moved between the first and second time steps. The motion vector of the object represents the direction and distance the object moved. The value obtained by dividing the distance the object moved by the predetermined time corresponds to the speed of the object.

[0073] The object recognition device 16 converts the motion vector of an object in each block into a scalar value of the object's motion (S3). The scalar value of an object's motion does not include a component of the direction of the object's motion, but only includes a component of the magnitude of the motion, i.e., the speed of the motion. The object recognition device 16 may extract motion vectors on a block-by-block basis for which the absolute value of the angular difference between the direction toward the door 72 of the vehicle 300 and the direction of the motion vector is within an angular threshold, and convert the extracted motion vectors into a scalar value of the object's motion. The angular threshold is a value that can be determined as appropriate.

[0074] The object recognition device 16 determines the moving blocks (S4). The object recognition device 16 determines blocks whose scalar value is greater than or equal to a threshold as moving blocks. The threshold is a value that is set as appropriate.

[0075] The object recognition device 16 removes motion noise (S5). The object recognition device 16 may exclude motion blocks that are isolated from other motion blocks from the motion blocks.

[0076] The object recognition device 16 determines the moving object (S6). The object recognition device 16 groups adjacent moving object blocks together.

[0077] The object recognition device 16 determines the ground contact position of the moving object (S7). The object recognition device 16 identifies the block located at the bottom edge of the image in groups of moving object blocks. The object recognition device 16 determines the grid number of the identified block located at the bottom edge.

[0078] The object recognition device 16 corrects the scalar value based on the ground contact position (S8). The object recognition device 16 obtains the scalar value correction coefficient for each moving block included in the group of moving blocks from the correction table shown in Table 1 above, etc., based on the grid number. The object recognition device 16 corrects the scalar value for each moving block by multiplying the scalar value by the scalar value correction coefficient.

[0079] The object recognition device 16 calculates the amount of movement for each moving object (S9). The object recognition device 16 accumulates the corrected scalar values ​​of the moving object blocks in groups. The scalar values ​​accumulated in groups represent the amount of movement of the moving object corresponding to the group of moving object blocks.

[0080] The driving control device 100 determines whether it has detected a moving body rushing onto the vehicle 300 (S10). The driving control device 100 determines that a moving body corresponding to a group of moving body blocks whose amount of movement is greater than or equal to a threshold has the intention to board the vehicle 300, and detects the moving body rushing onto the vehicle 300. In other words, the driving control device 100 determines that it has detected a moving body rushing onto the vehicle 300 if the amount of movement in one or more groups of moving body blocks is greater than or equal to a threshold.

[0081] If the driving control device 100 does not detect a moving object rushing onto the vehicle 300 (S10: NO), it controls the door control unit 190 to not close the door 72 (S11). If the driving control device 100 detects a moving object rushing onto the vehicle 300 (S10: YES), it controls the door control unit 190 to close the door 72 (S12). After executing the procedure in S11 or S12, the vehicle control system 1 finishes executing the procedure in the flowchart of Figure 7.

[0082] The object recognition device 16 may perform the pedestrian exclusion determination procedure exemplified in Figure 8 during the feature calculation procedure (S2) in Figure 7.

[0083] The object recognition device 16 detects a person in the captured image using the person detection unit 162 (S21). The object recognition device 16 performs ReID on the detected person using the person detection ReID unit 164 and tracks the person's movement (S22). The object recognition device 16 detects the area in which the person is captured using the person region extraction unit 166 (S23).

[0084] The object recognition device 16 determines whether a person is a pedestrian (S24). A pedestrian means a person who does not intend to board the vehicle 300. The object recognition device 16 may determine whether a person is not facing the door 72 of the vehicle 300 based on an angle threshold in the time-series trajectory of the detected person, and determine whether a person is not facing the door 72 as a pedestrian. The angle threshold is a value that can be determined as appropriate. The object recognition device 16 may determine whether a person's face is not facing the door 72 based on the orientation of the detected person's face, and determine whether a person's face is not facing the door 72 as a pedestrian. In order to improve the accuracy of face orientation determination, the object recognition device 16 may acquire multiple images of the person whose face orientation is to be determined and determine the face orientation in each image.

[0085] The object recognition device 16 determines that group movement is occurring (S25). As illustrated in Figure 3, if there are multiple blocks of pedestrians 8 lined up, the object recognition device 16 may determine that the multiple pedestrians 8 are moving as a group and may connect the blocks of the multiple pedestrians 8 to form a single pedestrian area 9. Even if there are other blocks between each of the blocks of the multiple pedestrians 8, if there is no room for other people to enter between the multiple pedestrians 8, the object recognition device 16 may connect the blocks of the multiple pedestrians 8 and the blocks in between them to form a single pedestrian area 9.

[0086] Returning to Figure 8, the object recognition device 16 determines the blocks to be excluded (S26). The object recognition device 16 excludes the blocks it considers to be in the pedestrian area from being determined as moving blocks in the flowchart of Figure 7 by setting the motion vector of the blocks to 0. After executing the procedure in S26, the object recognition device 16 finishes executing the procedure in the flowchart of Figure 8.

[0087] (summary) As described above, the vehicle control system 1 according to this disclosure can recognize the movement of a person in a captured image of the area around the vehicle 300, for each block into which the captured image has been divided. Based on the direction of the person's movement and the positional relationship with the vehicle 300's door 72, the vehicle control system 1 can determine whether the recognized person is a pedestrian. Based on the determination result that the person is a pedestrian, the opening and closing of the door 72 is controlled. As a result, the opening and closing of the door 72 is controlled appropriately. In addition, noise is efficiently removed by recognizing the person's movement for each block. As a result, the accuracy of estimating whether or not a person in the captured image intends to board the vehicle is improved.

[0088] In the embodiments described above, the vehicle control system 1 recognized the movement of a person using captured images, but it may also recognize the movement of a person using captured data in a form other than images, such as point cloud data detected by the radar 12 or finder 14.

[0089] The entities that execute the operations of the vehicle control system 1 are not limited to the components described above. For example, the operation of the object recognition device 16 may be performed by other devices such as the driving control device 100. The operation of the motion vector calculator 180, the motion vector corrector 182, or the door control unit 190 may be performed by various other components. The entities that execute the operations described above are also collectively referred to as control units.

[0090] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means or each step, etc., can be rearranged in a logically consistent manner, and multiple means or steps, etc., can be combined into one or divided.

[0091] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] The system includes a control unit that controls the opening and closing of a door provided in the opening of a vehicle based on imaging data of the imaging range in which one or more moving objects exist in the external area relative to the opening of the vehicle. The control unit, Based on the time-series data of the aforementioned photographic data, it is determined whether the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The timing of opening and closing the doors is changed based on the determination result of whether or not the person intends to board the vehicle. Door control device. [Note 2] The control unit determines, when determining whether the one or more moving bodies intend to board the vehicle, whether the one or more moving bodies intend to board the vehicle based on the positional relationship between the direction of movement of the shooting range in which the one or more moving bodies are located and the opening in the shooting data, as described in [Appendix 1]. [Note 3] The door control device according to [Appendix 1] or [Appendix 2], wherein the control unit, when the distance between the shooting ranges of a plurality of moving bodies that have been determined from the shooting data to have no intention of boarding the vehicle is less than a predetermined value, sets as a pedestrian area an area that includes the shooting range of each of the plurality of moving bodies that have been determined to have no intention of boarding the vehicle, and the block between the shooting ranges among the plurality of blocks divided in the shooting data. [Note 4] The control unit, In each of the plurality of blocks included in the aforementioned shooting range, a vector is calculated that includes the direction and magnitude of the movement of the moving object being photographed. Blocks whose vectors satisfy the conditions are determined to be moving body blocks corresponding to the moving body, while blocks whose vectors do not satisfy the conditions are determined to be blocks containing moving bodies that have been determined not to have the intention to board the vehicle. A door control device as described in any one of the following [Appendix 1] to [Appendix 3]. [Note 5] The control unit removes blocks from the moving body blocks that satisfy certain conditions as noise, as described in [Appendix 4]. [Note 6] The control unit groups adjacent blocks among the moving body blocks and considers the group of moving body blocks as the moving body, as described in [Appendix 5]. [Note 7] The control unit corrects the magnitude of the movement of the moving body based on the ground contact position of the moving body in the captured data, as described in any one of the Door Control Devices from [Appendix 4] to [Appendix 6]. [Note 8] The control unit corrects the magnitude of the movement of the moving body by multiplying it by a coefficient corresponding to the combination of the ground contact position of the moving body in the captured data and the vertical position of the block in which the moving body is captured. (See Appendix 7) [Note 9] A vehicle control system comprising a door control device described in any one of [Appendix 1] to [Appendix 8], the vehicle, and a camera for capturing the captured data. [Note 10] A method for controlling the opening and closing of a door provided in an opening of a vehicle, based on imaging data of an imaging range in which one or more moving objects exist in the external area relative to the opening of the vehicle, Based on the time-series data of the aforementioned photographic data, it is determined whether the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The timing of opening and closing the doors is changed based on the determination of whether or not the person intends to board the vehicle. Door control method, including [Note 11] The door control method according to [Appendix 10], wherein determining whether the one or more moving bodies intend to board the vehicle includes determining, in the shooting data, whether the one or more moving bodies intend to board the vehicle based on the positional relationship between the direction of movement of the shooting range in which the one or more moving bodies are located and the opening. [Note 12] The door control method according to [Appendix 10] or [Appendix 11], which includes setting a pedestrian area as the area including the respective shooting ranges of multiple moving bodies that have been determined from the shooting data to have no intention of boarding the vehicle, and the block between the shooting ranges among the multiple blocks divided in the shooting data, when the distance between the shooting ranges of multiple moving bodies that have been determined to have no intention of boarding the vehicle is less than a predetermined value. [Note 13] In each of the plurality of blocks included in the aforementioned shooting range, a vector is calculated that includes the direction and magnitude of the movement of the moving object being photographed. Blocks whose vector satisfies the condition are determined to be moving body blocks corresponding to the moving body, while blocks whose vector does not satisfy the condition are determined to be blocks containing moving bodies that have been determined not to have the intention to board the vehicle. A door control method described in any one of the following [Appendix 10] to [Appendix 12], including the above. [Note 14] The door control method according to [Appendix 13], which includes grouping adjacent blocks among the moving body blocks and considering the group of moving blocks as the moving body. [Note 15] A door control method according to [Appendix 13] or [Appendix 14], which includes correcting the magnitude of the movement of the moving body based on the ground contact position of the moving body in the aforementioned photographic data. [Note 16] The door control method according to [Appendix 15], which includes correcting the magnitude of the movement of the moving body by multiplying it by a coefficient corresponding to the combination of the grounding position of the moving body in the aforementioned photographic data and the vertical position of the block in which the moving body is photographed. [Note 17] A program that controls the opening and closing of a door provided in an opening of a vehicle, based on imaging data of an imaging range in which one or more moving objects exist in the external area relative to the opening of the vehicle, Based on the time-series data of the aforementioned photographic data, it is determined whether the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The timing of opening and closing the doors is changed based on the determination of whether or not the person intends to board the vehicle. A door control program that instructs the processor to execute. [Note 18] The door control program described in [Note 17] includes determining whether the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the shooting range in which the one or more moving bodies are located and the opening in the shooting data. [Note 19] A door control program according to [Appendix 17] or [Appendix 18], which causes the processor to set a pedestrian area when the distance between the shooting ranges of multiple moving objects that have been determined from the shooting data to have no intention of boarding the vehicle is less than a predetermined value, and the area that includes the shooting range of each of the multiple moving objects that have been determined to have no intention of boarding the vehicle, and the block between the shooting ranges in the multiple blocks divided in the shooting data. [Note 20] In each of the plurality of blocks included in the aforementioned shooting range, a vector is calculated that includes the direction and magnitude of the movement of the moving object being photographed. Blocks whose vectors satisfy the conditions are determined to be moving body blocks corresponding to the moving body, while blocks whose vectors do not satisfy the conditions are determined to be blocks showing moving bodies that have been determined not to have the intention to board the vehicle. A door control program described in any one of [Appendix 17] to [Appendix 19] that causes the processor to execute the above. [Explanation of Symbols]

[0092] 1. Vehicle control system 2 Remote Center 3, 4 Moving object (3A, 4A: ground position) 5 Blocks (5S: Scalar value representation, 5V: Vector quantity representation) 6. Moving Blocks 7 grid 8 passerby 9. Pedestrian block 16. Object recognition device (162: Human detection unit, 164: Human detection ReID unit, 166: Human area detection unit) 100 Driving control device (110: external environment recognition unit, 120: vehicle position recognition unit, 130: operation detection unit, 140: driving control unit, 150: contact possibility determination unit, 160: driver state recognition unit, 170: driving support control unit, 172: braking force control unit, 174: cancellation control unit, 180: motion vector calculator, 182: motion vector corrector, 190: door control unit) 300 Vehicle (10: Camera, 10A: Shooting range, 12: Radar, 14: Finder, 20: Communication device, 30: HMI, 40: Vehicle sensor, 50: In-cabin camera, 60: Biometric information detection sensor, 70: Boarding / Alighting control device, 72: Door, 74: Display unit, 76: Speaker, 80: Driver's controls, 82: Accelerator pedal, 84: Brake pedal, 86: Steering wheel, 200: Driving force output device, 210: Brake device, 220: Steering device, 230: Driving control device)

Claims

1. The system includes a control unit that controls the opening and closing of a door provided in the opening of a vehicle based on imaging data of the imaging range in which one or more moving objects exist in the external area relative to the opening of the vehicle. The control unit, Based on the time-series data of the aforementioned photographic data, it is determined whether the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The timing of opening and closing the doors is changed based on the determination result of whether or not the person intends to board the vehicle. Door control device.

2. The door control device according to claim 1, wherein the control unit determines whether the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the shooting range in which the one or more moving bodies are located and the opening in the shooting data.

3. The door control device according to claim 1, wherein, when the distance between the shooting ranges of a plurality of moving bodies determined from the shooting data to have no intention of boarding the vehicle is less than a predetermined value, the control unit sets a region including the shooting range of each of the plurality of moving bodies determined to have no intention of boarding the vehicle and the block between the shooting ranges among the plurality of blocks divided in the shooting data as a pedestrian area.

4. The control unit, In each of the plurality of blocks included in the aforementioned shooting range, a vector is calculated that includes the direction and magnitude of the movement of the moving object being photographed. Blocks whose vectors satisfy the conditions are determined to be moving body blocks corresponding to the moving body, while blocks whose vectors do not satisfy the conditions are determined to be blocks containing moving bodies that have been determined not to have the intention to board the vehicle. The door control device according to claim 1.

5. The door control device according to claim 4, wherein the control unit removes blocks from the moving body blocks that satisfy the conditions as noise.

6. The door control device according to claim 4, wherein the control unit groups adjacent blocks among the moving body blocks and considers the group of moving body blocks as the moving body.

7. The door control device according to any one of claims 4 to 6, wherein the control unit corrects the magnitude of the movement of the moving body based on the ground contact position of the moving body in the captured data.

8. The door control device according to claim 7, wherein the control unit corrects the magnitude of the movement of the moving body by multiplying it by a coefficient corresponding to the combination of the ground contact position of the moving body in the captured data and the vertical position of the block in which the moving body is captured.

9. A vehicle control system comprising a door control device according to any one of claims 1 to 5, the vehicle, and a camera for capturing the captured data.

10. A method for controlling the opening and closing of a door provided in an opening of a vehicle, based on imaging data of an imaging range in which one or more moving objects exist in the external area relative to the opening of the vehicle, Based on the time-series data of the aforementioned photographic data, it is determined whether or not the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The timing of opening and closing the doors is changed based on the determination of whether or not the person intends to board the vehicle. Door control method, including

11. The door control method according to claim 10, wherein determining whether the one or more moving bodies intend to board the vehicle includes determining, in the shooting data, whether the one or more moving bodies intend to board the vehicle based on the positional relationship between the direction of movement of the shooting range in which the one or more moving bodies are located and the opening.

12. The door control method according to claim 10 or 11, further comprising setting a pedestrian area as the area including the respective shooting ranges of multiple moving bodies that are determined from the shooting data to have no intention of boarding the vehicle, and the block between the shooting ranges among the multiple blocks divided in the shooting data, when the distance between the shooting ranges of multiple moving bodies that are determined to have no intention of boarding the vehicle is less than a predetermined value.

13. In each of the plurality of blocks included in the aforementioned shooting range, a vector is calculated that includes the direction and magnitude of the movement of the moving object being photographed. Blocks whose vector satisfies the condition are determined to be moving body blocks corresponding to the moving body, while blocks whose vector does not satisfy the condition are determined to be blocks containing moving bodies that have been determined not to have the intention to board the vehicle. The door control method according to claim 10, including the method described in claim 10.

14. The door control method according to claim 13, further comprising grouping adjacent blocks among the moving body blocks and considering the group of moving blocks as the moving body.

15. The door control method according to claim 13 or 14, further comprising correcting the magnitude of the movement of the moving body based on the ground contact position of the moving body in the aforementioned photographic data.

16. The door control method according to claim 15, further comprising multiplying the magnitude of the movement of the moving body by a coefficient corresponding to the combination of the ground contact position of the moving body in the aforementioned photographic data and the vertical position of the block in which the moving body is photographed, and correcting the movement.

17. A program that controls the opening and closing of a door provided in an opening of a vehicle, based on imaging data of an imaging range in which one or more moving objects exist in the external area relative to the opening of the vehicle, Based on the time-series data of the aforementioned photographic data, it is determined whether or not the one or more moving bodies intend to board the vehicle, based on the positional relationship between the direction of movement of the one or more moving bodies and the opening. The timing of opening and closing the doors is changed based on the determination of whether or not the person intends to board the vehicle. A door control program that instructs the processor to execute.

18. The door control program according to claim 17, wherein determining whether the one or more moving bodies intend to board the vehicle includes determining, in the shooting data, whether the one or more moving bodies intend to board the vehicle based on the positional relationship between the direction of movement of the shooting range in which the one or more moving bodies are located and the opening.

19. The door control program according to claim 17 or 18, wherein, if the distance between the shooting ranges of multiple moving bodies determined from the shooting data to have no intention of boarding the vehicle is less than a predetermined value, the processor is instructed to set a region including the shooting range of each of the multiple moving bodies determined to have no intention of boarding the vehicle, and the block between the shooting ranges among the multiple blocks divided in the shooting data, as a pedestrian area.

20. In each of the plurality of blocks included in the aforementioned shooting range, a vector is calculated that includes the direction and magnitude of the movement of the moving object being photographed. Blocks whose vector satisfies the condition are determined to be moving body blocks corresponding to the moving body, while blocks whose vector does not satisfy the condition are determined to be blocks showing a moving body that has been determined not to have the intention to board the vehicle. The door control program according to claim 17, which causes the processor to execute the following.

Images