Vehicle control system and control method of vehicle control system

Abstract

To provide a vehicle control system capable of opening a door under closing control by accurately determining a person's intention to get in a vehicle.SOLUTION: A vehicle control system that controls opening and closing of a door at an entrance of a vehicle includes an out-of-vehicle camera that captures an image of a boarding area near the entrance, a boarding intention determination unit that determines whether or not there is a person who is making a predetermined boarding intention gesture with respect to the vehicle based on time series data of the image captured by the out-of-vehicle camera when door closing control is performed, and a door control unit that interrupts the door closing control and opens the door when the boarding intention determination unit determines that there is a person who is making the boarding intention gesture.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a vehicle control system and a control method for a vehicle control system. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2018-047993 is a known technical document related to a system for detecting a person's intention to board an elevator. This publication describes an elevator user detection system that detects changes in the foot position of the person closest to the door based on time-series data of images captured by a camera, and then detects whether or not a user intends to board the elevator from the change in foot position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-047993 A Summary of the Invention [Problem to be solved by the invention]

[0004] It has been considered to appropriately determine whether a person intends to board a vehicle such as a bus and to use this information to control the vehicle's entrance. However, when determining whether a person intends to board a vehicle rather than an elevator, there is a possibility that there may be people waiting for the next bus or people seeing them off, and there is a problem in that it is difficult to determine with sufficient accuracy just by focusing on changes in the position of the person's feet. [Means for solving the problem]

[0005] One aspect of the present invention is a vehicle control system that controls the opening and closing of a door at a vehicle entrance, and includes an exterior camera that captures images of the boarding area near the entrance, a boarding intention determination unit that determines whether or not a person is making a predetermined boarding intention gesture toward the vehicle based on time series data of images captured by the exterior camera when door closing control is being executed, and a door control unit that interrupts door closing control and opens the door when the boarding intention determination unit determines that a person is making a boarding intention gesture.

[0006] According to one aspect of the vehicle control system of the present invention, when it is determined that a person is making a predetermined gesture of intent to get in the vehicle, the door closing control is interrupted and the door is opened. Therefore, compared to when only focusing on the person's position, it is possible to accurately determine the person's intent to get in and open the door during closing control.

[0007] In a vehicle control system according to one aspect of the present invention, the boarding intention determination unit may determine whether or not a person is making a boarding intention gesture from the time series data of the captured image using a machine learning model that inputs time series data of the captured image and outputs a determination result as to whether or not a person is making a boarding intention gesture. According to this vehicle control system, it is possible to appropriately determine whether or not a person is making a gesture of intent to get in using a machine learning model.

[0008] In one embodiment of the vehicle control system of the present invention, the vehicle control system may further include a feature calculation unit that divides an image into a plurality of blocks and calculates a motion vector for each block based on time series data of the image, wherein the feature calculation unit calculates the magnitude of the boarding entrance direction component of the motion vector for each block, and the boarding intention determination unit may use the magnitude of the boarding entrance direction component of the motion vector for each block to determine whether or not someone is making a boarding intention gesture. According to this vehicle control system, it is assumed that people who intend to board will approach the boarding gate or raise their hand, so by focusing on the magnitude of the gate-direction component of the motion vector for each block, it is possible to appropriately determine whether or not someone is making a gesture indicating their intention to board.

[0009] In one embodiment of the vehicle control system of the present invention, the system further includes a correction unit that performs correction so that the magnitude of the boarding entrance direction component of the motion vector for each block increases the farther the block's position in the captured image is from the boarding entrance, and the boarding intention determination unit may determine whether or not someone is making a boarding intention gesture by performing a predetermined boarding intention gesture determination process on a group of blocks whose corrected boarding entrance direction component has a magnitude equal to or greater than a threshold value. According to this vehicle control system, the further away a person is from the boarding gate, the smaller the person appears in the captured image and the smaller their gesture movement will appear.Therefore, the system corrects the magnitude of the boarding gate direction component of the motion vector for each block so that it increases the farther away from the boarding gate.A predetermined boarding intention gesture determination process is performed on groups of blocks where the magnitude of the corrected boarding gate direction component is equal to or greater than a threshold, thereby making it possible to appropriately determine whether or not someone is making a boarding intention gesture.

[0010] Another aspect of the present invention is a control method for a vehicle control system that controls the opening and closing of a door at a vehicle entrance, and includes the steps of: determining whether or not a person is making a predetermined intention-to-board gesture toward the vehicle based on time series data of images captured by an exterior camera that captures an image of the boarding area near the entrance when door closing control is being executed; and interrupting door closing control and opening the door if it is determined in the intention-to-board determination step that a person is making a gesture indicating an intention to board.

[0011] According to a control method for a vehicle control system relating to another aspect of the present invention, when it is determined that a person is making a predetermined gesture of intent to get in the vehicle, the door closing control is interrupted and the door is opened. Therefore, compared to when focusing only on the person's position, it is possible to accurately determine the intent to get in and open the door during closing control. [Effects of the Invention]

[0012] According to each aspect of the present invention, it is possible to accurately determine whether a person near the entrance intends to board a vehicle and to open a door that is currently under closing control. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating a vehicle control system according to an embodiment. [Figure 2] FIG. 1 is a perspective view showing an example of a bus equipped with a vehicle control system. [Figure 3] 1A is a diagram illustrating an example of an image frame divided into a plurality of blocks, and FIG. 1B is a diagram illustrating how to determine a moving object heading toward a bus entrance in the image frame. [Figure 4] FIG. 10 is an enlarged view for explaining extraction of a moving object block in an image frame. [Figure 5] 1A is a diagram illustrating the bottom side of a group of moving body blocks that make up a moving body, and FIG. 1B is a diagram illustrating grid numbers. [Figure 6] 10A is a diagram illustrating an example of block row numbers in an image frame, and FIG. 10B is an example of a correction table according to block row numbers and ground contact positions. [Figure 7] FIG. 10 is a diagram illustrating an example of the distance from the boarding entrance to the target area. [Figure 8] Graphs (a) and (b) show an example of how the strength of super-resolution processing changes depending on the distance from the boarding entrance to the target area. [Figure 9]10A and 10B are graphs showing an example of changes in the human detection threshold and the strength of the gesture discrimination threshold depending on the distance from the boarding gate to the target group, respectively. [Figure 10] 3 is a flowchart illustrating an example of a control method for the vehicle control system. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] Fig. 1 is a block diagram showing a vehicle control system 1 according to one embodiment. In Fig. 1, the vehicle control system 1 of this embodiment includes an exterior camera 3, a communication device 4, a door driver 5, an announcement device 6, and an ECU (Electronic Control Unit) 10 mounted on a bus 2 (see Fig. 2), which is a vehicle for transporting passengers, as well as a monitoring control device 31 and a communication device 32 in a remote center 30. Note that the vehicle is not limited to the bus 2, and may be any vehicle that has a boarding entrance.

[0016] As shown in Fig. 2, a boarding entrance 2a (entrance / exit) where passengers get on and off is located in the center of the left side of the bus 2. The bus 2 has a door 25 that opens and closes the boarding entrance 2a. An exterior camera 3 is provided to capture images of the boarding area near the boarding entrance 2a.

[0017] The exterior camera 3 is disposed on the left side of the roof of the bus 2. The exterior camera 3 is a camera that captures an image of the boarding area outside the bus 2, including the boarding entrance 2a, to detect objects present in the boarding area. For example, a bird's-eye view camera with RGB functionality is used as the exterior camera 3. The exterior camera 3 is not particularly limited as long as it can be used for object detection. The boarding area is an area where object detection is performed by the exterior camera 3. The boarding area is an area for detecting people who intend to board. The boarding area may or may not include the boarding entrance 2a of the bus 2. The boarding area may coincide with the imaging area of ​​the exterior camera 3. The position of the exterior camera 3 is not particularly limited as long as it is a position where it can capture an image of the boarding area.

[0018] The communication device 4 communicates information relating to the monitoring and control of the bus 2 with the remote center 30 via wireless communication.

[0019] As described above, the remote center 30 has the monitoring control device 31 and the communication device 32. The monitoring control device 31 monitors the status of the bus 2 and provides driving support for the bus 2. The communication device 32 communicates wirelessly with the communication device 4 of the bus 2.

[0020] The door driving unit 5 is a driving unit that opens and closes the doors 25. The announcement unit 6 notifies information about the status of the bus 2 and the like to passengers inside and outside the bus by voice guidance.

[0021] The ECU 10 is an electronic control unit having a central processing unit (CPU) and a storage unit. The storage unit is composed of, for example, a read-only memory (ROM), a random access memory (RAM), and an electrically erasable programmable read-only memory (EEPROM). The ECU 10 realizes various functions by, for example, executing programs stored in the storage unit in the CPU. The ECU 10 may be composed of multiple electronic units.

[0022] The ECU 10 has a feature calculation unit 11, a feature conversion unit 12, a moving object block extraction unit 13, a noise removal unit 14, a grouping unit 15, a correction unit 16, a moving object selection unit 17, a super-resolution processing unit 18, a person detection unit 19, a boarding intention determination unit 20, and a door control unit 21.

[0023] The feature amount calculation unit 11 generates an image frame F by dividing the image captured by the outside-vehicle camera 3 into a plurality of blocks B, and calculates a feature amount representing the direction and amount of movement for each block B. The feature amount calculation unit 11 calculates a motion vector V as a feature amount for each block B from the time-series data of the captured image. The motion vector V has a magnitude and a direction. The time-series data of the captured image is data in which the captured images are arranged in chronological order.

[0024] There is no particular limitation on the method for calculating the motion vector V. Block matching, or the Lucas-Kanade method or the Horn-Schunck method using optical flow may be used to calculate the motion vector V. A machine learning model that is trained to input time-series data of a captured image and output a motion vector V for each block B may be used to calculate the motion vector V.

[0025] The machine learning model is a neural network such as a convolutional neural network. The neural network may include multiple layers, including multiple convolutional layers and pooling layers. A deep learning network based on deep learning is used as the neural network. A recurrent neural network may also be used as the machine learning model. The above content can also be adopted in various machine learning models that will be described below.

[0026] 3(a) is a diagram showing an example of an image frame F divided into multiple blocks B. FIG. 3(a) shows the image frame F, blocks B, a ground area R1, a bus ceiling area R2, a sky area R3, a boarding entrance 2a, an object m1, an object m2, and a motion vector V.

[0027] Blocks B are primarily set in the ground area R1 located outside the boarding gate 2a of the bus 2 in the captured image. The ground area R1 is captured between the bus ceiling area R2 and the sky area R3 in the captured image. In each block B, the direction and amount of movement are expressed by the direction and magnitude of the motion vector V. Note that in the image frame F shown in Figure 3(a), the direction and magnitude of the motion vector V are expressed only for four blocks B (see the upper left corner of the figure), but the direction and magnitude of the motion vector V are expressed for all blocks B. Objects m1 and m2 are objects detected from the image captured by the exterior camera 3.

[0028] The feature conversion unit 12 calculates the magnitude of the entrance direction component of the motion vector V calculated by the feature calculation unit 11. FIG. 3(b) is a diagram for explaining the determination of a moving object heading toward the bus entrance in an image frame. The magnitude of the entrance direction component of the motion vector V is the magnitude of the projection vector when the motion vector V is projected onto a line from each block B toward the entrance 2a. The line from each block B toward the entrance 2a can be, for example, a line connecting the center point of each block B to an arbitrary point on the entrance 2a (for example, the center of the door opening when the entrance 2a is open). A certain amount of error is allowed for the entrance direction. The method for calculating the magnitude of the entrance direction component of the motion vector V is not particularly limited. The magnitude of the entrance direction component of the motion vector V may be directly output from a machine learning model using time-series data of the captured image as input. The feature conversion unit 12 converts the magnitude of the entrance direction component of the motion vector V into a scalar value.

[0029] The moving object block extraction unit 13 extracts, as a moving object block Bm, a block B that has movement of a specified amount or more toward the boarding gate 2a of the bus 2, based on the scalar value converted for each block B by the feature conversion unit 12. Figure 4 is an enlarged view for explaining the extraction of a moving object block in an image frame. The moving object block extraction unit 13 extracts, as a moving object block Bm, a block B whose scalar value is equal to or greater than a predetermined threshold (see Figures 3(b) and 4).

[0030] The noise removal unit 14 removes isolated moving body blocks Bm from the moving body blocks Bm extracted by the moving body block extraction unit 13 as noise.

[0031] The grouping unit 15 determines a moving object moving toward the boarding gate 2a of the bus 2 by grouping adjacent moving object blocks Bm that include the same moving object among the moving object blocks Bm extracted by the moving object block extraction unit 13 but not removed by the noise removal unit 14 (see FIG. 3(b)). The grouping unit 15 determines a moving object moving toward the boarding gate 2a of the bus 2 by grouping adjacent moving object blocks Bm among the moving object blocks Bm.

[0032] In the image frame F shown in Figure 3(b), two moving objects m1 and m2 are determined to be heading towards the boarding gate 2a of the bus 2. The moving objects m1 and m2 are respectively included in moving object detection frames W1 and W2, which are made up of multiple moving object blocks Bm. Well-known methods can be used to set the moving object detection frames W1 and W2 on the image frame F.

[0033] The correction unit 16 corrects the scalar value to improve the accuracy of selecting the moving object. The correction unit 16 first determines the ground contact position of the moving object. The correction unit 16 identifies the moving object block Bm at the bottom side in the captured image for each group of moving object blocks Bm, and determines the grid number to which the bottom side of the moving object block Bm belongs, thereby determining the ground contact position.

[0034] 5(a) is a diagram illustrating the bottom sides of the group of moving object blocks that make up a moving object. Fig. 5(a) shows the bottom side g1 of the group of moving object blocks Bm that corresponds to moving object m1 (moving object detection frame W1) and the bottom side g2 of the group of moving object blocks Bm that corresponds to moving object m2 (moving object detection frame W2).

[0035] Fig. 5(b) is a diagram for explaining grid numbers. Fig. 5(b) shows grid numbers P0 to P4 and the bottom side g1 of moving object m1 in image frame F. As shown in Fig. 5(a) and Fig. 5(b), the correction unit 16 identifies the bottom side g1 of the group of moving object blocks Bm corresponding to moving object m1 from the captured image, and determines the grid number P1 to which the bottom side g1 belongs as the ground contact position P1 of moving object m1.

[0036] The correction unit 16 corrects the scalar value (the magnitude of the entrance direction component of the motion vector V) of each moving body block Bm in groups. The correction unit 16 corrects the scalar value of each moving body block Bm that constitutes the moving body m1 using a predetermined correction table, for example, based on the ground contact position P1 of the moving body m1 and the position of each moving body block Bm that constitutes the moving body m1.

[0037] Here, Fig. 6(a) is a diagram for explaining an example of block row numbers in an image frame F. In Fig. 6(a), block row numbers are set as B0 to B7 starting from the row closest to the boarding gate 2a. The correction unit 16 identifies the block row numbers shown in Fig. 6(a) as the positions of each moving body block Bm constituting the moving body m1.

[0038] FIG. 6(b) is an example of a correction table according to block row number and ground contact position. The correction table shown in FIG. 6(b) is set so that the correction coefficient increases the farther the position of block B is from the boarding entrance. Note that the correction table may set different correction coefficients according to the individual positions of block B, rather than the block row number. In this case, too, the correction coefficient is set so that the farther the position of block B is from the boarding entrance, the larger the correction coefficient. The correction unit 16 performs correction by multiplying the scalar value of each moving body block Bm by the correction coefficient obtained from the correction table shown in FIG. 6(b) based on the ground contact position P1 (grid number) of the moving body m1 and the position of each moving body block Bm that constitutes the moving body m1.

[0039] The moving object selection unit 17 selects moving objects to be subjected to super-resolution processing, which will be described later. The moving object selection unit 17 first calculates the average value of scalar values ​​for each group. For example, the moving object selection unit 17 calculates the average value by adding up the scalar values ​​of each moving object block Bm that constitutes the moving object m1 and dividing the sum by the number of moving object blocks Bm that constitute the moving object m1.

[0040] It should be noted that the moving body selection unit 17 does not necessarily need to calculate the average value. Any evaluation value that can appropriately determine that the moving body m1 is heading towards the boarding gate 2a will suffice. The moving body selection unit 17 may calculate the median value instead of the average value. The moving body selection unit 17 may also use a value obtained from a predetermined arithmetic formula using as input a scalar value for each moving body block Bm that constitutes the moving body m1.

[0041] The moving object selection unit 17 selects a moving object (a group of moving object blocks Bm) whose calculated average value (evaluation value) is equal to or greater than a threshold value. Note that the number of moving objects to be selected is not limited to one, and multiple moving objects may be selected. Here, it is assumed that moving objects m1 and m2 are selected.

[0042] The super-resolution processing unit 18 performs super-resolution processing on the moving object selected by the moving object selection unit 17. The super-resolution processing is image processing for improving the accuracy of person detection and boarding intention gesture determination, which will be described later. The super-resolution processing unit 18 performs cropping and resizing using a rectangular frame circumscribing the moving object to determine the target area for super-resolution processing. Note that the method for determining the target area is not limited to the above. The super-resolution processing unit 18 performs super-resolution processing on the target area, which involves at least one of noise removal, lens distortion correction, and upscaling.

[0043] The super-resolution processor 18 changes the strength of the super-resolution processing and the strength of the noise removal depending on the distance from the boarding gate 2a to the target area in the image frame F. Here, FIG. 7 is a diagram for explaining an example of the distance from the boarding gate 2a to the target area. In FIG. 7, the target area of ​​moving object m2 is the same as the inner area of ​​the moving object detection frame W2. The target area of ​​moving object m1 is a rectangular area that substantially overlaps with the moving object detection frame W1. The target areas are not shown here. In FIG. 7, the distance d1 from the boarding gate 2a to the target area of ​​moving object m1 and the distance d2 from the boarding gate 2a to the target area of ​​moving object m2 are shown.

[0044] FIG. 8(a) is a graph showing an example of the change in strength of super-resolution processing according to the distance from the boarding gate 2a to the target area. The vertical axis represents the strength of super-resolution processing. The horizontal axis represents the distance from the boarding gate 2a to the target area. As shown in FIG. 8(a), the super-resolution processor 18 increases the strength of super-resolution processing as the distance from the boarding gate 2a to the target area increases. For example, since distance d1<distance d2, when the strength of super-resolution processing for moving body m1 is set to 1.0, the super-resolution processor 18 performs super-resolution processing with the strength of super-resolution processing for moving body m2 set to 1.3.

[0045] FIG. 8(b) is a graph showing an example of the change in noise removal strength according to the distance from the boarding entrance 2a to the target area. The vertical axis represents the noise removal strength. The horizontal axis represents the distance from the boarding entrance 2a to the target area. As shown in FIG. 8(b), the super-resolution processor 18 increases the noise removal strength as the distance from the boarding entrance 2a to the target area increases. For example, since distance d1 is less than distance d2, the super-resolution processor 18 performs noise removal with the noise removal strength for moving body m1 set to 1.0 and the noise removal strength for moving body m2 set to 1.2. Note that the strength of the super-resolution processing and the strength of the noise removal do not necessarily need to be changed according to the distance from the boarding entrance 2a to the target area. The strength of the super-resolution processing and the strength of the noise removal may be fixed.

[0046] The human detection unit 19 performs human detection on the moving objects m1 and m2 that have been subjected to super-resolution processing by the super-resolution processing unit 18. The human detection unit 19 may perform human detection using a machine learning model. For human detection using a machine learning model, a method such as Haar Cascade can be adopted. The human detection unit 19 may also perform human detection using a well-known method such as pattern matching.

[0047] For example, the human detection unit 19 may calculate the reliability of the detection that the moving object is a human as the human detection. The machine learning model may be configured to output the reliability of the human detection instead of the judgment result of the human detection. The human detection unit 19 may be configured to detect the moving object as a human if the reliability is equal to or greater than the human detection threshold. The human detection unit 19 may change the human detection threshold depending on the distance from the boarding gate 2a to the target group in the image frame F. The target group is a group of moving object blocks Bm that make up the moving object that is the target of human detection.

[0048] FIG. 9(a) is a graph showing an example of the change in the human detection threshold depending on the distance from the boarding gate 2a to the target group. The vertical axis is the human detection threshold. The horizontal axis is the distance from the boarding gate 2a to the target group. Here, the distance d1 from the boarding gate 2a to the target area of ​​the moving object m1 is treated as the distance from the boarding gate 2a to the target group of the moving object m1 (the group of moving object blocks Bm that make up the moving object m1). The same applies to the distance d2 of the moving object m2.

[0049] As shown in FIG. 9(a), the human detection unit 19 decreases the human detection threshold as the distance from the boarding gate 2a to the target group increases. Since distance d1 is smaller than distance d2, when the human detection threshold for moving body m1 is set to 90, the human detection unit 19 sets the human detection threshold for moving body m2 to 60 to determine whether to detect a person. The human detection unit 19 does not need to detect people riding bicycles or the like as people. The human detection unit 19 may detect people in wheelchairs as people.

[0050] When the person detection unit 19 detects that the moving object is a person, the boarding intention determination unit 20 determines whether or not there is a person making a boarding intention gesture toward the bus 2.

[0051] The intention-to-board gesture is a predetermined movement that indicates an intention to board the bus 2. The intention-to-board gesture can be, for example, a movement of raising a hand while approaching the bus 2. The intention-to-board gesture can be a movement of waving a hand while approaching the bus 2. The intention-to-board determination unit 20 may determine that the gesture is an intention-to-board gesture when, in addition to a hand movement, there is a movement of opening the mouth to convey an intention to board. The intention-to-board determination unit 20 may also determine that the movement of running toward the boarding gate 2a is an intention-to-board gesture.

[0052] The boarding intention determination unit 20 determines whether or not a person is making a boarding intention gesture from the time-series data of the captured image, using, for example, a machine learning model. The machine learning model is trained to input the time-series data of the captured image and output a determination result as to whether or not a person is making a boarding intention gesture. Note that the motion vector V for each moving body block Bm constituting the moving body m1 determined to be a person may also be added as an input. Also, the output result of another machine learning model (such as a machine learning model for human detection) may also be used as an input.

[0053] The boarding intention determination unit 20 may determine whether or not a person is making a boarding intention gesture without using a machine learning model. The boarding intention determination unit 20 may determine the boarding intention gesture using at least one of pattern matching, silhouette analysis, optical flow analysis, HOG (Histogram Of Gradient) analysis, etc. The boarding intention determination unit 20 may employ a well-known gesture determination method.

[0054] The boarding intention determination unit 20 may calculate the reliability of the determination that a person is making a boarding intention gesture. The machine learning model may be configured to output the reliability of the boarding intention gesture instead of the determination result of the boarding intention gesture. The boarding intention determination unit 20 may be configured to determine that a person is making a boarding intention gesture when the reliability is equal to or greater than a gesture determination threshold. The boarding intention determination unit 20 may change the gesture determination threshold depending on the distance from the boarding entrance 2a to the target group in the image frame F.

[0055] FIG. 9(b) is a graph showing an example of changes in the strength of the gesture discrimination threshold according to the distance from the boarding gate to the target group. The vertical axis is the gesture discrimination threshold. The horizontal axis is the distance from the boarding gate 2a to the target group. As shown in FIG. 9(b), the boarding intention determination unit 20 reduces the gesture discrimination threshold as the distance from the boarding gate 2a to the target group increases. Since distance d1 is less than distance d2, when the gesture discrimination threshold for moving body m1 is set to 90, the boarding intention determination unit 20 performs gesture discrimination by setting the gesture discrimination threshold for moving body m2 to 60.

[0056] The door control unit 21 controls the opening and closing of the door 25 at the boarding entrance 2a of the bus 2. When the driver performs an operation to open or close the door, the door control unit 21 transmits a control signal to the door driving unit 5 to execute the door opening control or the door closing control.

[0057] When the boarding intention determination unit 20 determines that a person is making a gesture of intent to board the bus 2 during execution of the door closing control, the door control unit 21 suspends the door closing control and opens the doors 25. In this case, the door control unit 21 may make an announcement through the announcement unit 6 inside the bus and near the boarding entrance that the door closing control will be suspended because a person intending to board has been detected.

[0058] Next, a control method of the vehicle control system according to this embodiment will be described with reference to the drawings. Fig. 10 is a flowchart showing an example of the control method of the vehicle control system. The flowchart shown in Fig. 10 is an example of a process that is started when control to close the doors 25 of the bus 2 is executed.

[0059] In FIG. 10, the ECU 10 first acquires an image captured by the outside camera 3 (step S101). The captured image is acquired as time-series data. As shown in FIG. 3(a), the ECU 10 generates an image frame F by dividing the image captured by the outside camera 3 into a plurality of blocks B (step S102). The procedure is synonymous with the step.

[0060] 3A, the ECU 10 calculates a motion vector V for each block B in the image frame F (step S103). Specifically, the ECU 10 calculates the motion vector V for each block B based on the image frame F obtained at the current time t and the image frame F obtained at the previous time t-1.

[0061] Next, the ECU 10 calculates the magnitude of the component of the motion vector V in the entrance direction for each block B (step S104). In the image frame F shown in FIG. 3(a), the motion vector V of the four blocks B located in the upper left corner is not extracted because it does not point toward the entrance 2a of the bus 2. Next, the ECU 10 converts the magnitude of the motion vector V toward the entrance 2a of the bus 2 for each block B into a scalar value (step S105), as shown in FIG.

[0062] Next, the ECU 10 determines moving object blocks Bm by comparing the scalar value with a predetermined threshold for each block B (step S106). In the image frame F shown in Fig. 4, the threshold is 7 as an example, and blocks B with a scalar value of 7 or more are determined to be moving object blocks Bm. Each moving object block Bm contains a portion of a moving object.

[0063] Next, the ECU 10 removes the isolated moving object block Bm as noise (step S107). In the image frame F shown in Fig. 3(b), the single moving object block Bm1 has a motion vector V pointing toward the boarding gate 2a of the bus 2, but since it is not adjacent to any other moving object block Bm, the isolated moving object block Bm is removed as noise.

[0064] Next, the ECU 10 determines moving objects that are moving toward the boarding gate 2a of the bus 2 by grouping adjacent moving object blocks Bm that include the same moving object (step S108). In the image frame F shown in FIG. 3(b), two moving objects m1 and m2 are determined to be moving toward the boarding gate 2a of the bus 2. The moving objects m1 and m2 are included in moving object detection frames W1 and W2, each of which is made up of a plurality of moving object blocks Bm.

[0065] Next, the ECU 10 determines the ground contact position of the moving object (step S109). The ECU 10 identifies the moving object block Bm at the bottom side in the captured image for each group of moving object blocks Bm, and determines the grid number to which the bottom side of the moving object block Bm belongs, thereby determining the ground contact position (see FIG. 6(a)).

[0066] The ECU 10 corrects the scalar value based on the ground contact position of the moving body (step S110). The ECU 10 corrects the scalar value of each moving body block Bm that constitutes the moving body using a predetermined correction table (see FIG. 6(b)) based on the ground contact position of the moving body and the position of each moving body block Bm that constitutes the moving body. The ECU 10 performs the correction by multiplying the scalar value of each moving body block Bm by a correction coefficient obtained from the correction table.

[0067] The ECU 10 calculates the amount of movement for each moving body (step S111). As the amount of movement for each moving body, the ECU 10 calculates the average value of scalar values ​​for each group of moving body blocks Bm that make up the moving body.

[0068] The ECU 10 selects a moving object to be subjected to the super-resolution processing (step S112). The moving object selection unit 17 selects a moving object whose average value calculated in S111 is equal to or greater than a threshold value as a moving object to be subjected to the super-resolution processing.

[0069] The ECU 10 performs super-resolution processing on the selected moving object (step S113). The ECU 10 performs cropping and resizing within a rectangular frame circumscribing the moving object to determine a target area for super-resolution processing. The ECU 10 performs super-resolution processing on the target area, which involves at least one of noise removal, lens distortion correction, and upscaling.

[0070] The ECU 10 performs human detection on the moving object on which the super-resolution processing has been performed (step S114). For example, the ECU 10 calculates the reliability of the detection that the moving object is a human. The ECU 10 detects the moving object as a human if the reliability is equal to or greater than a human detection threshold (see FIG. 9(a)). The ECU 10 determines whether a human has been detected (step S114).

[0071] The ECU 10 determines whether a person is making a gesture of intent to get in (step S116). The ECU 10 determines whether a person is making a gesture of intent to get in from time-series data of the captured image using, for example, a machine learning model. The ECU 10 may calculate the reliability of the determination that a person is making a gesture of intent to get in. If the reliability is equal to or greater than a gesture determination threshold, the ECU 10 determines that a person is making a gesture of intent to get in (see FIG. 9(b)). The ECU 10 determines whether a person is making a gesture of intent to get in (step S117).

[0072] If it is determined in step S117 that a person is making a gesture indicating their intention to get in, the ECU 10 suspends the door closing control and opens the door 25 (step S118). The ECU 10 performs the door opening control by transmitting a control signal to the door driving unit 5. The ECU 10 may cause the announcement unit 6 to make an announcement that the door closing control will be suspended and the door 25 will be opened.

[0073] If no person is detected in step S114, or if it is not determined in step S117 that there is a person making a gesture indicating an intention to get in, the ECU 10 continues the door closing control and closes the door 25 (step S119).

[0074] According to the vehicle control system 1 and the control method thereof of this embodiment described above, when it is determined that a person is making a predetermined gesture of intent to board the bus 2, the closing control of the door 25 is interrupted and the door 25 is opened. Therefore, compared to when only the position of the person is focused on, it is possible to accurately determine the intent of a person near the boarding entrance 2a to board and open the door 25 that is currently under closing control.

[0075] Moreover, the vehicle control system 1 can appropriately determine whether or not a person is making a gesture of intent to get on using a machine learning model. Furthermore, since it is considered that a person intending to get on approaches the boarding gate or raises their hand, the vehicle control system 1 can appropriately determine whether or not a person is making a gesture of intent to get on by focusing on the magnitude of the gate direction component of the motion vector V for each block B.

[0076] Furthermore, according to the vehicle control system 1, the further away from the boarding gate 2a the person appears smaller in the captured image and the smaller the gesture movement appears. Therefore, a correction is made so that the magnitude (scalar value) of the boarding gate direction component of the motion vector V for each block B increases the farther away from the boarding gate 2a, and by performing a boarding intention gesture determination process on a group of moving body blocks Bm whose corrected scalar value is greater than or equal to a threshold value, it is possible to properly determine whether or not someone is making a boarding intention gesture.

[0077] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. The present invention can be embodied in various forms, including the above-described embodiments, with various modifications and improvements made based on the knowledge of those skilled in the art.

[0078] For example, the vehicle control system 1 may use a machine learning model to directly detect a person from the time-series data of the captured image.The vehicle control system 1 may use a machine learning model to directly determine whether a person is making a gesture of intent to get in from the time-series data of the captured image.

[0079] The vehicle control system 1 does not necessarily need to correct the scalar values. The vehicle control system 1 may select a moving object using the scalar values ​​without correction. In this case, the vehicle control system 1 does not need to determine the ground position of the moving object. When the vehicle control system 1 performs correction, the correction table is not limited to that shown in FIG. 6(b). The correction table may be set so that the correction coefficient increases with increasing distance from the boarding gate 2a. Furthermore, super-resolution processing is not essential. [Explanation of symbols]

[0080] 1...vehicle control system, 2...bus (vehicle), 2a...boarding entrance, 3...exterior camera, 10...ECU, 11...feature calculation unit, 12...feature conversion unit, 13...moving object block extraction unit, 14...noise removal unit, 15...grouping unit, 16...correction unit, 17...moving object selection unit, 18...super-resolution processing unit, 19...person detection unit, 20...boarding intention determination unit, 21...door control unit, 25...door, B...block (see the upper left corner of the figure), V...motion vector.

Claims

1. A vehicle control system that controls opening and closing of a vehicle entrance door, an exterior camera that captures an image of the boarding area near the boarding entrance; a boarding intention determination unit that determines whether or not a person is making a predetermined boarding intention gesture toward the vehicle based on time-series data of images captured by the exterior camera when the door closing control is being executed; and a door control unit that, when the boarding intention determination unit determines that a person is making the boarding intention gesture, suspends the door closing control and opens the door; A vehicle control system comprising:

2. 2. The vehicle control system of claim 1, wherein the boarding intention determination unit determines whether or not a person is making the boarding intention gesture from the time series data of the captured image using a machine learning model that inputs time series data of the captured image and outputs a determination result as to whether or not a person is making the boarding intention gesture.

3. a feature amount calculation unit that divides the captured image into a plurality of blocks and calculates a motion vector for each of the blocks based on time-series data of the captured image; the feature amount calculation unit calculates a magnitude of a component of the motion vector in the entrance direction for each of the blocks; The vehicle control system according to claim 1 , wherein the boarding intention determination unit determines whether or not a person is making the boarding intention gesture by using a magnitude of the boarding entrance direction component of the motion vector for each block.

4. a correction unit that performs a correction such that the magnitude of the entrance direction component of the motion vector for each of the blocks increases as the position of the block in the captured image is farther from the entrance, The vehicle control system of claim 3, wherein the boarding intention determination unit determines whether or not a person is making the boarding intention gesture by performing a predetermined boarding intention gesture determination process on a group of blocks in which the magnitude of the corrected boarding entrance direction component is equal to or greater than a threshold value.

5. A control method for a vehicle control system that controls opening and closing of a vehicle entrance door, comprising: determining whether or not a person is making a predetermined gesture of intent to get in the vehicle based on time series data of an image captured by an exterior camera that captures an image of a boarding area near the boarding entrance when the door closing control is being executed; When it is determined that a person is making the gesture of intending to get in, interrupting the door closing control and opening the door; A control method for a vehicle control system, comprising:

Images