Behavior recognition method and vehicle control device

Abstract

The present application provides a behavior recognition method and a vehicle control device capable of easily grasping the state of a passenger. The behavior recognition method includes the steps of: detecting a passenger of a vehicle; determining whether the passenger is in a first seated state using a first machine learning model; and in the case where it is determined that the passenger is in the first seated state, determining a second seated state of the passenger using a second machine learning model different from the first machine learning model.

Description

Technical Field

[0001] This invention relates to a behavior recognition method and a vehicle control device. Background Technology

[0002] As described in Patent Document 1, there is a known device that uses in-vehicle sensors to obtain the position and posture of passengers inside the vehicle and provides safety-related notifications based on the passengers' riding status.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2016-062414 Summary of the Invention

[0004] The process becomes complex when considering changes in passenger position and posture to understand their state. A simpler way to grasp passenger status is required.

[0005] In view of this situation, the object of the present invention is to easily grasp the status of passengers.

[0006] An embodiment of the present invention relates to a behavior recognition method comprising the following steps: detecting passengers in a vehicle; determining whether the passenger is in a first seated state using a first machine learning model; and if the passenger is determined to be in the first seated state, determining the passenger's second seated state using a second machine learning model different from the first machine learning model.

[0007] An embodiment of the present invention relates to a vehicle control device comprising an identification unit and a determination unit for identifying passengers in a vehicle. The identification unit performs the following processing: detecting the passenger and determining whether the passenger is in a first seated state using a first machine learning model; and if the passenger is determined to be in the first seated state, determining a second seated state for the passenger using a second machine learning model different from the first machine learning model. The determination unit determines whether the vehicle can depart based on whether the passenger is in the second seated state.

[0008] Invention Effects

[0009] The behavior recognition method and vehicle control device of the present invention can easily grasp the status of passengers. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating a structural example of the vehicle control device according to the present invention.

[0011] Figure 2 This is a diagram illustrating an example of a target behavior identified by a behavior detector and a behavior discriminator.

[0012] Figure 3This is a flowchart illustrating an example of the steps involved in the behavior recognition method of the present invention.

[0013] Figure 4 This is a diagram illustrating an example of the correspondence between the seating arrangement of a bus and the second task. Detailed Implementation

[0014] (Example of the structure of control device 1) Figure 1 As shown, a control device 1 according to one embodiment of the present invention includes an identification unit 2 and a judgment unit 6. The identification unit 2 includes a human detector 3, a behavior detector 4, and a behavior discriminator 5. The control device 1 is a device for controlling a vehicle, also referred to as a vehicle control device. The identification unit 2 is connected to a camera 7 or a sensor 8. The camera 7 is mounted on the vehicle and captures images of the position and posture of passengers inside the vehicle. The sensor 8 is mounted on the vehicle and detects the position and posture of passengers inside the vehicle.

[0015] The identification unit 2 and the judgment unit 6 may be configured with one or more processors or dedicated circuits. The processor may be configured with a CPU or GPU, etc. The dedicated circuit may be configured with an FPGA or ASIC, etc. The processors of the identification unit 2 or the judgment unit 6 may be configured independently or as a single unit. The processors of the human detector 3, behavior detector 4, or behavior discriminator 5 of the identification unit 2 may be configured independently or as a single unit.

[0016] The control device 1 may include a storage unit. The storage unit stores any information or program used to control the operation of the control device 1. The storage unit may be configured as, for example, a semiconductor memory, magnetic memory, or optical memory, but is not limited to these. The storage unit may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit may be configured using an electromagnetic storage medium such as a magnetic disk. The storage unit may be configured using a non-transitory computer-readable medium. The storage unit may be included in the identification unit 2 or the determination unit 6.

[0017] The control device 1 may include a communication device for internal or external communication. The communication device may correspond to at least one of various wired or wireless communication standards.

[0018] The control device 1 may include an operation unit. The operation unit receives operation input from the control device 1. The operation unit may be configured as a touch panel, a touch sensor, or physical keys. The operation unit may include a display. The display may be included in the touch panel. The operation unit may be configured as a camera or microphone, etc. The operation unit may include various other input devices. The control device 1 may include a display device such as a liquid crystal display (LCD).

[0019] (Example of Control Device 1's Operation) The recognition unit 2 of control device 1 detects the position and posture of passengers inside the vehicle using a human detector 3 based on the image captured by camera 7 or the detection result of sensor 8. The human detector 3 can use a machine learning model to detect the passenger's position and posture. The machine learning model can be configured to output the passenger's position and posture when the image captured by camera 7 or the detection result of sensor 8 is input. The machine learning model can be a You Only Look Once (YOLO) model or a Convolutional Neural Network (CNN) model, etc.

[0020] The recognition unit 2 identifies passenger behavior using a behavior detector 4 and a behavior discriminator 5. Passenger behavior includes whether the passenger is seated or not.

[0021] Behavior detector 4 and behavior discriminator 5 can use machine learning models to identify passenger behavior. The machine learning model can be configured to output the passenger's behavior in response to input passenger position and posture. The machine learning model used by behavior detector 4 is also referred to as machine learning model 1. The machine learning model used by behavior discriminator 5 is also referred to as machine learning model 2.

[0022] Machine learning model 1 and machine learning model 2 are different models. Machine learning model 1 and machine learning model 2 can be generated as different models using different learning data, but they are models using the same algorithm. Machine learning model 1 and machine learning model 2 can be models using different algorithms.

[0023] The judgment unit 6 of the control device 1 determines whether the vehicle can depart based on the recognition result of the recognition unit 2 on the behavior of the passengers in the vehicle. An example of the operation of the control device 1 will be described below.

[0024] The behavior detector 4 of the recognition unit 2 can be based on Figure 2 The system uses the passenger's position and posture to identify whether the passenger is seated or not. The seated state of the passenger identified by the behavior detector 4 is also called the first seated state.

[0025] State (1) is the state in which the passenger leaves the seat and stands. When a person's state is (1), the behavior detector 4 detects that the person is in the state of leaving the seat.

[0026] State (2) is the state where the passenger is standing next to the seat. When the person's state is (2), the behavior detector 4 detects that the person is in a state of leaving the seat.

[0027] State (3) is the state in which the passenger wants to sit in the seat but is at a high waist. When the person is in state (3), the behavior detector 4 detects that the person is in a state of leaving the seat.

[0028] State (4) is the state in which the passenger intends to sit in the seat and their waist is slightly raised off the seat. When the person's state is (4), the behavior detector 4 detects that the person is either in an off-seat state or a seated state. Which state (4) is detected as, the off-seat state or the seated state, can be adjusted by the first machine learning model used in the behavior detector 4. For example, the first machine learning model can be generated using learning data that establishes a correspondence between state (4) and either the off-seat state or the seated state, thereby adjusting which state (4) is detected as, the off-seat state or the seated state.

[0029] State (5) is the state where the passenger is sitting in the seat. When the person's state is (5), the behavior detector 4 detects that the person is in a seated state.

[0030] The passenger's state can transition from state (1) to state (5). Furthermore, the passenger's state can transition from state (5) to state (1).

[0031] The behavior discriminator 5 of the recognition unit 2 determines the status of a second task performed by a passenger who is detected as seated by the behavior detector 4. The second task includes adopting a posture different from the usual seating posture. The usual seating posture is sitting upright with the back extended. The second task may include sitting at an angle, leaning against an armrest or window near the seat, reclining, sleeping, or raising one's hands, etc. The seating state of performing the second task as determined by the behavior discriminator 5 is also called the second seating state.

[0032] The recognition unit 2 outputs the recognition results of the passenger's behavior based on the behavior detector 4 and the behavior discriminator 5 to the determination unit 6. The determination unit 6 determines whether the passengers in the vehicle are sitting in a safe posture. If multiple passengers are detected inside the vehicle, the determination unit 6 can determine whether all passengers are sitting in a safe posture. If the determination unit 6 determines that all passengers are sitting in a safe posture, it can determine that the vehicle can depart.

[0033] Identification unit 2 can perform Figure 3 The behavior recognition method illustrated herein can be implemented as a behavior recognition program executed by the processor of the recognition unit 2. The behavior recognition program can be stored in a non-transitory computer-readable medium.

[0034] The human detector 3 detects the passengers in the vehicle and their positions and postures based on the images captured by the camera 7 or the detection results from the sensor 8 (S1). The behavior detector 4 determines whether the seating flag corresponding to the detected passenger is 1 (S2). The seating flag is set to 1 when the behavior detector 4 determines that the passenger is seated and set to 0 when it determines that the passenger is not seated. The value of the flag corresponding to the passenger's seated or unseated state is not limited to the example above. The seating flag can be used in a way other than a numerical value, such as using a string to indicate whether the passenger is seated or unseated.

[0035] If the passenger's seating indicator detected by behavior detector 4 is not 1 (S2: No), the system determines whether the passenger is seated based on the detected passenger's position and posture (S3). If behavior detector 4 does not determine that the passenger is seated (S3: No), the recognition unit 2 returns to the passenger detection step in S1 and performs the detection and behavior recognition of other passengers.

[0036] If the passenger is determined to be seated (S3: Yes), the behavior detector 4 sets the passenger's seated flag to 1 (S4). After executing step S4, the identification unit 2 returns to the passenger detection step in S1 to perform the detection of other passengers and the identification of their behaviors.

[0037] If the passenger's seating indicator is 1 as detected by behavior detector 4 (S2: Yes), behavior discriminator 5 selects a discriminator corresponding to the seat position of the detected passenger (S5). Behavior discriminator 5 can have multiple discriminators corresponding to the characteristics of the seat position. Behavior discriminator 5 can have discriminators of different types for each seat. The type of discriminator can be distinguished according to the model used in that discriminator. If behavior discriminator 5 uses multiple models different for each seat as a second machine learning model, it can have discriminators corresponding to each model. Behavior discriminator 5 can use the selected discriminator to determine the implementation status of the detected passenger's second task.

[0038] The behavior discriminator 5 may have a discriminator corresponding to a combination of second tasks conceived at the positions of each seat 11. For example, such as Figure 4 As shown, when the vehicle 10 has multiple seats 11, some of the multiple actions included in the second task can be conceived on all seats 11. Conversely, some of the multiple actions included in the second task can be conceived on a subset of seats 11. In other words, the second task can include actions not conceived on any single seat 11. The discriminator can be configured to identify actions conceived on a specific seat 11 and ignore unconceived actions.

[0039] exist Figure 4In the example, the actions of "sitting diagonally," "sleeping," or "raising hands" in the second task are conceived on any seat 11. Due to the shape of the seats, "leaning back" in the second task is conceived on seats 11 numbered #4, #5, and #6, but not on seats 11 numbered #1, #2, and #3. "Leaning against the armrest" in the second task is conceived on seats 11 numbered #3, #4, and #6 where armrests 13 are nearby, but not on seats 11 numbered #1, #2, and #5 where armrests 13 are not present. "Leaning against the window" in the second task is conceived on seats 11 numbered #4 and #6 facing the window 12, but not on seats 11 numbered #1, #2, #3, and #5 not facing the window 12.

[0040] The behavior discriminator 5 can be equipped with a discriminator that uses a model corresponding to the combination of second tasks conceived in each seat 11. The second tasks conceived in seats 11 with seat numbers #1 and #2 are the same. Therefore, the discriminator applicable to passengers seated in seats 11 with seat numbers #1 and #2 is generalized. Figure 4 In the example, discriminator B1 is assigned.

[0041] Furthermore, the second task envisioned for seats 11 with seat numbers #4 and #6 is the same. Therefore, the discriminator applicable to passengers seated in seats 11 with seat numbers #4 and #6 is universalized. Figure 4 In the example, discriminator B2 is assigned.

[0042] The second tasks envisioned for seats 11 in seats #3 and #5 are different. Therefore, different discriminators are applied to the passengers seated in seats 11 in seats #3 and #5 respectively. Figure 4 In the example, discriminator B3 is assigned to seat 11 with seat number #3, and discriminator B4 is assigned to seat 11 with seat number #5.

[0043] Two seats, representing different combinations of the envisioned second task, are referred to as the first seat and the second seat. The second machine learning model used to determine the implementation status of the second task may include a third machine learning model determining the implementation status of the envisioned second task on the first seat and a fourth machine learning model determining the implementation status of the envisioned second task on the second seat. The third and fourth machine learning models can be different models.

[0044] Behavior discriminator 5 selects a discriminator corresponding to the seat number, i.e., the seat position, to determine the implementation status of the second task of the passenger seated on each seat 11. By selecting a discriminator based on the seat position, unplanned second tasks on each seat 11 can be ignored, improving the discrimination accuracy of planned second tasks on each seat 11. Furthermore, the processing load is reduced. As a result, the implementation status of the passenger's second task can be determined easily.

[0045] Return to Figure 3 The behavior discriminator 5 determines whether the passenger's second task implementation status detected in step S1 is a behavior similar to sitting down (S6).

[0046] If the passenger's second task is determined to be a behavior similar to sitting down (S6: Yes), the behavior discriminator 5 performs behavior classification processing (S7). For example, the behavior discriminator 5 can perform further detailed processing to determine the passenger's behavior.

[0047] If the passenger's second task is determined to be not a behavior similar to sitting down (S6: No), the behavior discriminator 5 sets the passenger's sitting flag to 0 (S8).

[0048] After performing step S7 or S8, identification unit 2 returns to the passenger detection step of S1 to perform the detection of other passengers and the identification of their behavior.

[0049] As described above, the identification unit 2 performs... Figure 3 The process involves identifying the behavior of passengers in the vehicle using the steps outlined in the flowchart. The judgment unit 6 of the control device 1 can refer to the identification results from the recognition unit 2 and determine that the vehicle is ready to depart if the seating indicator for all passengers in the vehicle is set to 1.

[0050] The judgment unit 6 can determine that the vehicle can depart if all passengers are identified as being in stable postures, regardless of the seating signs.

[0051] The judgment unit 6 can determine whether the vehicle can depart based on the second seating status of the passengers.

[0052] (Summary) As described above, in the control device 1 of this invention, the model for detecting whether a passenger in a vehicle is seated and the model for determining the behavior of a seated passenger are separated. In other words, the determination models for the first seated state and the second seated state are separated. By separating the models, the determination accuracy of the models is improved or the load of the determination processing is reduced.

[0053] The determination of the second seating state is performed on passengers who are determined to be in the first seating state. Thus, the determination of the first seating state does not require a determination of the second seating state. As a result, model amplification is suppressed and the computational load of the model is reduced. With these results, the status of passengers in a vehicle can be easily grasped according to the control device 1 and behavior recognition method of the present invention.

[0054] The embodiments of the present invention have been described based on the figures and examples; however, those skilled in the art can make various modifications and alterations to the present invention. Therefore, it should be noted that these modifications and alterations are included within the scope of the present invention.

[0055] Symbol Explanation

[0056] 1-Control device, 2-Identification unit, 3-Person detector, 4-Behavior detector, 5-Behavior discriminator, 6-Judgment unit, 7-Camera, 8-Sensor, 10-Vehicle, 11-Seat, 12-Window, 13-Handrail.

Claims

1. A behavior recognition method, characterized in that, Includes the following steps: Passengers in the vehicle were inspected; The first machine learning model is used to determine whether the passenger is in the first seated state; and If the passenger is determined to be in the first seated state, a second machine learning model, which is different from the first machine learning model, is used to determine the passenger's second seated state.

2. The behavior recognition method according to claim 1, characterized in that, The vehicle is equipped with a first seat and a second seat. The second machine learning model includes: a third machine learning model that determines the second seating state of the passenger in the first seat; and a fourth machine learning model that determines the second seating state of the passenger in the second seat and is different from the third machine learning model.

3. A vehicle control device, characterized in that, have: The vehicle's passenger identification and judgment unit. The identification unit performs the following processing: The passenger is detected and the first machine learning model is used to determine whether the passenger is in the first seated state; If the passenger is determined to be in the first seated state, a second machine learning model, which is different from the first machine learning model, is used to determine the passenger's second seated state. and The determination unit determines whether the vehicle can depart based on whether the passenger is in the second seated state.

Images