Action recognition method and vehicle control device
A dual-machine learning model approach simplifies passenger state assessment by separating seated and secondary task determinations, enhancing accuracy and reducing computational load in vehicle passenger recognition systems.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2024-12-05
- Publication Date
- 2026-06-17
AI Technical Summary
Existing systems complicate the processing by attempting to grasp all variations in passenger position and posture, necessitating a simpler method to determine passenger state.
A behavior recognition method using two distinct machine learning models to determine a passenger's first and second seating states, allowing separate determination of seated and secondary tasks, respectively, thereby simplifying the passenger state assessment.
The method allows for easy and accurate determination of passenger states, reducing computational complexity and improving accuracy in passenger status recognition.
Smart Images
Figure 2026098389000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a behavior recognition method and a vehicle control device.
Background Art
[0002] As described in Patent Document 1, there is known a device that acquires the position and posture of a passenger in a vehicle using an in-vehicle sensor and gives a notification regarding the safety of the passenger based on the riding state of the passenger.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When grasping the state of a passenger by covering all variations in the position and posture of the passenger, the processing becomes complicated. It is required to simply grasp the state of the passenger.
[0005] In view of such circumstances, an object of the present disclosure is to simply grasp the state of a passenger.
Means for Solving the Problems
[0006] A behavior recognition method according to an embodiment of the present disclosure includes detecting a passenger in a vehicle, determining whether the passenger is in a first seating state using a first machine learning model, and when it is determined that the passenger is in the first seating state, determining a second seating state of the passenger using a second machine learning model different from the first machine learning model.
[0007] A vehicle control device according to one embodiment of the present disclosure comprises a recognition unit and a determination unit for recognizing passengers in a vehicle. The recognition unit detects the passenger and determines whether the passenger is in a first seated state using a first machine learning model. If it is determined that the passenger is in the first seated state, it determines whether the passenger is in a second seated state using a second machine learning model different from the first machine learning model. The determination unit determines whether the vehicle can depart based on whether the passenger is in the second seated state. [Effects of the Invention]
[0008] The behavior recognition method and vehicle control device disclosed herein can easily grasp the status of passengers. [Brief explanation of the drawing]
[0009] [Figure 1] This is a block diagram showing an example configuration of a vehicle control device related to this disclosure. [Figure 2] This figure shows an example of a target behavior recognized by the behavior detector and behavior discriminator. [Figure 3] This flowchart shows an example of the procedure for the behavior recognition method related to this disclosure. [Figure 4] This figure shows an example of the correspondence between bus seating arrangements and secondary tasks. [Modes for carrying out the invention]
[0010] (Example of the configuration of the control device 1) As shown in Figure 1, a control device 1 according to one embodiment of the present disclosure comprises a recognition unit 2 and a determination unit 6. The recognition unit 2 comprises a person detector 3, an action detector 4, and an action discriminator 5. The control device 1 is a device that controls a vehicle and is also referred to as a vehicle control device. The recognition unit 2 is connected to a camera 7 or a sensor 8. The camera 7 is mounted on the vehicle and captures images of the position and posture of passengers inside the vehicle. The sensor 8 is mounted on the vehicle and detects the position and posture of passengers inside the vehicle.
[0011] The recognition unit 2 and the decision unit 6 may be configured to include one or more processors or dedicated circuits. The processor may include a CPU or GPU, etc. The dedicated circuit may include an FPGA or ASIC, etc. The processors of the recognition unit 2 or the decision unit 6 may be configured separately or as a single unit. The processors of the person detector 3, behavior detector 4, or behavior discriminator 5 of the recognition unit 2 may be configured separately or as a single unit.
[0012] The control device 1 may include a storage unit. The storage unit stores any information or programs used in the operation of the control device 1. The storage unit may include, but is not limited to, semiconductor memory, magnetic memory, or optical memory. The storage unit may function as, for example, a main memory, auxiliary memory, or cache memory. The storage unit may include an electromagnetic storage medium such as a magnetic disk. The storage unit may include a non-temporary computer-readable medium. The storage unit may be included in the recognition unit 2 or the determination unit 6.
[0013] The control device 1 may include a communication device for internal or external communication. The communication device may support at least one of various wired or wireless communication standards.
[0014] The control device 1 may include an operating unit. The operating unit receives operation input from the control device 1. The operating unit may include a touch panel, a touch sensor, or physical keys. The operating unit may include a display. The display may be a touch panel. The operating unit may include a camera or a microphone, etc. The operating unit may include various other input devices. The control device 1 may include a display device such as a liquid crystal display.
[0015] (Operation Example of Control Device 1) The recognition unit 2 of the control device 1 detects the position and posture of the passengers inside the vehicle based on the captured image of the camera 7 or the detection result of the sensor 8 by the person detector 3. The person detector 3 may detect the position and posture of the passengers using a machine learning model. The machine learning model may be configured to output the position and posture of the passengers when the captured image of the camera 7 or the detection result of the sensor 8 is input. The machine learning model may be a model such as YOLO (You Only Look Once) or CNN (Convolutional Neural Network).
[0016] The recognition unit 2 recognizes the actions of the passengers using the action detector 4 and the action discriminator 5. The actions of the passengers include whether the passengers are in a seated state or a standing state.
[0017] The action detector 4 and the action discriminator 5 may recognize the actions of the passengers using a machine learning model. The machine learning model may be configured to output the actions of the passengers when the position and posture of the passengers are input. The machine learning model used by the action detector 4 is also referred to as the first machine learning model. The machine learning model used by the action discriminator 5 is also referred to as the second machine learning model.
[0018] The first machine learning model and the second machine learning model are different models. The first machine learning model and the second machine learning model may be generated as different models by using different learning data for models with the same algorithm. The first machine learning model and the second machine learning model may be models with different algorithms.
[0019] The determination unit 6 of the control device 1 determines whether the vehicle can start based on the recognition result of the actions of the passengers in the vehicle by the recognition unit 2. Hereinafter, the operation example of the control device 1 will be described.
[0020] The action detector 4 of the recognition unit 2 may recognize whether the passengers are in a seated state or a standing state based on the position and posture of the passengers illustrated in FIG. 2. The seated state of the passengers recognized by the action detector 4 is also referred to as the first seated state.
[0021] State (1) is a state where the passenger stands away from the seat. When the action detector 4 determines that the state of a person is (1), it detects that the person is in a state of leaving the seat.
[0022] State (2) is a state where the passenger stands beside the seat. When the action detector 4 determines that the state of a person is (2), it detects that the person is in a state of leaving the seat.
[0023] State (3) is a state where the passenger tries to sit on the seat but has a high waist. When the action detector 4 determines that the state of a person is (3), it detects that the person is in a state of leaving the seat.
[0024] State (4) is a state where the passenger tries to sit on the seat and the waist is slightly lifted from the seat. When the action detector 4 determines that the state of a person is (4), it detects that the person is either in a state of leaving the seat or in a state of sitting. Whether to detect state (4) as either a state of leaving the seat or a state of sitting may be adjusted by the first machine learning model used in the action detector 4. For example, by generating the first machine learning model using learning data corresponding to state (4) being either a state of leaving the seat or a state of sitting, whether to detect state (4) as either a state of leaving the seat or a state of sitting may be adjusted.
[0025] State (5) is a state where the passenger sits on the seat. When the action detector 4 determines that the state of a person is (5), it detects that the person is in a state of sitting.
[0026] The state of the passenger may transition from state (1) to state (5). Also, the state of the passenger may transition from state (5) to state (1).
[0027] The behavior discriminator 5 of the recognition unit 2 determines whether a passenger is performing a second task for passengers who have been detected as being seated by the behavior detector 4. The second task includes taking a posture different from the normal seating posture while seated. The normal seating posture is defined as sitting upright with the back straight in the seat. The second task may include sitting at an angle, leaning against a nearby handrail or window, leaning back, lying down, or raising one's hand. The seating state in which a second task is being performed, as determined by the behavior discriminator 5, is also referred to as the second seating state.
[0028] The recognition unit 2 outputs the recognition results of the passengers' behavior by the behavior detector 4 and the behavior discriminator 5 to the decision unit 6. The decision unit 6 determines whether the passengers in the vehicle are sitting in their seats in a safe posture. If multiple passengers are detected inside the vehicle, the decision unit 6 may determine whether all passengers are sitting in their seats in a safe posture. If the decision unit 6 determines that all passengers are sitting in their seats in a safe posture, it may determine that it is possible to depart the vehicle.
[0029] The recognition unit 2 may execute the behavior recognition method illustrated in Figure 3. The behavior recognition method may be implemented as a behavior recognition program to be executed by the processor of the recognition unit 2. The behavior recognition program may be stored on a non-temporary computer-readable medium.
[0030] The person detector 3 detects passengers in the vehicle, as well as their position and posture, from the image captured by the camera 7 or the detection result of the sensor 8 (S1). The behavior detector 4 determines whether the seated flag corresponding to the detected passenger is 1 (S2). The seated flag is set to 1 when the behavior detector 4 determines that the passenger is seated, and to 0 when the behavior detector 4 determines that the passenger is vacant. The numerical value of the flag corresponding to the seated or vacant state of the passenger is not limited to the example above. The seated flag may also represent whether the passenger is seated or vacant in a manner other than numerical value, such as a string.
[0031] If the seating flag of the detected passenger is not 1 (S2:NO), the behavior detector 4 determines whether the passenger is seated based on the detected passenger's position and posture (S3). If the behavior detector 4 does not determine that the passenger is seated (S3:NO), the recognition unit 2 returns to the passenger detection procedure of S1 and performs the detection of other passengers and recognition of their actions.
[0032] If the behavior detector 4 determines that a passenger is seated (S3: YES), it sets the seated flag for that passenger to 1 (S4). After executing the procedure in S4, the recognition unit 2 returns to the passenger detection procedure in S1 and performs the detection of other passengers and recognition of their behavior.
[0033] If the seating flag of a passenger detected by the behavior detector 4 is 1 (S2: YES), the behavior discriminator 5 selects a discriminator corresponding to the seat position where the detected passenger is sitting (S5). The behavior discriminator 5 may have multiple types of discriminators corresponding to the characteristics of the seat position. The behavior discriminator 5 may have different types of discriminators for each seat. The type of discriminator may be distinguished by the model used in that discriminator. If the behavior discriminator 5 uses multiple different models for each seat as the second machine learning model, it may have a discriminator corresponding to each model. The behavior discriminator 5 may use the selected discriminator to determine the status of the detected passenger's second task.
[0034] The behavior discriminator 5 may include discriminators corresponding to the combinations of second tasks expected at each seat 11. For example, as shown in Figure 4, when the vehicle 10 has multiple seats 11, some of the multiple actions included in the second task may be expected at all seats 11. Some of the multiple actions included in the second task may be expected at some seats 11. Conversely, the second task may include actions that are not expected at each seat 11. The discriminator may be configured to discriminate actions expected at a particular seat 11 and ignore actions that are not expected.
[0035] In the example in Figure 4, the second tasks of "sitting diagonally," "lying down," or "raising hands" are expected to be performed in any of the seats 11. The second task of "leaning back" is expected in seats 11 of seat numbers #4, #5, and #6 due to the shape of the seats, but not in seats 11 of seat numbers #1, #2, and #3. The second task of "leaning against the handrail" is expected in seats 11 of seat numbers #3, #4, and #6 where a handrail 13 is nearby, but not in seats 11 of seat numbers #1, #2, and #5 where a handrail 13 is not present. The second task of "leaning against the window" is expected in seats 11 of seat numbers #4 and #6 which face the window 12, but not in seats 11 of seat numbers #1, #2, #3, and #5 which do not face the window 12.
[0036] The behavior discriminator 5 may include a discriminator that uses a model corresponding to the combination of secondary tasks expected for each seat 11. The secondary tasks expected for seats 11 with seat numbers #1 and #2 are the same. Therefore, the discriminator applied to passengers seated in seats 11 with seat numbers #1 and #2 is the same. In the example in Figure 4, discriminator B1 is assigned.
[0037] Furthermore, the expected second task for seat 11 with seat numbers #4 and #6 is the same. Therefore, the classifier applied to passengers seated in seat 11 with seat numbers #4 and #6 is the same. In the example in Figure 4, classifier B2 is assigned.
[0038] The expected secondary tasks for seat 11 with seat numbers #3 and #5 are different. Therefore, different classifiers are applied to the passengers seated in seat 11 with seat numbers #3 and #5. In the example in Figure 4, classifier B3 is assigned to seat 11 with seat number #3, and classifier B4 is assigned to seat 11 with seat number #5.
[0039] Let's say there are two seats, the first and second seats, that have different expected combinations of second tasks. The second machine learning model used to determine the status of the second tasks may include a third machine learning model that determines the status of the expected second tasks in the first seat and a fourth machine learning model that determines the status of the expected second tasks in the second seat. The third and fourth machine learning models may be different models.
[0040] The behavior classifier 5 selects a classifier corresponding to the seat number, i.e., the seat location, to determine the status of the second task performed by each passenger seated in each seat 11. By selecting a classifier according to the seat location, second tasks not expected at each seat 11 are ignored, and the accuracy of determining the expected second tasks at each seat 11 can be improved. In addition, the load on the determination process is reduced. As a result, the status of the passenger's second task can be easily determined.
[0041] Returning to Figure 3, the behavior discriminator 5 determines whether the passenger's performance on the second task, as detected in step S1, is an action similar to sitting down (S6).
[0042] If the behavior classifier 5 determines that the passenger's second task is an action similar to sitting (S6: YES), it performs action-specific processing (S7). For example, the behavior classifier 5 may perform processing to further determine the passenger's actions.
[0043] If the behavior discriminator 5 determines that the passenger's second task is not an action similar to sitting down (S6: NO), it sets the passenger's seated flag to 0 (S8).
[0044] After executing the procedure in S7 or S8, the recognition unit 2 returns to the passenger detection procedure in S1 and performs the detection of other passengers and recognition of their actions.
[0045] As described above, the recognition unit 2 recognizes the actions of the passengers in the vehicle by executing the steps in the flowchart of Figure 3. The decision unit 6 of the control device 1 may refer to the recognition result by the recognition unit 2 and determine that the vehicle can depart when the seated flag for all passengers in the vehicle becomes 1.
[0046] The determination unit 6 may determine that the vehicle can depart if it recognizes that the posture of all passengers is stable, regardless of the seated flag.
[0047] The determination unit 6 may determine whether the vehicle can depart based on the passengers' second seating position.
[0048] (Summary) As described above, in the control device 1 according to this disclosure, the model for detecting whether a passenger in the vehicle is seated and the model for determining the behavior of a seated passenger are separated. In other words, the determination models for the first seated state and the second seated state are separated. Separating the models improves the accuracy of the determination or reduces the load on the determination process.
[0049] The determination of the second seating state is performed on passengers who have been determined to be in the first seating state. In this way, it becomes unnecessary to determine the second seating state when determining the first seating state. As a result, the expansion of the model is suppressed and the computational complexity of the model is reduced. As a result, the control device 1 and behavior recognition method according to this disclosure allow the status of passengers in a vehicle to be easily grasped.
[0050] While embodiments relating to this disclosure have been described based on the drawings and examples, those skilled in the art can make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of this disclosure. [Explanation of Symbols]
[0051] 1: Control device, 2: Recognition unit, 3: Human detector, 4: Behavior detector, 5: Behavior discriminator, 6: Decision unit, 7: Camera, 7: Sensor, 10: Vehicle, 11: Seat, 12: Window, 13: Handrail
Claims
1. Detecting passengers in the vehicle, The first machine learning model determines whether the passenger is in the first seated position, If it is determined that the passenger is in the first seated state, the second seated state of the passenger is determined by a second machine learning model different from the first machine learning model, Behavior recognition method.
2. The aforementioned vehicle is equipped with a first seat and a second seat, The second machine learning model includes a third machine learning model for determining the second seating state of the passenger in the first seat, and a fourth machine learning model different from the third machine learning model for determining the second seating state of the passenger in the second seat. The behavior recognition method according to claim 1.
3. It comprises a recognition unit that recognizes passengers in the vehicle and a decision unit, The aforementioned recognition unit, The passenger is detected and the first machine learning model determines whether the passenger is in the first seated position. If it is determined that the passenger is in the first seated state, the passenger's second seated state is determined by a second machine learning model different from the first machine learning model. The determination unit determines whether the vehicle can depart based on whether the passenger is in the second seated position.