A passenger state monitoring method, device, terminal equipment and vehicle

Abstract

This application relates to the field of intelligent vehicle technology, providing a passenger status monitoring method, device, terminal equipment, and vehicle. The method includes: acquiring user information within the vehicle's cabin when the vehicle is powered on; determining, based on the user information, whether a driver and a target passenger are present in the cabin; wherein the target passenger is a passenger whose seat is in the rear seat of the vehicle and whose age is within a preset age range; and, when both the driver and the target passenger are present in the cabin, sending real-time monitoring video of the target passenger to a remote device for remote passenger status monitoring. This method enables efficient and convenient passenger status monitoring of target passengers in the vehicle.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle technology, and in particular to a passenger status monitoring method, device, terminal equipment, and vehicle. Background Technology

[0002] With the improvement of people's living standards and the rapid development of intelligent vehicle technology, driving has become one of the most common ways for people to travel. When a driver is driving, the state of other passengers in the car cabin may affect the driver's driving state; for example, when there are children in the car cabin, their emotional instability, crying, and dangerous actions will all affect the driver's driving state.

[0003] In the current technical solution, when a child's condition affects driving safety, the driver manually turns on the monitoring device and establishes a communication connection with the remote device to send the monitoring video to the remote device. The remote device can then monitor the condition of the child passenger in the vehicle or intervene with the user. However, this technical solution is complicated to operate.

[0004] Therefore, how to efficiently and conveniently monitor passenger status is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a passenger status monitoring method, device, terminal equipment, vehicle, and computer-readable storage medium, which are designed to efficiently and conveniently monitor passenger status.

[0006] Firstly, this application provides a passenger status monitoring method. The method includes:

[0007] When the vehicle under monitoring is powered on, acquire user information from inside the vehicle's cabin.

[0008] Based on the user information, determine whether there is a driver and a target passenger in the vehicle cabin; wherein, the target passenger is a passenger whose seat is in the back seat of the vehicle to be monitored and whose age is within a preset age range;

[0009] When the driver and the target passenger are in the vehicle cabin, real-time monitoring video of the target passenger is sent to a remote device so that the target passenger's status can be monitored using the remote device.

[0010] In one embodiment, when the driver and the target passenger are present in the vehicle cabin, sending real-time monitoring video of the target passenger to a remote device to monitor the passenger status using the remote device includes:

[0011] When the driver and the target passenger are present in the vehicle cabin, determine the number of passengers of the target passenger.

[0012] If the number of passengers is within a preset range, real-time monitoring videos corresponding to each target passenger are sent to a remote device, and each target passenger is displayed in a split screen on the remote device according to the number of passengers, so as to monitor the passenger status of each target passenger using the remote device.

[0013] In one embodiment, the step of sending real-time monitoring videos corresponding to each of the target passengers to a remote device, and displaying each target passenger in a split-screen format on the remote device according to the number of passengers, so as to monitor the passenger status of each target passenger using the remote device, includes:

[0014] If the number of passengers is within a preset range, target images corresponding to each target passenger are extracted from the real-time monitoring video;

[0015] Send target images corresponding to each of the target passengers to a remote device, and display each of the target images in a split-screen format on the remote device;

[0016] Determine whether the center point of the face frame in two adjacent target images of the same target passenger has shifted according to a preset time interval;

[0017] If displacement occurs, the target image of the target passenger is redefined, and the updated target image is sent to the remote device so that the remote device can be used to monitor the passenger status of each target passenger.

[0018] In one embodiment, the passenger status monitoring method further includes:

[0019] If the number of passengers is not within the preset range, the captured image is segmented according to the display ratio of the remote device to obtain the real-time monitoring video.

[0020] In one embodiment, the user information includes image information and voiceprint information of the vehicle to be monitored; determining whether a driver and target passenger are present in the vehicle cabin based on the user information includes:

[0021] Determine whether there is a driver and rear seat passengers in the vehicle cabin based on the image information;

[0022] The image information and the voiceprint information are used to determine whether the passenger in the back seat is the target passenger.

[0023] In one embodiment, determining whether a driver and rear-seat passengers exist in the vehicle cabin based on the image information includes:

[0024] When the target to be detected is within the shooting range of the image acquisition device, image information of the target to be detected is acquired;

[0025] The image information is used for facial recognition to determine whether there is a driver and rear seat passengers in the car cabin.

[0026] In one embodiment, when the driver and the target passenger are present in the vehicle cabin, a real-time monitoring video of the target passenger is sent to a remote device to monitor the passenger's status using the remote device. The passenger status monitoring method further includes:

[0027] Acquire the driver's hand gestures and / or voice prompts;

[0028] The remote device is operated and controlled according to the gestures and / or voice prompts; the operation control includes bringing up or hiding the real-time monitoring video.

[0029] Secondly, this application also provides a passenger status monitoring device. The device includes:

[0030] The acquisition module is used to acquire user information inside the cabin of the vehicle under monitoring when the vehicle under monitoring is powered on.

[0031] The identification module is used to determine whether there is a driver and a target passenger in the car cabin based on the user information; wherein, the target passenger is a passenger whose seat is in the rear seat of the vehicle to be monitored and whose age is within a preset age range;

[0032] The monitoring module is used to send real-time monitoring video of the target passenger to a remote device when the driver and the target passenger are in the vehicle cabin, so as to monitor the passenger status of the target passenger using the remote device.

[0033] Thirdly, this application also provides a terminal device. The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described above.

[0034] Fourthly, this application also provides a vehicle, including a vehicle body and a controller, the controller performing the steps of the above method.

[0035] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described above.

[0036] This application provides a passenger status monitoring method, which actively sends real-time monitoring videos of the target passengers to a remote device when the presence of the driver and rear seat passengers in the cabin of the vehicle to be monitored is detected. That is, real-time monitoring videos can be sent to remote devices without relying on manual operation by the driver. Therefore, this method can efficiently and conveniently monitor the passenger status of the target passengers in the vehicle to be monitored.

[0037] It is understood that the passenger status monitoring device, terminal equipment, vehicle, and computer-readable storage medium provided in the embodiments of this application have the same beneficial effects as the passenger status monitoring method described above, and will not be repeated here. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a passenger status monitoring method provided in this application embodiment;

[0040] Figure 2 This is a schematic diagram of a three-dimensional coordinate system constructed with the tip of the nose of the target being detected as the origin in an embodiment of this application;

[0041] Figure 3 A flowchart illustrating another passenger status monitoring method provided in this application embodiment;

[0042] Figure 4 The diagram shown is a structural schematic of a passenger status monitoring device provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0044] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0045] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0046] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0047] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0048] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0049] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. "A plurality" means "two or more."

[0050] The passenger status monitoring method provided in this application embodiment can be executed by the processor of the terminal device when running the corresponding computer program.

[0051] Figure 1 The flowchart illustrates a passenger status monitoring method provided in this application embodiment. For ease of explanation, only the parts relevant to this embodiment are shown. The method provided in this embodiment includes the following steps:

[0052] S100: When the vehicle under monitoring is powered on, acquire user information inside the vehicle's cabin.

[0053] The vehicle to be monitored refers to the vehicle for which passenger status monitoring is required. It can be a fuel-powered vehicle or an electric vehicle. This embodiment does not limit the specific type of vehicle to be monitored.

[0054] The term "power-on state" for the vehicle under monitoring refers to the state after the vehicle is ignited, including the state of preparing to start driving or the state of being in motion. More specifically, for gasoline-powered vehicles, the power-on state is the state of the vehicle after the user unlocks the door and swipes their card; for electric vehicles, the power-on state is the state after the vehicle has undergone a soft start. In one specific embodiment, each time the vehicle is powered on, if it is in P (Parking) gear, user information inside the vehicle's cabin is acquired.

[0055] Among them, the vehicle cabin refers to the driving and riding space in the vehicle to be monitored; user information includes information corresponding to the driver and passengers in the vehicle cabin, specifically including whether there is a main driver, the location of the passengers in the vehicle to be monitored, and the number of passengers.

[0056] In practice, image acquisition equipment pre-installed in the vehicle to be monitored can be used to acquire image information inside the car cabin, and facial recognition can be performed on the image information. The user information can be determined by combining the facial recognition results, the installation location of the image acquisition equipment, and the seating position of the user corresponding to the identified face.

[0057] S200: Determine whether there is a driver and a target passenger in the vehicle cabin based on user information; the target passenger is a passenger whose seat is in the back seat of the vehicle to be monitored and whose age is within a preset age range.

[0058] In this step, the identified user information is analyzed to determine whether the driver and target passenger are present in the passenger compartment of the vehicle to be monitored. Specifically, determining the presence of the driver and target passenger includes at least two scenarios: one is whether only the driver and target passenger are present in the passenger compartment, i.e., the driver is present in the driver's seat, there is no passenger in the front passenger seat, and there are passengers in the rear seats whose ages fall within a preset age range; the other is whether the passenger compartment includes the driver and target passenger, i.e., there are other passengers besides the driver and target passenger, such as passengers in the front passenger seat.

[0059] It should be noted that this embodiment does not limit the preset age range of the target passenger; it can be set according to actual needs. For example, the preset age range can represent the age range of a child or the age range of an elderly person. In a specific embodiment, based on a smart care monitoring scenario for child users, the preset age range is set to represent the age range of a child, and the specific value range of the preset age range can be set according to needs; this embodiment does not limit this. More specifically, if the age range of a child is defined as less than or equal to 12 years old, and the age range of an adult is defined as greater than 12 years old, then the preset age range corresponding to the target passenger is less than or equal to 12 years old. In actual operation, an error range can also be set; that is, when the identified passenger's age is within the error range of the preset age range, the passenger's age is determined to be within the preset age range; the error range can be the preset age range ±3 years.

[0060] S300: When the driver and the target passenger are present in the vehicle cabin, the system sends real-time monitoring video of the target passenger to a remote device so that the target passenger's status can be monitored remotely.

[0061] In this step, once the presence of the driver and the target passenger in the vehicle's cabin is confirmed, a remote device is used to monitor the target passenger's status. Specifically, it determines whether the remote device has already retrieved live video feed, i.e., whether the remote device's intelligent rear-seat monitoring is enabled. If live video feed is not retrieved (intelligent rear-seat monitoring is disabled), the live video feed for the target passenger is determined, the intelligent rear-seat monitoring on the remote device is enabled, and the live video feed for the target passenger is sent to the remote device. If live video feed is already retrieved (intelligent rear-seat monitoring is enabled), the live video feed is not sent to the remote device again. After receiving the live video feed, the remote device displays it through an application or in a widget location. Other users can then monitor the status of the target passenger in the vehicle being monitored through the live video feed displayed on the remote device.

[0062] It should be noted that in actual operation, the vehicle to be monitored can also be remotely operated through remote devices. For example, when an abnormal emotional state is detected in a target passenger, the smart device of the vehicle to be monitored can be controlled to play music or other types of audio, or a voice call can be established with the vehicle to be monitored to soothe the target passenger's emotions. Abnormal emotions include crying, irritability, and fear.

[0063] This application provides a passenger status monitoring method. When the presence of a target passenger, including the driver and a passenger in the rear seat, is detected in the cabin of the vehicle to be monitored, the method actively sends real-time monitoring video of the target passenger to a remote device. This method can send real-time monitoring video to a remote device without relying on the driver's manual operation. Therefore, this method can efficiently and conveniently monitor the passenger status of the target passenger in the vehicle to be monitored.

[0064] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, when the driver and the target passenger are present in the car cabin, real-time monitoring video of the target passenger is sent to a remote device so as to monitor the passenger status of the target passenger using the remote device, including:

[0065] When the main driver and the target passenger are present in the car cabin, determine the number of passengers in the target passenger's cabin;

[0066] If the number of passengers is within the preset range, real-time monitoring videos corresponding to each target passenger are sent to the remote device so that the remote device can monitor the passenger status of each target passenger.

[0067] The preset number range is generally within the limit of the number of passengers in the rear seats, and can be set according to actual needs. When the driver and the target passenger are present in the car cabin, facial recognition is further performed on the image information, and the number of passengers of the target passenger is determined based on the number of recognized faces and the number of identical faces.

[0068] Then, it is determined whether the number of passengers is within a preset range. If the number of passengers is within the preset range, real-time monitoring videos corresponding to each target passenger are sent to the remote device. The remote device then displays each target passenger in a split-screen format based on the passenger count. In other words, for each target passenger, the remote device displays the corresponding real-time monitoring video in a split-screen format. Other users can monitor the passenger status of each target passenger by viewing the real-time monitoring videos displayed on the remote device.

[0069] In one specific embodiment, if the target passenger is a child passenger in the back seat of the vehicle to be monitored, when two children are detected in the back seat, the display area of ​​the remote device is divided in two. The center point of the child's face frame is output through the perception algorithm. The back seat shooting screen is cut according to the ratio after the horizontal division of the remote device (based on the UI design), with the center point of each child's face frame as the center. Then, the cut back seat shooting screen is scaled proportionally to fit the display area corresponding to each split screen of the remote device.

[0070] According to the method of this embodiment, a corresponding real-time monitoring video can be displayed individually for each target passenger, avoiding interference from other factors and improving the effectiveness of passenger status monitoring.

[0071] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, if the number of passengers is within a preset range, real-time monitoring videos corresponding to each target passenger are sent to a remote device, and each target passenger is displayed in a split-screen format on the remote device according to the number of passengers, so as to monitor the passenger status of each target passenger using the remote device, including:

[0072] If the number of passengers is within the preset range, extract the target image corresponding to each target passenger from the real-time monitoring video;

[0073] Send the target images corresponding to each target passenger to the remote device, and display each target image in a split screen on the remote device;

[0074] Determine whether the center point of the face bounding box in two adjacent target images of the same target passenger has shifted according to a preset time interval;

[0075] If displacement occurs, the target image of the target passenger is redefined, and the updated target image is sent to the remote device so that the remote device can monitor the passenger status of each target passenger.

[0076] Specifically, after acquiring the real-time monitoring video, if the number of passengers is within a preset range, the target frame image in the real-time monitoring video is determined, each target passenger in the target frame image is identified, and the target image corresponding to each target passenger is extracted from the target frame image; the target image corresponding to each target passenger is sent to the remote device, and each target image is displayed in split screen on the remote device.

[0077] Then, according to a preset time interval, the next target frame image in the real-time monitoring video is determined, and each target passenger in the next target frame image is identified to obtain the next target image. For each target passenger, it is determined whether the position of the center point of the face bounding box in the target image and the next target image has shifted; that is, whether the deviation value of the center point position of the face bounding box in the target image and the next target image exceeds the deviation threshold. If a shift occurs (exceeding the deviation threshold), the next target image is used as the updated target image for the target passenger, and the updated target image is sent to the remote device so that the remote device can monitor the passenger status of each target passenger. In this embodiment, the specific value of the preset time interval is not limited, and the preset time interval can be 5 seconds.

[0078] In other words, this embodiment determines whether the center point of the face frame corresponding to each target passenger has shifted at preset time intervals, and updates the target image corresponding to the target passenger displayed on the remote device when a shift is determined. Therefore, it can update the status of the target passenger in a timely manner while ensuring the stability of the display on the remote device, thereby improving the accuracy of passenger status monitoring.

[0079] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, a passenger status monitoring method further includes:

[0080] If the number of passengers is not within the preset range, the captured image will be segmented according to the display ratio of the remote device to obtain real-time monitoring video.

[0081] Specifically, in this embodiment, when it is determined that the number of target passengers is not within the preset range, the rear-seat camera footage is directly segmented to obtain a real-time monitoring video that can adapt to the display ratio of the remote device.

[0082] More specifically, when the number of target passengers exceeds the upper limit of the preset number range, the display ratio of the remote device and the center point of the rear-row shooting screen can be determined first. Then, the shooting screen is segmented according to the display ratio of the remote device, with the center point of the rear-row shooting screen as the center, to obtain real-time monitoring video.

[0083] Alternatively, when the number of passengers in the target area is 1, the display ratio of the remote device and the center point of the child's face frame can be determined first. Then, the captured image can be segmented according to the display ratio of the remote device, with the center point of the child's face frame as the center, to obtain real-time monitoring video.

[0084] In one specific embodiment, if the target passenger is a child passenger in the back seat of the vehicle to be monitored, assuming a preset number range of 2-3, when only one child is detected in the back seat, the center point of the child's face frame is output through a perception algorithm. Using this center point as the center, the back seat image is cropped according to the display ratio of the remote device (based on UI design), and the cropped image is scaled proportionally to fit the display area of ​​the remote device. When three or more children are detected in the back seat, the back seat image is cropped according to the display ratio of the remote device, using the center point as the center, and the cropped image is displayed on the remote device, i.e., the real-time monitoring video is displayed.

[0085] As can be seen, the method of this embodiment can display the status of each target passenger completely and comprehensively, thereby improving the comprehensiveness and accuracy of passenger status monitoring.

[0086] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, the user information includes image information and voiceprint information inside the vehicle to be monitored; determining whether a driver and target passenger exist in the vehicle cabin based on the user information includes:

[0087] Determine whether there is a driver and rear-seat passengers in the car cabin based on the image information;

[0088] The system uses image and voiceprint information to determine whether the rear-seat passengers are the target passengers.

[0089] In this embodiment, the user information includes image information and voiceprint information inside the vehicle to be monitored; the image information refers to the information inside the car cabin obtained by the image acquisition device, and the voiceprint information refers to the information inside the car cabin obtained by the sound acquisition device.

[0090] Specifically, after acquiring user information from inside the car cabin, facial recognition is first performed on the image information. User information is determined based on the facial recognition results, the installation location of the image acquisition device, and the seating position of the user corresponding to the identified face. More specifically, a first imaging device can be used to acquire image information corresponding to the driver's seat and the front passenger seat. If facial information is identified in this image information, the seating position of the identified face and the installation location of the first imaging device are used to determine whether there is a driver in the driver's seat and whether there is a passenger in the front passenger seat. A second imaging device is used to acquire image information corresponding to the rear seats. If facial information is identified in this image information, it is determined that there are passengers in the rear seats of the car cabin.

[0091] After identifying the passengers in the back seats, age recognition is performed based on the corresponding image and voiceprint information to determine whether the age of the passengers in the back seats is within the preset age range, that is, to determine whether the passengers in the back seats are the target passengers.

[0092] In this embodiment, the determination of whether the rear seat in the vehicle to be monitored is the target passenger is based on a combination of image information and voiceprint information. This can more accurately identify the target passenger, avoid misidentification of the target passenger, and thus improve the accuracy of passenger status monitoring.

[0093] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, determining whether there is a driver and rear seat passengers in the car cabin based on image information includes:

[0094] When the target to be detected is within the shooting range of the image acquisition device, image information of the target to be detected is acquired;

[0095] Facial recognition is performed on the image information to determine whether there is a driver and rear-seat passengers in the car cabin.

[0096] In practice, after the vehicle to be monitored is powered on, the image acquisition device determines whether the target is within its shooting range. If the target is within the shooting range, image information of the target is acquired, i.e., user information inside the vehicle cabin is obtained. If the target is not within the shooting range, the detection continues until the target is within the shooting range or a control command to end the detection is received. After acquiring the image information, facial recognition is performed to determine whether the driver and rear-seat passengers are present in the vehicle cabin.

[0097] In one specific embodiment, it is assumed that the target being detected is the main driver, located in the driver's seat, with eyes looking straight ahead, facing the image acquisition device, and the distance between the target's face and the image acquisition device is between 40-60cm; a three-dimensional coordinate system is constructed with the tip of the target's nose as the origin, as follows: Figure 2 As shown, the z-axis represents the line-of-sight direction. The recognition range of the image acquisition device is as follows:

[0098] x(PITCH): [-30°, 30°];

[0099] y(YAW): [-45°, 45°];

[0100] z(ROLL): [-30°, 30°].

[0101] In other words, when the distance between the target's face and the image acquisition device is within a specified range, and the target's face is within the recognition range of the image acquisition device, the image acquisition device acquires the image information of the target.

[0102] It should be noted that when collecting image information, the pixel count of the image information must be greater than or equal to a preset standard, such as 96px*96px, to ensure the accuracy of facial recognition based on the image information.

[0103] In addition, the facial recognition algorithm used in the process of facial recognition can support facial recognition under different lighting conditions, including daytime, nighttime, front lighting, side lighting, backlighting, alternating sunlight and shadow, and ambient lighting. The facial recognition algorithm can also support facial recognition when the face is occluded, including when it is covered by a mask, glasses, or sunglasses. Furthermore, the facial recognition algorithm can support facial recognition when there is makeup applied, including changes to the eyes, eyebrows, mouth, nose, and skin.

[0104] As can be seen, this embodiment acquires image information of the car cabin only when the target to be detected is within the shooting range of the image acquisition device. Therefore, the image information acquired according to the method of this embodiment is more accurate and can improve the accuracy of passenger status monitoring based on the image information.

[0105] Figure 3 A flowchart of another passenger status monitoring method provided in this application embodiment. Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, when the driver and the target passenger are present in the car cabin, real-time monitoring video of the target passenger is sent to a remote device so that the target passenger's status can be monitored using the remote device. The passenger status monitoring method further includes:

[0106] Acquire the driver's hand gestures and / or voice prompts;

[0107] Operators control remote devices based on gestures and / or voice prompts; operation controls include bringing up or hiding real-time monitoring video.

[0108] Specifically, when the vehicle under test is powered on, passenger identification is performed using the OMS (Occupancy Monitoring System). If the driver and the target passenger are present in the car cabin, it is determined whether the intelligent rear-seat monitoring of the remote device is enabled. If enabled, the driver's gestures and / or voice prompts are acquired. Based on the gestures and / or voice prompts, the corresponding operation instructions are determined and sent to the remote device. The remote device determines the corresponding operation control based on the received operation instructions, including displaying or hiding the real-time monitoring video. If the driver or the target passenger is not present in the car cabin, the process ends.

[0109] In one specific embodiment, the operation and control of a remote device based on gestures is as follows:

[0110] Scenario 1: When the "Intelligent Rear Seat Monitoring" switch is on, if it detects that "the palm is facing the driver's side window, the palm and fingers are in the same plane, the wrist is used as the rotation point to swing to the left, the left swing amplitude exceeds [-20°, 0°], and the right side fixed point duration is greater than or equal to 1 second", then a "screen recall" command will be generated to recall the real-time monitoring video of the remote device.

[0111] Scenario 2: When the remote device has real-time monitoring video enabled, if it detects that "the palm faces the driver's window, the palm and fingers are in the same plane, the hand swings to the right with the wrist as the rotation point, the right swing amplitude exceeds [0°, +20°], and the left fixed point duration is greater than or equal to 1 second", then a "screen collapse" command is generated to collapse the real-time monitoring video of the remote device.

[0112] As can be seen, the method according to this embodiment can operate and control the real-time monitoring video based on the driver's hand gestures or voice prompts, which can further improve the convenience of passenger status monitoring.

[0113] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0114] Figure 4 The diagram shown is a structural schematic of a passenger status monitoring device provided in an embodiment of this application. Figure 4 As shown, the passenger status monitoring device in this embodiment includes an acquisition module 410, an identification module 420, and a monitoring module 430; wherein,

[0115] The acquisition module 410 is used to acquire user information inside the cabin of the vehicle under monitoring when the vehicle under monitoring is powered on.

[0116] The identification module 420 is used to determine whether there is a driver and a target passenger in the car cabin based on user information; wherein, the target passenger is a passenger whose seat is in the back seat of the vehicle to be monitored and whose age is within a preset age range.

[0117] The monitoring module 430 is used to send real-time monitoring video of the target passenger to a remote device when the driver and the target passenger are present in the car cabin, so as to monitor the passenger status of the target passenger using the remote device.

[0118] The passenger status monitoring device provided in this application embodiment has the same beneficial effects as the passenger status monitoring method described above.

[0119] In one embodiment, the monitoring module 430 includes:

[0120] The quantity determination submodule is used to determine the number of passengers in the target passenger's cabin when the main driver and the target passenger are present.

[0121] The data transmission submodule is used to send real-time monitoring videos corresponding to each target passenger to the remote device if the number of passengers is within a preset range, and to display each target passenger in a split screen on the remote device according to the number of passengers, so as to monitor the passenger status of each target passenger using the remote device.

[0122] In one embodiment, the data transmission submodule includes:

[0123] The image determination unit is used to extract target images corresponding to each target passenger from the real-time monitoring video if the number of passengers is within a preset range.

[0124] The sending unit is used to send target images corresponding to each target passenger to the remote device and display each target image in a split screen on the remote device;

[0125] The judgment unit is used to determine whether the center point of the face frame in two adjacent target images of the same target passenger has shifted according to a preset time interval.

[0126] The update unit is used to redetermine the target image of the target passenger if displacement occurs, and send the updated target image to the remote device so that the remote device can monitor the passenger status of each target passenger.

[0127] In one embodiment, a passenger status monitoring device further includes:

[0128] The adjustment unit is used to cut the captured image according to the display ratio of the remote device if the number of passengers is not within the preset range, so as to obtain real-time monitoring video.

[0129] In one embodiment, the user information includes image information and voiceprint information inside the vehicle to be monitored; the recognition module 420 includes:

[0130] The first determination submodule is used to determine whether there is a driver and rear seat passengers in the car cabin based on image information;

[0131] The second determination submodule is used to determine whether the rear seat passenger is the target passenger based on image information and voiceprint information.

[0132] In one embodiment, the first determining submodule includes:

[0133] The acquisition unit is used to acquire image information of the target when the target is within the shooting range of the image acquisition device;

[0134] The recognition unit is used to perform facial recognition on image information to determine whether there is a driver and rear seat passengers in the car cabin.

[0135] In one embodiment, a passenger status monitoring device further includes:

[0136] The operation acquisition module is used to acquire the driver's hand gestures and / or voice prompts;

[0137] The control module is used to operate and control the remote device based on gestures and / or voice prompts; the operation control includes bringing up or hiding the real-time monitoring video.

[0138] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0140] Figure 5 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 5 As shown, the terminal device 500 of this embodiment includes a memory 501, a processor 502, and a computer program 503 stored in the memory 501 and executable on the processor 502; when the processor 502 executes the computer program 503, it implements the steps in the various passenger status monitoring method embodiments described above, for example... Figure 1 S100 to S300 are shown; or when the processor 502 executes the computer program 503, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 The functions of the acquisition module, identification module, and monitoring module are shown.

[0141] For example, computer program 503 can be divided into one or more modules / units, one or more of which are stored in memory 501 and executed by processor 502 to implement the method of the embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 503 in terminal device 500. For example, computer program 503 can be divided into an acquisition module, an identification module, and a monitoring module, with the specific functions of each module as follows:

[0142] The acquisition module is used to acquire user information inside the cabin of the vehicle under monitoring when the vehicle under monitoring is powered on.

[0143] The identification module is used to determine whether there is a driver and a target passenger in the car cabin based on user information; the target passenger is a passenger whose seat is in the back seat of the vehicle to be monitored and whose age is within a preset age range.

[0144] The monitoring module is used to send real-time monitoring video of the target passenger to a remote device when the driver and the target passenger are present in the car cabin, so as to monitor the passenger status of the target passenger using the remote device.

[0145] In applications, terminal device 500 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 500 may include, but is not limited to, memory 501 and processor 502. Those skilled in the art will understand that... Figure 5 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components. For example, a terminal device may also include input / output devices, network access devices, buses, etc.; among which, input / output devices may include cameras, audio acquisition / playback devices, displays, etc.; network access devices may include communication modules for wireless communication with external devices.

[0146] In applications, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0147] In applications, memory can be an internal storage unit of a terminal device, such as its hard drive or RAM; it can also be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card; or it can include both internal and external storage units. Memory is used to store operating systems, applications, boot loaders, data, and other programs, such as computer program code. Memory can also be used to temporarily store data that has been output or will be output.

[0148] This application also provides a vehicle, including a vehicle body and a controller, which executes the steps in the above-described method embodiments.

[0149] The vehicle provided in this application embodiment has the same beneficial effects as the passenger status monitoring method described above.

[0150] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0151] The computer-readable storage medium provided in this application embodiment has the same beneficial effects as the passenger status monitoring method described above.

[0152] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0154] Those skilled in the art will recognize that the device and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interface, or the device may be indirectly coupled or communicated, and may be electrical, mechanical, or other forms.

[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A passenger state monitoring method characterized by, The method includes: When the vehicle under monitoring is powered on, acquire user information from inside the vehicle's cabin. Based on the user information, determine whether there is a driver and a target passenger in the vehicle cabin; wherein, the target passenger is a passenger whose seat is in the back seat of the vehicle to be monitored and whose age is within a preset age range; When the driver and the target passenger are present in the vehicle cabin, real-time monitoring video of the target passenger is sent to a remote device to monitor the passenger status of the target passenger using the remote device. This includes: determining the number of passengers of the target passenger when the driver and the target passenger are present in the vehicle cabin; if the number of passengers is within a preset range, sending real-time monitoring video corresponding to each target passenger to the remote device, and displaying each target passenger in a split screen on the remote device according to the number of passengers; if the number of passengers is not within the preset range, first determining the display ratio of the remote device and the center point of the rear-seat camera image, using the center point of the rear-seat camera image as the center, segmenting the camera image according to the display ratio of the remote device to obtain the real-time monitoring video, and sending it to the remote device to monitor the passenger status of each target passenger using the remote device.

2. The method of claim 1, wherein, The step of sending real-time monitoring videos corresponding to each of the target passengers to a remote device, and displaying each target passenger in a split-screen format on the remote device according to the number of passengers, so as to monitor the passenger status of each target passenger using the remote device, includes: If the number of passengers is within a preset range, target images corresponding to each target passenger are extracted from the real-time monitoring video; Send target images corresponding to each of the target passengers to a remote device, and display each of the target images in a split-screen format on the remote device; Determine whether the center point of the face frame in two adjacent target images of the same target passenger has shifted according to a preset time interval; If displacement occurs, the target image of the target passenger is redefined, and the updated target image is sent to the remote device so that the remote device can be used to monitor the passenger status of each target passenger.

3. The method according to claim 1, characterized in that, The user information includes image information and voiceprint information of the vehicle to be monitored; determining whether a driver and target passenger are present in the vehicle cabin based on the user information includes: Determine whether there is a driver and rear seat passengers in the vehicle cabin based on the image information; The image information and the voiceprint information are used to determine whether the passenger in the back seat is the target passenger.

4. The method according to claim 3, characterized in that, Determining whether a driver and rear-seat passengers exist in the vehicle cabin based on the image information includes: When the target is within the shooting range of the imaging device, image information of the target is acquired; The image information is used for facial recognition to determine whether there is a driver and rear seat passengers in the car cabin.

5. The method according to any one of claims 1 to 4, characterized in that, When the driver and the target passenger are present in the vehicle cabin, the method further includes sending real-time monitoring video of the target passenger to a remote device to monitor the passenger's status using the remote device. Acquire the driver's hand gestures and / or voice prompts; The remote device is operated and controlled according to the gestures and / or voice prompts; the operation control includes bringing up or hiding the real-time monitoring video.

6. A passenger status monitoring device, characterized in that, The device includes: The acquisition module is used to acquire user information inside the cabin of the vehicle under monitoring when the vehicle under monitoring is powered on. The identification module is used to determine whether there is a driver and a target passenger in the car cabin based on the user information; wherein, the target passenger is a passenger whose seat is in the rear seat of the vehicle to be monitored and whose age is within a preset age range; The monitoring module is used to send real-time monitoring video of the target passenger to a remote device when the driver and the target passenger are in the vehicle cabin, so as to use the remote device to monitor the passenger status of the target passenger. The monitoring module includes: The quantity determination submodule is used to determine the number of passengers of the target passenger when the driver and the target passenger are present in the vehicle cabin; The data transmission submodule is used to send real-time monitoring videos corresponding to each of the target passengers to a remote device if the number of passengers is within a preset range, and to display each of the target passengers in a split screen on the remote device according to the number of passengers. The passenger status monitoring device also includes: The adjustment unit is used to determine the display ratio of the remote device and the center point of the rear-row shooting image if the number of passengers is not within the preset range. Then, the shooting image is segmented according to the display ratio of the remote device with the center point of the rear-row shooting image as the center to obtain the real-time monitoring video and send it to the remote device so as to use the remote device to monitor the passenger status of each target passenger.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A vehicle, comprising a vehicle body, characterized in that, It also includes a controller that performs the steps of the method as described in any one of claims 1 to 5.

Images