Riding safety control method and device, vehicle and storage medium

Abstract

The invention provides a riding safety control method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining a visual monitoring result for a riding object in a vehicle cabin through a visual sensor; monitoring vital signs in a vehicle cabin, and determining a life body monitoring result; according to the life body monitoring result and the visual monitoring result, a fusion monitoring result for the target riding object is determined; according to the fusion monitoring result and the vehicle riding scene, riding safety control is carried out, and efficient and accurate riding safety control can be realized.

Description

Technical Field

[0001] This application relates to the field of vehicle monitoring technology, and more specifically, to a passenger safety control method, device, vehicle, and storage medium. Background Technology

[0002] With the development of intelligent vehicles, passenger safety has become a core concern in vehicle safety rating systems.

[0003] Currently, in vehicle-riding scenarios, safety monitoring of passengers, including children, is necessary to implement passenger safety controls based on the monitoring results. For example, it's required to monitor for children who have been left behind or entered the vehicle on their own, and to alert children to ensure their safety within the cabin. Therefore, achieving efficient and accurate passenger safety control is the technical problem this application aims to solve. Summary of the Invention

[0004] This application provides a vehicle safety control method, device, vehicle, and storage medium. It can acquire visual monitoring results of passengers in the vehicle cabin using a visual sensor, and monitor vital signs within the vehicle cabin to determine the results of the vital sign monitoring. Then, based on the vital sign monitoring results and visual monitoring results, a fused monitoring result for the target passenger can be determined. Based on the fused monitoring result and the vehicle riding scenario, vehicle safety control can be implemented. Therefore, by fusing visual monitoring and vital sign monitoring results, dual verification and complementary perception of the status of target passengers such as children in the vehicle cabin can be achieved. This overcomes the limitations of single monitoring schemes in complex scenarios such as occlusion and low light, and improves the recall rate and recognition accuracy of detecting the presence of target passengers such as children in the vehicle cabin. Furthermore, it can be further combined with the vehicle riding scenario for automated vehicle safety control, thereby achieving efficient and accurate vehicle safety control and ensuring the safety of passengers in a real driving environment.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a vehicle passenger safety control method is provided, comprising: acquiring visual monitoring results of a passenger in a vehicle cabin using a visual sensor; monitoring vital signs in the vehicle cabin to determine the vital signs monitoring results; determining a fusion monitoring result for a target passenger based on the vital signs monitoring results and the visual monitoring results; and performing vehicle passenger safety control based on the fusion monitoring results and the vehicle riding scenario.

[0007] According to one aspect of this application, a passenger safety control device is provided, comprising: a visual monitoring module, used to: acquire visual monitoring results of a passenger in the vehicle cabin through a visual sensor; a vital signs monitoring module, used to: monitor vital signs in the vehicle cabin and determine the vital signs monitoring results; a fusion monitoring module, used to: determine fusion monitoring results for a target passenger based on the vital signs monitoring results and the visual monitoring results; and a passenger safety control module, used to: perform passenger safety control based on the fusion monitoring results and the vehicle riding scenario.

[0008] According to one aspect of this application, a vehicle / electronic device is provided, comprising: a processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the steps of the above-described passenger safety control method.

[0009] According to one aspect of this application, a chip is provided, comprising: a processor for calling and running a computer program from a memory, causing the processor to perform the steps of the above-described vehicle safety control method.

[0010] According to one aspect of this application, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the steps of the above-described passenger safety control method.

[0011] Based on the above technical solution, by fusing visual monitoring results of passengers in the vehicle cabin obtained from visual sensors and vital sign monitoring results obtained from monitoring vital signs in the vehicle cabin, a fused monitoring result for the target passenger can be obtained. Passenger safety control can then be performed based on the fused monitoring result. Therefore, it is possible to achieve dual verification and complementary perception of multiple types of monitoring results of the status of target passengers such as children in the vehicle cabin, overcoming the limitations of single monitoring schemes in monitoring and control under complex scenarios such as occlusion and low light. This improves the recall rate and recognition accuracy of the detection of the presence of target passengers such as children in the vehicle cabin. Furthermore, it can be further combined with the vehicle riding scenario for automated passenger safety control, thereby achieving efficient and accurate passenger safety control and ensuring the safety of passengers in the real driving environment.

[0012] Other features and advantages of the embodiments of this application will become apparent from the following detailed description, or may be learned in part by practice of this application.

[0013] It should be understood that the above general description and the following detailed description are merely exemplary and do not constitute a limitation on this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] Figure 1 This diagram illustrates an application scenario provided according to an embodiment of the present application. Figure 2 A flowchart illustrating a passenger safety control method according to an embodiment of this application is shown; Figure 3A A schematic diagram of a passenger safety control method according to an embodiment of this application is shown; Figure 3B A schematic diagram illustrating a passenger safety control method according to another embodiment of this application is shown; Figure 4 A block diagram of a passenger safety control device according to an embodiment of this application is shown; Figure 5 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present application is shown. Detailed Implementation

[0016] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more complete and to fully convey the concept of the exemplary embodiments to those skilled in the art. The accompanying drawings are schematic illustrations of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0017] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0018] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different networks, processor devices, or microcontroller devices.

[0019] Figure 1 This illustration shows an application scenario diagram according to an embodiment of the present application, such as... Figure 1 As shown, this application scenario includes terminal 110 and server 120.

[0020] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 110 can be a vehicle (specifically, an intelligent vehicle, or a controller within the vehicle, such as a vehicle cockpit domain controller or in-vehicle terminal controller), or it can be a smartphone, tablet, laptop, desktop computer, smart speaker, or smartwatch, but is not limited thereto. Terminal 110 and server 120 can be directly or indirectly connected via wired or wireless communication, and this application is not limited in this regard.

[0021] In some embodiments, the technical solution of this application is mainly applied to monitoring, alerting, and controlling scenarios related to passenger safety. For example, it can be applied to monitoring the presence of children in the vehicle cabin, the status of children while riding, and scenarios involving passenger safety alerts and control, but is not limited thereto. Furthermore, this application does not limit the type of vehicle.

[0022] In some embodiments, a visual sensor may be deployed on the terminal 110. Through the visual sensor, the terminal 110 can acquire visual monitoring results of passengers in the vehicle cabin. The terminal 110 can also monitor vital signs in the vehicle cabin and determine the vital signs monitoring results. Afterward, the terminal 110 can send the vital signs monitoring results and the visual monitoring results to the server 120. The server 120 can determine the fused monitoring results for the target passenger based on these results and perform passenger safety control based on the fused monitoring results and the vehicle riding scenario.

[0023] It is understood that the above application scenario is merely an example and does not constitute a limitation on the passenger safety control method provided in the embodiments of this application. For example, either the terminal 110 or the server 120 can independently execute the passenger safety control method provided in the embodiments of this application, or they can collaboratively execute the passenger safety control method provided in the embodiments of this application. This application does not impose any limitations on this.

[0024] The specific implementation process of the embodiments of this application will be described in detail below.

[0025] Figure 2 A flowchart illustrating a passenger safety control method according to an embodiment of this application is shown. This passenger safety control method can be executed by a device with computing power, such as the aforementioned terminal 110 or server 120. (Refer to...) Figure 2 As shown, this passenger safety control method may include at least S210 to S240, which are described in detail below: In S210, visual monitoring results of passengers in the vehicle cabin are acquired through a visual sensor.

[0026] The vehicle cabin can refer to the passenger space inside the vehicle.

[0027] For example, the vision sensor can be a camera, such as a 2D (two-dimensional) camera. Specifically, the vision sensor can be installed in the vehicle cabin, such as in the roof, the rearview mirror area, etc., but is not limited to these.

[0028] For example, a visual sensor can be used to perform imaging scans of the vehicle cabin (specifically, continuous or periodic scans) to determine the cabin image; each region in the cabin image can be segmented and classified to identify each passenger, thereby obtaining the visual monitoring results of the passenger.

[0029] The passenger can be a person with vital signs, such as an adult, a child, or a pet (e.g., a dog or a cat); or it can be a person without vital signs, such as an object or background area. The object can be a child safety seat (or child restraint system, CRS), a toy, etc., but is not limited to these.

[0030] For example, a pre-trained deep learning model can be run to segment and classify each region in the cabin image to identify each passenger. For instance, a deep learning model can be run by the vision system in the device implementing the technical solution of this application. The deep learning model may include, but is not limited to, a convolutional neural network-based object detection model and a semantic segmentation model.

[0031] For example, visual monitoring results may include: the occupants in each monitoring area within the vehicle cabin and their monitoring results. Specifically, the occupants may be in the form of labels, such as "adult," "child," "child safety seat," "unknown / background," etc. The monitoring results may include at least one of the following: location information of the occupants in the corresponding area (e.g., their position in the vehicle cabin, such as image coordinates corresponding to a cabin image, or spatial coordinates within the vehicle cabin converted from image coordinates using preset parameters of the visual sensor), occupant status information (e.g., posture information), monitoring bounding boxes of the occupants in the corresponding area (which may be displayed on the vehicle's screen to identify the occupant's position in the vehicle cabin / cabin image), and attribute information of the occupants in the corresponding area (e.g., the type of child safety seat: infant carrier, rear-facing child safety seat, or forward-facing child safety seat).

[0032] In the above embodiments, visual monitoring results containing classification labels of passengers, spatial location, attribute information, etc., can be generated by visual sensors deployed in vehicles, which can provide visual perception dimension information for subsequent fusion with vital sign monitoring results.

[0033] In S220, vital signs are monitored inside the vehicle cabin to determine the results of the vital signs monitoring.

[0034] For example, vital sign monitoring can be used to monitor vital signs within the vehicle cabin. This vital sign monitoring device can be radar (e.g., millimeter-wave radar, ultra-wideband radar, etc.), infrared thermal imaging equipment, but is not limited to these. Specifically, the vital sign monitoring device can be deployed in the vehicle, for example, in the roof, rearview mirror area, etc., to perform wide-area, blind-spot-free vital sign detection within the vehicle cabin. For instance, a millimeter-wave radar sensor deployed within the cabin can transmit and receive millimeter-wave signals to scan the cabin space and achieve vital sign monitoring.

[0035] For example, radar can be used to identify vital signs signals within the cockpit that correspond to life characteristics; the region corresponding to the vital signs signals can then be identified as the area containing the living organism. Specifically, a vital signs energy map can be generated based on the vital signs signals and their corresponding regions; the region containing the living organism can then be identified based on the vital signs energy map; and the monitoring results can be determined based on the region containing the living organism.

[0036] For example, signal processing algorithms, such as spectrum analysis and micro-motion feature extraction, can be used to identify micro-motion signals in the original radar signal that conform to life characteristics / signs (such as breathing and heartbeat). Specifically, these micro-motion signals can be periodic and have specific frequency ranges and amplitude characteristics. The corresponding spatial range can then be marked as the area where "life exists," i.e., the life zone.

[0037] For example, a vital signs energy map can be a two-dimensional or three-dimensional spatial energy distribution map generated based on radar signals. The energy intensity at different locations in the vital signs energy map can reflect the strength of vital sign signals in the corresponding areas. Specifically, the identified vital sign signals can be mapped onto the spatial coordinates of the corresponding vehicle cabin to generate a vital signs energy map, thereby intuitively representing the possibility of the presence of living beings / life forms in various areas of the vehicle cabin. Correspondingly, a living area can be a spatial range in the vital signs energy map where the energy intensity exceeds or continuously exceeds a preset threshold and the corresponding signal characteristics conform to the laws of vital signs / life forms (e.g., human life). This area can be marked as "where living beings exist".

[0038] For example, the monitoring results for living organisms can include the monitoring results of whether living organisms exist in each monitoring area. For example, it can include the correspondence between each monitoring area and a living organism area or a non-living organism area, where a living organism area refers to the area where living organisms exist, that is, the area that indicates the presence of living organisms in the corresponding monitoring area, and a non-living organism area refers to the area where living organisms do not exist, that is, the area that indicates the absence of living organisms in the corresponding monitoring area.

[0039] In the above process, the characteristics of millimeter-wave radar, which can penetrate obstructions and is unaffected by light, can be used to achieve non-contact and continuous monitoring of living beings in the vehicle cabin, and provide key evidence of the presence of life for subsequent fusion monitoring results.

[0040] In some embodiments, to improve the accuracy of vital sign monitoring results, the monitoring parameters of the vital sign monitor corresponding to the acquired vital sign monitoring results for monitoring vital signs inside the vehicle cabin can be adjusted based on the visual monitoring results. Specifically, this can be achieved through the following steps S220-1 to S220-3: S220-1, based on visual monitoring results, identifies the passengers in each monitoring area within the vehicle cabin.

[0041] For example, in conjunction with the embodiment in S210, the monitoring area can be a region obtained by spatially dividing the vehicle cabin (specifically, the seats in the cabin), such as the driver's seat area, the front passenger seat area, the left rear seat area, and the right rear seat area, but not limited to these. Correspondingly, the occupants in the area can be adults, children, rear-facing child safety seats, front-facing child safety seats, other objects, or empty spaces, etc.

[0042] For example, a vision sensor can continuously acquire cabin images of a vehicle's cockpit, process the cabin images by running a pre-trained deep learning model (e.g., an object detection and classification network), and output the bounding boxes of the identified targets (i.e., passengers in the area) and their category labels. Based on the position of the bounding boxes in the cabin images, they are mapped to the actual monitoring area of ​​the vehicle's cockpit through a pre-defined coordinate mapping relationship to obtain the category of passengers in each monitoring area.

[0043] S220-2, Adjust the monitoring parameters of the vital signs monitor according to the passengers in the area.

[0044] For example, monitoring parameters can refer to adjustable / configurable parameters that can affect the monitoring performance of a vital signs monitor, including but not limited to: detection sensitivity (used to determine the threshold for recognizing various weak signals), signal gain, beam scanning mode, and the signal frequency band of focus. Specifically, monitoring parameters include at least one of the following: monitoring thresholds used to determine the monitoring results of vital signs, and strategy parameters corresponding to the vital signs perception strategy.

[0045] For example, based on the category of passengers in the area, a corresponding parameter adjustment command can be sent to a vital signs monitor (e.g., radar). Specifically, this command can be sent to the control unit of the vital signs detector, such as the radar signal processing unit of the radar. The vital signs monitor adjusts its monitoring parameters according to the parameter adjustment command. The parameter adjustment command may include, but is not limited to, the adjusted monitoring parameters. Alternatively, the device implementing the technical solution of this application can directly adjust the monitoring parameters of the vital signs monitor.

[0046] For example, in response to the passenger being an adult, the sensitivity of the vital signs monitor for the corresponding monitoring area can be reduced. Specifically, if the vision system (i.e., the vision sensor) determines that the passenger is an adult, the vision system can instruct the radar to reduce the monitoring sensitivity for the monitoring area where the adult is located. This allows the radar to increase the signal strength threshold for identifying "vital signs" in that monitoring area. Alternatively, periodic micro-motion signals detected in that monitoring area that are consistent with human activity can be directly labeled as "adult activity," thus effectively reducing noise in adult areas, avoiding misinterpreting subtle adult movements as those of a child left behind, and reducing the system's false alarm rate.

[0047] For example, in response to the passenger being a rear-facing child safety seat, the sensitivity of the vital signs monitor for the monitoring area corresponding to the passenger can be increased and / or specific monitoring parameters can be adjusted. These specific monitoring parameters include parameters instructing the vital signs monitor to analyze specific frequency band signals corresponding to vital signs (e.g., the specific frequency band signal could be the frequency band signal corresponding to the child's breathing frequency range). Specifically, if the vision system, i.e., the vision sensor, determines that the passenger being a rear-facing child safety seat, the radar is instructed to increase the monitoring sensitivity for the monitoring area where the rear-facing child safety seat is located (e.g., lower the corresponding judgment threshold) and / or adjust the beam for focused scanning. Furthermore, monitoring parameters can be adjusted to instruct the vital signs monitor to focus on analyzing signals in specific frequency bands (e.g., focusing more on the child's breathing frequency range) to extract weak vital sign signals attenuated by the seat shell, achieving enhanced scanning for rear-facing child safety seats. This can improve the reliability of detecting children in rear-facing child safety seats and avoid riding hazards to children in rear-facing child safety seats in high-risk scenarios such as collisions.

[0048] For example, in response to whether the passenger in the area is wearing a forward-facing child safety seat or not wearing one, the monitoring parameters are adjusted to the parameters corresponding to the preset monitoring mode. If the vision system, i.e., the vision sensor, determines that the passenger in the area is wearing a forward-facing child safety seat or not wearing one, the radar is instructed to adjust the monitoring parameters in that monitoring area back to the parameters corresponding to the preset, balanced monitoring mode, and perform a regular scan to balance detection performance and system resources.

[0049] S220-3, based on the adjusted vital signs monitor, monitor vital signs in the vehicle cabin and determine the monitoring results.

[0050] In the above embodiments, the parameter settings of the vital signs monitor can be dynamically guided and optimized through visual recognition results, enabling the vital signs monitor's perception capability to adapt to different scenarios. This improves the detection recall rate while reducing the false alarm rate, achieving higher accuracy and reliability in passenger safety control.

[0051] It should be noted that this application does not restrict the execution order of each step. For example, S210 and S220 can be executed simultaneously, or S210 can be executed first and then S220, or S220 can be executed first and then S210.

[0052] In S230, the fusion monitoring results for the target passenger are determined based on the results of life monitoring and visual monitoring.

[0053] For example, the target passenger can be any passenger, such as a child, and this application does not limit this.

[0054] S230 can be implemented, for example, by any of the following methods or combinations thereof, but is not limited thereto: Method 1 involves supplementing visual monitoring results with biological monitoring results to obtain fused monitoring results.

[0055] Specifically, the target passenger is a passenger with vital signs; S230 includes any of the following situations: Scenario 1: In response to the visual monitoring results indicating that the passenger includes the target passenger, the type and location of the target passenger are determined to obtain the fused monitoring result. For example, when the visual monitoring results have already identified the target passenger (such as a child), the type and location of the target passenger can be directly determined based on the visual monitoring results to obtain the fused monitoring result. Alternatively, the type and location of the target passenger can also be determined based on the living organism monitoring results, but this is not limited to these methods.

[0056] Scenario 2: In response to visual monitoring results indicating the absence of a target passenger in the "living object area," the fused monitoring results are determined to include the presence of an obscured target passenger in the vehicle cabin. The "living object area" refers to the area where a living being is present in the living object monitoring results. For example, if the living object monitoring results show the presence of a living being in the monitored area (i.e., the "living object area"), but the visual monitoring results do not identify a target passenger in that area (e.g., the visual indication is that there is no child or only a child safety seat is identified but not associated with a child), it can be determined that an obscured target passenger exists in that monitored area (e.g., a child covered by a blanket), and this conclusion is established as the fused monitoring result.

[0057] Scenario 3: In response to visual monitoring results indicating the absence of a target passenger in a non-living area, the fused monitoring results are determined to include the absence of a target passenger in the vehicle cabin. Here, the non-living area refers to the area where no living organisms are detected in the living organism monitoring results. For example, if the living organism monitoring results show no living organisms in the monitored area (i.e., a "non-living area"), and the visual monitoring results also fail to identify a target passenger in that area, the system can determine that no target passenger exists in that monitored area, and the fused monitoring results will accordingly exclude the presence of a target passenger in that monitored area.

[0058] For example, such as Figure 3A As shown, in conjunction with the above embodiments, the visual sensor and millimeter-wave radar can simultaneously scan the vehicle cabin. The vision system can run a deep learning model to segment and classify each region in the cabin image acquired by the visual sensor, outputting labels such as "adult," "child," "child safety seat," or "unknown / background," obtaining the visual sensor's recognition result, i.e., the visual monitoring result. The millimeter-wave radar continuously scans the entire cabin space, acquiring corresponding radar signals and generating a vital signs energy map. The system's built-in fusion decision algorithm (e.g., based on DS evidence theory or Bayesian inference) correlates the outputs of the two sensors. Specifically, when the millimeter-wave radar detects vital signs in a certain area but the visual sensor does not identify adult features (e.g., the classification result is "unknown" or "child safety seat" but not associated with a child), the system determines that there is an occluded child in that area, thus compensating for missed detections in visual blind spots; if the visual sensor identifies a child and the radar confirms vital signs, the system has the highest confidence level. Thus, it can directly address extreme scenarios such as children being covered by blankets or placed in child safety seats, ensuring coverage of all possible child locations within the cabin.

[0059] In the above process, visual sensors have significant limitations in real cabin environments: children or children placed in child safety seats are easily ignored by algorithms due to their small size and low pixel ratio; at the same time, physical obstructions (such as blankets, sun visors, and the high side wings and shell of child safety seats) can create blind spots, leading to missed detections. To solve these problems, the above embodiments can introduce millimeter-wave radar as a complementary and verification source for visual perception. By utilizing its ability to penetrate non-metallic materials and detect weak vital signs, it can achieve wide-range, blind-spot-free monitoring of the presence of living beings in the cabin, making up for missed detections caused by small targets or blind spots, improving the recall rate of child detection, effectively improving the detection rate and reliability of child safety monitoring in the cabin, and ensuring that the presence of children can be reliably detected in any possible scenario where a child is left behind or enters the cabin, laying the foundation for subsequent safety warnings and proactive interventions.

[0060] Method 2: Generate visual monitoring results based on the life monitoring results, and obtain fused monitoring results based on the generated visual monitoring results and the life monitoring results.

[0061] Specifically, visual monitoring results can be generated based on the monitoring results of living organisms through the embodiments corresponding to S220-1 to S220-3. To avoid repetition, this application will not elaborate on this.

[0062] Specifically, the types of passengers in the visual monitoring results and the size and micro-motion characteristics of the target in the living organism monitoring results can be fused to determine the fused monitoring results.

[0063] For example, the type label and its corresponding first confidence level for each detected passenger in a monitoring area can be extracted from the visual monitoring results; the estimated area size (e.g., volume, height, area, or strength of ability) and micromotion features (e.g., whether the micromotion frequency is within a specified breathing / heartbeat range, the regularity of micromotion, etc., specifically features corresponding to vital signs) can be extracted from the living organism monitoring results. Then, a consistency correlation check is performed on the type label, estimated area size, and micromotion features for each monitoring area. The first confidence level is adjusted based on the consistency correlation check result to obtain a second confidence level; for example, if the type label is "child," but the estimated size is larger than the preset child size range, and the micromotion features are closer to those of an adult, the first confidence level can be reduced. Finally, the fused monitoring result can be determined based on the second confidence level. For example, the second confidence level can be compared with a preset confidence threshold: if the second confidence level is higher than the first confidence threshold, the type of the target passenger in the monitoring area is determined to be the type indicated by the type label; if the second confidence level is lower than the first confidence threshold but higher than the second confidence threshold, the fusion monitoring result can be re-determined, which can be determined according to other methods in the embodiments of this application; if the second confidence level is lower than the second confidence threshold, the monitoring result corresponding to the type label is discarded, and the type of the target passenger in the monitoring area is re-determined by combining the area size estimate and micro-motion characteristics. For example, the type label can be modified according to the size estimate and micro-motion characteristics, and the modified type label is determined as the fusion monitoring result. For example, "child" can be modified to "small adult". For the type label "child safety seat but internal condition unknown", if the radar detects micro-motion characteristics that conform to vital signs in the area, it can be modified to "child safety seat (including child)".

[0064] For example, such as Figure 3BAs shown in the above embodiments, based on the visual signals output by the visual sensors, i.e., the visual monitoring results, the position of the adult passenger, the type (forward / rear-facing) of the child safety seat, and its position within the cabin can be determined. A perception context is then generated and fed back to the radar signal processing unit in real time for dynamic adjustment of the radar's perception strategy and parameter thresholds. If the perception context identifies an adult, i.e., a seat is occupied by an adult, the radar can be instructed to reduce its detection sensitivity in that area, or to directly classify specific micro-motion signals from that area as "adult activity," effectively avoiding misinterpreting subtle adult movements as child abandonment and reducing the system's false alarm rate. If the perception context identifies a rear-facing child safety seat, the radar is instructed to lower the detection threshold and focus the beam in that area to extract weak vital signs signals attenuated by the seat shell. Subsequently, more accurate occupant classification can be achieved by fusing the classification labels provided by the visual sensors (such as "child" or "child safety seat") with the target physical characteristics directly measured by the radar (such as size, micro-motion amplitude, etc.).

[0065] In the aforementioned process, visual monitoring results can be used to finely guide and calibrate the radar perception process, thereby addressing the inherent limitations of radar in target discrimination: the difficulty in accurately distinguishing children from small adults or pets. Furthermore, it overcomes the problem of radar signal attenuation due to physical obstruction in rear-facing child safety seat scenarios. Specifically, the fine-grained semantic information provided by the visual sensor, such as the positions of adults and safety seats, allows the radar to adaptively adjust its perception threshold and focusing strategy, thereby enhancing the reliability of detection in high-risk, concealed scenarios (such as children in rear-facing child safety seats) and achieving a comprehensive improvement in overall detection accuracy and system reliability.

[0066] In the above embodiments, not only can multiple types of monitoring results be integrated to overcome the technical defects of individual monitoring results, and realize all-weather, highly robust, and intelligent monitoring and protection of the safety status of children in the cabin; but also, compared with fusion methods such as simple superposition of two monitoring results, the above embodiments achieve deep fusion of the perception layer through two-way closed-loop optimization of radar-assisted vision to improve the detection rate and visual feedback radar to improve classification accuracy, thereby improving the overall detection accuracy and reliability.

[0067] In S240, passenger safety control is implemented based on the results of integrated monitoring and the vehicle riding scenario.

[0068] For example, S240 may include any of the following possible implementations, but is not limited thereto: One possible implementation method is to integrate the monitoring results to determine that a child is present in the vehicle cabin, and the vehicle riding scenario is a scenario where the vehicle is turned off and the doors are locked. S240 includes: in response to a first preset time after the doors are locked, triggering a first passenger safety alarm; in response to the end of the first passenger safety alarm and the detection that a child is present in the vehicle cabin, triggering a second passenger safety alarm within a second preset time, wherein the level of the second passenger safety alarm is higher than that of the first passenger safety alarm; and performing at least one of the following safety control steps: controlling the temperature in the vehicle cabin (e.g., automatically starting the air conditioner for cooling or heating to prevent the cabin temperature from being too high or too low and causing harm to the child), activating the escape mechanism in the vehicle cabin (e.g., automatically unlocking the doors or lowering part of the windows to provide a passage for the child to rescue themselves or for external rescue), and reporting the corresponding riding information (e.g., sending alarm information, vehicle location, etc. to the owner's mobile phone or other alarm platform or cloud service platform through the vehicle networking function).

[0069] For example, the first passenger safety warning may be a combination of visual (such as flashing high-mounted brake lights) and audible (such as a specific beeping sound) warning information to ensure it is noticeable outside the vehicle. The second passenger safety warning may have a higher intensity, frequency, or manner than the first passenger safety warning (such as louder volume or more rapid lights) to attract attention over a wider area.

[0070] For example, child abandonment detection and tiered intervention in parking scenarios can be achieved through the aforementioned feasible methods. Specifically, after the vehicle is turned off and the doors are locked, the fusion system can continuously monitor the state inside the cabin. If the fusion monitoring results determine that a child has been left behind, an alarm sequence can be initiated immediately. First, after the vehicle is locked (with a delay of no more than 15 seconds), an initial warning is triggered. The initial warning combines visual (such as a special flashing of the high-mounted brake light) and radar (such as emitting a specific tone through the vehicle's buzzer) to ensure that the warning is heard and seen by people outside the vehicle and nearby. If the child is still detected after the initial warning ends, an escalation warning is initiated within 90 seconds. This ensures a reliable judgment of the continued presence of the child and avoids premature termination of the warning due to sensor misjudgment. In addition, intervention measures can be implemented, such as automatically executing additional interventions like temperature control intervention, escape intervention, and information reporting.

[0071] Option 2 can be implemented where the fusion monitoring results indicate the presence of a child in a rear-facing child safety seat in the vehicle cabin, and the vehicle riding scenario is a vehicle collision scenario; S240 includes: turning off and / or prompting to turn off the passenger-side airbag in the vehicle cabin.

[0072] For example, the front side airbag of a rear-facing child safety seat can be automatically disabled, and the status can be clearly displayed on the dashboard: passenger airbag deactivated.

[0073] For example, the aforementioned feasible methods can be used to implement airbag linkage for collision scenarios. Specifically, since the airbag must be deactivated for any rear-facing child safety seat in the front passenger seat, otherwise it will cause serious injury to the child, a visual sensor can be used to identify the type of child safety seat (infant carrier, car seat, etc.) and its installation direction (forward or rear-facing). When the visual sensor identifies a rear-facing child safety seat and the radar confirms vital signs, the highest safety strategy is adopted: the front passenger airbag can be automatically disabled, and the "passenger airbag deactivated" status can be clearly displayed on the instrument panel, such as with pictographs and text. If the system is designed as a suggestion, it will issue a strong, unavoidable visual and auditory prompt to the driver, suggesting that they manually deactivate the airbag, thereby fundamentally avoiding the risk of fatal secondary head and neck injuries to rear-facing children when the airbag deploys.

[0074] In addition, in some embodiments, a visual sensor can be used to acquire the real-time posture of the target passenger; based on the real-time posture, it can be determined whether the target passenger is engaging in any preset dangerous riding behavior; and a warning can be issued for any dangerous riding behavior.

[0075] For example, real-time attitude includes at least one of the following: the position of key body parts, joint angles, and motion trajectory. Specifically, it can be extracted from cockpit images acquired by visual sensors using image processing and attitude estimation algorithms.

[0076] For example, a pre-trained behavior recognition model, such as a deep learning-based time series classification model, can be used to analyze real-time posture data and determine whether it conforms to preset dangerous riding behaviors. Dangerous riding behaviors include, but are not limited to: standing in a seat, leaning out of a window or sunroof, and unfastening or removing the seatbelt. Specifically, the results can be output by comparing with a behavior feature database to determine whether dangerous riding behaviors exist and their categories.

[0077] For example, the above methods can be used to provide early warnings of dangerous child behaviors in driving scenarios. Specifically, in addition to detecting children leaving behind objects, visual sensors can continuously analyze the posture of child occupants in real time during driving. Through a trained behavior recognition model, dangerous riding behaviors such as children standing in their seats, leaning out of windows or sunroofs, or excessively unfastening seatbelts can be detected. Upon recognizing such behaviors, regardless of whether the radar signal generates a strong response due to the child's large movements, a real-time and clear warning will be immediately issued to the driver through the in-vehicle audio system or the central control display screen. This allows the driver to intervene and correct the situation immediately, preventing potential accidents.

[0078] In the above embodiments, comprehensive protection for in-cabin driving and child-leaving scenarios can be achieved, as well as multi-level and intelligent safety alarm and linkage strategies. This enables graded alarms and interventions for children left behind when the vehicle is parked, accurate identification of rear-facing child safety seats in collision scenarios and linkage of airbag switches, and real-time warnings of dangerous behaviors during driving, thereby more comprehensively and reliably meeting the requirements for child safety control.

[0079] The technical solution of this application can compensate for the omissions in the detection of target passengers by the visual monitoring results obtained by the visual sensor (e.g., omissions that may occur when the passenger is covered by a blanket, is in the blind spot of the child safety seat, or is in a low-light environment). The visual monitoring results obtained by the visual sensor can guide the radar to improve its classification accuracy and anti-interference ability. For example, it can guide the radar to perform adaptive perception, improve its ability to detect weak vital signs in rear-facing child safety seats, and effectively distinguish between adults and children, reducing the false alarm rate. Thus, it can realize the deep integration of radar and visual sensors for cabin child passenger safety control, forming a multimodal perception system with complementary advantages.

[0080] At the same time, the fusion detection results can be combined with the vehicle riding scenario to achieve full-chain safety protection from the detection of children's presence to intelligent early warning and proactive intervention.

[0081] Figure 4 A block diagram of a passenger safety control device according to an embodiment of this application is shown. This passenger safety control device may be a software unit or a hardware unit, or a combination of both, as part of a computer device. Figure 4 As shown, the passenger safety control device 400 provided in this application embodiment may specifically include: The visual monitoring module 410 is used to: acquire visual monitoring results of passengers in the vehicle cabin through a visual sensor; the vital signs monitoring module 420 is used to: monitor vital signs in the vehicle cabin and determine the vital signs monitoring results; the fusion monitoring module 430 is used to: determine the fusion monitoring results for the target passenger based on the vital signs monitoring results and the visual monitoring results; and the passenger safety control module 440 is used to: perform passenger safety control based on the fusion monitoring results and the vehicle riding scenario.

[0082] In some embodiments, the vital signs monitoring module 420 is specifically used for: determining the passengers in each monitoring area of ​​the vehicle cabin based on the visual monitoring results; adjusting the monitoring parameters of the vital signs monitor based on the passengers in the area; and performing vital signs monitoring in the vehicle cabin based on the adjusted vital signs monitor to determine the vital signs monitoring results.

[0083] In some embodiments, the vital signs monitoring module 420 is specifically configured to: reduce the monitoring sensitivity of the vital signs monitor for the monitoring area corresponding to the passenger in the area when the passenger is an adult; or, increase the monitoring sensitivity of the vital signs monitor for the monitoring area corresponding to the passenger in the area when the passenger is a rear-facing child safety seat and / or adjust specific monitoring parameters, the specific monitoring parameters including parameters that instruct the vital signs monitor to analyze specific frequency band signals corresponding to vital signs; or, adjust the monitoring parameters to the parameters corresponding to a preset monitoring mode when the passenger is a front-facing child safety seat or does not have a child safety seat.

[0084] In some embodiments, the fusion monitoring module 430 is specifically used to: fuse the type of the passenger in the visual monitoring results and the size and micro-motion characteristics of the target in the living organism monitoring results to determine the fusion monitoring results.

[0085] In some embodiments, the fusion monitoring module 430 is specifically configured to: determine the type and location of the target passenger as a passenger with vital signs; in response to the visual monitoring result indicating that the passenger includes the target passenger, and obtain the fusion monitoring result; or, in response to the visual monitoring result indicating that the target passenger does not exist in the living body area, determine that the fusion monitoring result includes the presence of an obscured target passenger in the vehicle cabin, wherein the living body area is the area where living bodies exist in the living body monitoring result; or, in response to the visual monitoring result indicating that the target passenger does not exist in the non-living body area, determine that the fusion monitoring result includes the absence of the target passenger in the vehicle cabin, wherein the non-living body area is the area where living bodies do not exist in the living body monitoring result.

[0086] In some embodiments, the vehicle safety control module 440 is specifically configured to: integrate monitoring results to determine that a child exists in the vehicle cabin, and the vehicle riding scenario is a scenario where the vehicle is turned off and the doors are locked; trigger a first vehicle safety alarm in response to a first preset time after the doors are locked; trigger a second vehicle safety alarm within a second preset time in response to the end of the first vehicle safety alarm and the detection that a child exists in the vehicle cabin, wherein the level of the second vehicle safety alarm is higher than that of the first vehicle safety alarm; and perform at least one of the following safety control steps: controlling the temperature in the vehicle cabin, activating the escape mechanism in the vehicle cabin, and reporting the corresponding riding information.

[0087] In some embodiments, the passenger safety control module 440 is specifically used to: integrate monitoring results to determine that there is a child in a rear-facing child safety seat in the vehicle cabin, and the vehicle riding scenario is a vehicle collision scenario; and to turn off and / or prompt the passenger-side airbag in the vehicle cabin to be turned off.

[0088] In some embodiments, the passenger safety control module 440 is further configured to: acquire the real-time posture of the target passenger through a visual sensor; determine whether the target passenger has engaged in any preset dangerous riding behavior based on the real-time posture; and provide a warning for any dangerous riding behavior.

[0089] The specific implementation of each module in the passenger safety control device provided in this application embodiment can refer to the content of the above-described passenger safety control method, and will not be repeated here.

[0090] The various modules in the aforementioned passenger safety control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to these modules.

[0091] Figure 5 A schematic diagram of a computer system suitable for implementing embodiments of this application is shown. It should be noted that... Figure 5 The illustrated vehicle computer system 500 is merely an example and should not limit the functionality or scope of the embodiments of this application. The computer system 500 can implement the steps in the above-described method embodiments. Furthermore, the embodiments of this application can also be applied to electronic devices such as computers, mobile phones, or servers. The schematic diagrams of these electronic devices are similar to those of the vehicle computer system 500 described above, and will not be elaborated upon further in this application.

[0092] like Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage section 508 into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0093] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network (LAN) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0094] Specifically, according to embodiments of this application, the processes described in the flowcharts above can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts above. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the various functions defined in the apparatus of this application.

[0095] In one embodiment, an electronic device is also provided, comprising: Processor; and Memory is used to store the processor's executable instructions; The processor is configured to execute the steps in the above method embodiments by executing executable instructions.

[0096] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0097] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0098] It should be noted that the computer-readable storage medium of this application can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, disk storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0099] This embodiment is only used to illustrate this application. The selection of software and hardware platform architecture, development environment, development language, message acquisition source, etc. in this embodiment can be varied. Based on the technical solution of this application, any improvement or equivalent transformation made to a certain part according to the principle of this application should not be excluded from the protection scope of this application.

[0100] It should be noted that the terminology used in the embodiments of this application and the appended claims is for the purpose of describing specific embodiments only, and is not intended to limit the embodiments of this application.

[0101] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0102] If implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of this application embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices, apparatuses and methods can be implemented in other ways.

[0105] For example, the division of units, modules, or components in the device embodiments described above is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some units, modules, or components may be ignored or not executed.

[0106] For example, the units / modules / components described above as separate / display components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units / modules / components can be selected to achieve the objectives of the embodiments of this application, depending on actual needs.

[0107] Finally, it should be noted that the mutual coupling or direct coupling or communication connection shown or discussed above can be an indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0108] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for controlling passenger safety, characterized in that, include: Visual sensors are used to obtain visual monitoring results of passengers in the vehicle cabin. Vital signs monitoring was conducted inside the vehicle cabin to determine the results of the vital signs monitoring. Based on the life monitoring results and the visual monitoring results, determine the fusion monitoring results for the target passenger. Based on the fusion monitoring results and vehicle riding scenarios, ride safety controls are implemented.

2. The method according to claim 1, characterized in that, The process of monitoring vital signs inside the vehicle cabin and determining the monitoring results includes: Based on the visual monitoring results, the passenger objects in each monitoring area within the vehicle cabin are determined. Adjust the monitoring parameters of the vital signs monitor according to the passengers in the area; Based on the adjusted vital signs monitor, vital signs are monitored inside the vehicle cabin to determine the monitoring results.

3. The method according to claim 2, characterized in that, Based on the passengers in the area, adjust the monitoring parameters of the vital signs monitor, including: In response to the fact that the passenger in the area is an adult, the sensitivity of the vital signs monitor for the monitoring area corresponding to the passenger in the area is reduced; or... In response to the passenger in the area being a rear-facing child safety seat, the monitoring sensitivity of the vital signs monitor for the monitoring area corresponding to the passenger in the area is increased and / or specific monitoring parameters are adjusted. These specific monitoring parameters include parameters instructing the vital signs monitor to analyze specific frequency band signals corresponding to vital signs; or... In response to the fact that the passenger in the area is using a forward-facing child safety seat or does not use a child safety seat, the monitoring parameters are adjusted to the parameters corresponding to the preset monitoring mode.

4. The method according to claim 1, characterized in that, The step of determining the fused monitoring results for the target passenger based on the life monitoring results and the visual monitoring results includes: The type of the passenger in the visual monitoring results and the size and micro-motion characteristics of the target in the living organism monitoring results are fused to determine the fused monitoring results.

5. The method according to claim 1, characterized in that, The target passenger is a passenger with vital signs. The step of determining the fused monitoring results for the target passenger based on the life monitoring results and the visual monitoring results includes: In response to the visual monitoring result indicating that the passenger includes the target passenger, the type and location of the target passenger are determined, and the fused monitoring result is obtained; or... In response to the visual monitoring result indicating that no target passenger is present in the living organism area, the fused monitoring result is determined to include the presence of an obscured target passenger in the vehicle cabin, wherein the living organism area is the area where a living organism is present in the living organism monitoring result; or... In response to the visual monitoring result indicating that there is no target passenger in the inanimate area, it is determined that the fused monitoring result includes that the target passenger is not in the vehicle cabin, wherein the inanimate area is the area where no living organism is found in the living organism monitoring result.

6. The method according to any one of claims 1-5, characterized in that, The fusion monitoring result indicates that there is a child in the vehicle cabin, and the vehicle riding scenario is a scenario where the vehicle is turned off and the doors are locked. The method of implementing passenger safety control based on the fused monitoring results and vehicle riding scenario includes: In response to a first preset time after the vehicle door is locked, a first passenger safety alarm is triggered. In response to the end of the first passenger safety alarm and the detection that the child is in the vehicle cabin, a second passenger safety alarm is triggered within a second preset time period. The level of the second passenger safety alarm is higher than that of the first passenger safety alarm. Perform at least one of the following safety control steps: control the temperature in the vehicle cabin, activate the escape mechanism in the vehicle cabin, and report the corresponding passenger information.

7. The method according to any one of claims 1-5, characterized in that, The fusion monitoring result indicates that there is a child in a rear-facing child safety seat in the vehicle cabin, and the vehicle riding scenario is a vehicle collision scenario; The method of implementing passenger safety control based on the fused monitoring results and vehicle riding scenario includes: Turn off and / or prompt to turn off the passenger-side airbags in the vehicle cabin.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: The real-time posture of the target passenger is obtained through the visual sensor. Based on the real-time posture, determine whether the target passenger has engaged in any pre-set dangerous riding behavior; The aforementioned dangerous riding behaviors are warned against.

9. A passenger safety control device, characterized in that, include: The visual monitoring module is used to: acquire visual monitoring results of passengers in the vehicle cabin through visual sensors; The vital signs monitoring module is used to: monitor vital signs in the vehicle cabin and determine the monitoring results. The fusion monitoring module is used to: determine the fusion monitoring results for the target passenger based on the living organism monitoring results and the visual monitoring results; The passenger safety control module is used to: perform passenger safety control based on the fusion monitoring results and the vehicle riding scenario.

10. A vehicle, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images