Multimodal perception fusion based vehicle-mounted life monitoring method, system and medium

Abstract

The application relates to the technical field of intelligent automobiles, in particular to a multi-modal perception fusion vehicle-mounted life monitoring method, system and medium. The method comprises the following steps: acquiring original perception data in a cabin; projecting point cloud data to image data in the cabin, and performing space-time alignment by using a global clock source to obtain a target correlation region; obtaining a radar recognition probability according to spectrum data corresponding to the target correlation region; obtaining a visual recognition probability according to image data corresponding to the target correlation region; obtaining a pressure recognition probability according to pressure data; performing decision fusion on the radar recognition probability, the visual recognition probability and the pressure recognition probability to calculate a fused life existence joint probability; and generating a corresponding grade vehicle-mounted control strategy according to the life existence joint probability. The application greatly improves the accuracy of vehicle-mounted life monitoring.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle technology, and in particular to a method, system and medium for in-vehicle life monitoring based on multimodal perception fusion. Background Technology

[0002] With the rapid development of the intelligent connected vehicle industry, passenger safety has become a core component of automotive safety systems. In recent years, accidents involving children and pets being left in vehicles and suffocating due to high temperatures have occurred frequently, making in-vehicle vital sign monitoring technology an essential safety feature in automobiles.

[0003] Currently, vehicle-mounted life monitoring technology mainly adopts a single-modal perception solution. Among the single-modal solutions, millimeter-wave radar has the ability to penetrate non-metallic obstructions, but it has the drawbacks of weak object classification ability and susceptibility to interference from external movement; visual camera solutions can accurately identify human features, but they are completely ineffective in scenarios such as direct sunlight, complete darkness at night, or when covered by blankets; pressure sensor solutions can detect seat occupancy, but they cannot distinguish between living people and heavy objects, resulting in an extremely high false alarm rate. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimodal perception fusion method, system, and medium for vehicle-mounted life monitoring to address the aforementioned technical problems.

[0005] Firstly, this application provides a multimodal perception fusion-based vehicle life monitoring method, the method comprising: Acquire raw sensing data from inside the cockpit, including point cloud data and spectrum data collected by the first sensor, image data collected by the second sensor, and seat pressure data collected by the third sensor. The point cloud data is projected onto the image data inside the cockpit, and spatiotemporal alignment is performed using a global clock source to obtain the target associated region; The radar recognition probability is obtained based on the spectral data corresponding to the target association region; the visual recognition probability is obtained based on the image data corresponding to the target association region; and the pressure recognition probability is obtained based on the pressure data. The radar recognition probability, visual recognition probability, and pressure recognition probability are fused together to calculate the joint probability of life existence. Based on the joint probability of life existence, generate corresponding levels of vehicle control strategies.

[0006] In one embodiment, point cloud data is projected onto image data within the cockpit, and spatiotemporal alignment is performed using a global clock source to obtain the target associated region, including: The point cloud data is filtered based on the set neighborhood radius; Clustering algorithms were used to cluster and analyze the filtered point cloud data to obtain the geometric centroid coordinates of each organism. The geometric centroid coordinates are transformed from the radar Cartesian coordinate system to the image pixel coordinate system, and spatiotemporal alignment is performed using a global clock source to obtain the target association region.

[0007] In one embodiment, if the time difference between the geometric centroid coordinates and the corresponding frame of the image data is greater than a threshold, linear interpolation is used to predict and compensate the geometric centroid coordinates.

[0008] In one embodiment, the radar recognition probability, visual recognition probability, and pressure recognition probability are fused to calculate the fused joint probability of life presence, which includes: Define a state space, which is constructed from multiple mutually exclusive subsets of propositions. These subsets of propositions are used to represent the life state of a seat. Based on each proposition subset and environmental adjustment coefficient, calculate the amount of evidence for radar recognition probability, visual recognition probability, and pressure recognition probability; Calculate the local conflict factor between the evidence quantities, and adjust the environmental adjustment coefficient according to the local conflict factor to calculate the new evidence quantity; Based on the new amount of evidence, the global joint conflict factor among the amounts of evidence is calculated; Based on the global joint conflict factor and the amount of new evidence, the joint probability of life existence is calculated.

[0009] In one embodiment, the state space is:

[0010] Where L represents the subset of propositions in which seats contain living beings, O represents the subset of propositions in which seats contain inanimate objects, and E represents the subset of propositions in which seats are vacant.

[0011] In one embodiment, the amount of evidence used to calculate the radar recognition probability, visual recognition probability, and pressure recognition probability based on each proposition subset and environmental adjustment coefficient includes:

[0012]

[0013]

[0014] in, This represents the amount of evidence assigned by the i-th sensor to the propositional subset L; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This represents the recognition probability corresponding to the i-th sensor; This represents the environmental conditioning coefficient corresponding to the i-th sensor.

[0015] In one embodiment, calculating the local conflict factor between amounts of evidence includes:

[0016] in, This represents the local conflict factor between the i-th sensor and the j-th sensor; This represents the amount of evidence assigned by the i-th sensor to the proposition subset A; Let A represent the amount of evidence assigned by the j-th sensor to propositional subset B; propositional subsets A and B are one of propositional subsets L, O, and E.

[0017] In one embodiment, the joint probability of life existence is calculated based on the global joint conflict factor and the new amount of evidence, including: Calculate the joint probability of the existence of life for a subset L of propositions. :

[0018]

[0019] in, Let A, B, and C represent the amount of evidence assigned by the first, second, and third sensors to propositional subsets A, B, and C, respectively, where propositional subsets A, B, and C are one of propositional subsets L, O, and E. This represents the global joint conflict factor.

[0020] Secondly, this application provides a multimodal perception fusion-based vehicle life monitoring system, comprising: Multimodal sensing unit, used to collect raw sensing data inside the cockpit; The processing and analysis unit is used to execute the steps of the method in the first aspect to generate the corresponding level of vehicle control strategy; The strategy execution unit is used to execute vehicle control strategies.

[0021] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of the first aspect.

[0022] The above-mentioned multimodal perception fusion-based vehicle life monitoring method, system, and medium have the following technical effects: Point cloud data is projected onto image data within the cockpit, and spatiotemporal alignment is performed using a global clock source to obtain the target association region. This eliminates spatial misalignment and temporal deviation of heterogeneous data, accurately locks the target association region within the cockpit, and provides a unified and reliable data foundation for probability calculation of recognition.

[0023] By fusing radar recognition probability, visual recognition probability, and pressure recognition probability, the combined probability of life presence is calculated, effectively compensating for the shortcomings of single sensor perception and significantly improving the accuracy of vehicle-mounted life monitoring.

[0024] Based on the joint probability of life presence, corresponding vehicle control strategies are generated to accurately determine the cabin life risk level and adopt the corresponding vehicle control strategies to construct a fully closed-loop protection mechanism. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a multimodal perception fusion method for vehicle-mounted life monitoring in one embodiment. Figure 2 This is a flowchart illustrating a spatiotemporal alignment method in one embodiment; Figure 3 This is a flowchart illustrating a method for calculating the joint probability of life existence in one embodiment; Figure 4 This is a schematic diagram of the structure of a vehicle-mounted life monitoring system with multimodal perception fusion in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] In one embodiment, such as Figure 1 As shown, a multimodal perception fusion-based vehicle life monitoring method is provided, the method including: S102, acquire raw perception data from inside the cockpit.

[0029] The raw sensing data includes point cloud data and spectrum data of the cabin collected by the first sensor, image data of the cabin collected by the second sensor, and pressure data of the seats collected by the third sensor.

[0030] The first sensor employs a 3D millimeter-wave radar, utilizing FMCW modulation to emit high-frequency electromagnetic waves (such as 60GHz or 77GHz). Doppler spectral analysis extracts respiratory characteristics (0.1Hz-0.5Hz) and heart rate characteristics (1Hz-2Hz) as spectral data. Simultaneously, it collects point cloud data from within the cockpit. Its core function is to achieve non-contact vital sign monitoring, and it possesses the ability to penetrate non-metallic obstructions such as blankets and quilts, thus overcoming blind spots.

[0031] The second sensor employs a vision unit, which acquires high-resolution image data via a vision camera. Its role is to provide rich detailed features, compensating for the sparsity in object classification by 3D millimeter-wave radar.

[0032] The third sensor employs a distributed pressure sensor array embedded within the seat cushion to sense the distribution of static load. Its function is to initially determine the seat occupancy status and, based on weight stability, assist in eliminating external moving interference that may be detected by the radar.

[0033] S104, Project the point cloud data onto the image data inside the cockpit, and use the global clock source for spatiotemporal alignment to obtain the target associated region; S106, obtain the radar recognition probability based on the spectrum data corresponding to the target association region; obtain the visual recognition probability based on the image data corresponding to the target association region; obtain the pressure recognition probability based on the pressure data; S108, the radar recognition probability, visual recognition probability and pressure recognition probability are fused together to calculate the fused joint probability of life existence; S110 generates a corresponding level of vehicle control strategy based on the joint probability of life existence.

[0034] In this embodiment, point cloud data is projected onto image data within the cockpit, and spatiotemporal alignment is performed using a global clock source to obtain the target association region. This eliminates spatial misalignment and timing deviation of heterogeneous data, accurately locks the target association region within the cockpit, and provides a unified and reliable data foundation for probability calculation.

[0035] By fusing radar recognition probability, visual recognition probability, and pressure recognition probability, the combined probability of life presence is calculated, effectively compensating for the shortcomings of single sensor perception and significantly improving the accuracy of vehicle-mounted life monitoring.

[0036] Based on the joint probability of life presence, corresponding vehicle control strategies are generated to accurately determine the cabin life risk level and adopt the corresponding vehicle control strategies to construct a fully closed-loop protection mechanism.

[0037] In one embodiment, such as Figure 2 As shown, point cloud data is projected onto image data within the cockpit, and spatiotemporal alignment is performed using a global clock source to obtain the target-related region, which includes: S202, filter the point cloud data according to the set neighborhood radius.

[0038] Neighborhood radius The value is set to 0.1m-0.2m, and this value is adaptively adjusted according to the density of the spatial distribution of the chest cavity undulations of the human body inside the cabin.

[0039] If the number of points within a certain neighborhood radius is less than the minimum number of points threshold If the clutter is isolated static (such as reflections from seat backs), it is identified and removed. Minimum number of points threshold. For example, set it to 5-10 points.

[0040] S204 uses a clustering algorithm to perform cluster analysis on the filtered point cloud data to obtain the geometric centroid coordinates of each organism.

[0041] Clustering algorithms group micro-moving points generated by the same organism into a single target instance by finding densely connected sets of points, and then calculate its three-dimensional geometric centroid. .

[0042] In one example embodiment, an improved DBSCAN clustering algorithm is used for sparse point cloud data generated by 3D millimeter-wave radar. This algorithm does not require a preset number of targets, can automatically identify and filter out static or periodic clutter such as metal walls inside the cockpit and air conditioning blades, and calculates the geometric centroid coordinates of each living target. .

[0043] S206 transforms the geometric centroid coordinates from the radar rectangular coordinate system to the image pixel coordinate system and uses a global clock source for spatiotemporal alignment to obtain the target association region.

[0044] The rotation matrix R and translation vector t obtained through offline calibration are used to transform the point cloud in the radar Cartesian coordinate system. Transform to camera space coordinates :

[0045] The rotation matrix R and translation vector t are related to the actual installation positions of the first sensor (3D millimeter-wave radar) and the second sensor (vision unit). The camera's intrinsic parameter matrix K1 (including focal length) is used. and the main point ), camera space coordinates The plane projected onto the image pixel coordinate system has the following coordinates: :

[0046] in It is a scale factor, which usually corresponds directly to the depth value of the geometric centroid coordinates in the image pixel coordinate system.

[0047] Spatiotemporal alignment is performed using the vehicle-mounted terminal T-box as the global clock source.

[0048] If the time difference between the geometric centroid coordinates and the corresponding frame of the image data is greater than a threshold, then linear interpolation is used to predict and compensate for the geometric centroid coordinates.

[0049] This threshold is, for example, 20ms.

[0050] Predictive compensation ensures that, under vehicle vibration conditions, the geometric centroid coordinates can be accurately superimposed on the corresponding human organs (such as the chest region) in the image data.

[0051] In one embodiment, the radar identification probability is obtained based on the spectrum data corresponding to the target association region, specifically as follows: Radar identification probability :

[0052] in The energy peaks were extracted in the 0.1Hz-0.5Hz (respiration) and 1Hz-2Hz (heartbeat) frequency bands using Fast Fourier Transform (FFT). This represents the total energy across the entire frequency band.

[0053] Based on the image data corresponding to the target associated region, the visual recognition probability is obtained, specifically: Visual recognition probability : Using the classification confidence score output by the deep learning model, if human features are detected within the target-related region of interest (ROI) and the skeleton integrity is greater than 70%, then the visual recognition probability is... The value is 1.

[0054] Based on the stress data, the stress recognition probability is obtained, specifically: Pressure recognition probability : Based on the static mass distribution stability calculation of the distributed pressure sensor array, if the weight change rate... Extremely small values ​​that fall within a preset range of human body weight are assigned a high weight.

[0055] In one embodiment, such as Figure 3 As shown, the radar recognition probability, visual recognition probability, and pressure recognition probability are fused together to calculate the joint probability of life existence after fusion, which includes: S302, Define the state space, which is constructed from multiple mutually exclusive subsets of propositions. The subsets of propositions are used to represent the life state of a seat.

[0056] Before performing probability fusion, we first define a mutually exclusive and complete state space. As the logical foundation for all sensor observations:

[0057] Where L represents the subset of propositions in which seats contain living beings, O represents the subset of propositions in which seats contain inanimate objects, and E represents the subset of propositions in which seats are vacant.

[0058] All evidence observed will be allocated in the state space. Above, including singleton sets and the union representing uncertainty (e.g.) This indicates an inability to distinguish between living beings and objects. (This indicates that the current status cannot be confirmed).

[0059] S304, calculate the amount of evidence for radar recognition probability, visual recognition probability, and pressure recognition probability based on each proposition subset and environmental adjustment coefficient.

[0060] To address potential "blind spots" in sensors, an environmental adjustment coefficient is introduced. (such as the multipath interference coefficient of 3D millimeter-wave radar and the illumination saturation of a camera).

[0061]

[0062]

[0063]

[0064] in, This represents the amount of evidence assigned by the i-th sensor to the propositional subset L; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated represents the "cognitive gap" caused by limitations in the objective environment, meaning that the sensor currently dares not make a conclusion. This represents the recognition probability corresponding to the i-th sensor; This represents the environmental conditioning coefficient corresponding to the i-th sensor.

[0065] S306, calculate the local conflict factor between the evidence quantities, and adjust the environmental adjustment coefficient according to the local conflict factor to calculate the new evidence quantity.

[0066] Calculate the local conflict factor between the amounts of evidence:

[0067] in, This represents the local conflict factor between the i-th sensor and the j-th sensor; This represents the amount of evidence assigned by the i-th sensor to the proposition subset A; Let A represent the amount of evidence assigned by the j-th sensor to propositional subset B; propositional subsets A and B are one of propositional subsets L, O, and E.

[0068] Local conflict factors include , , . This indicates a conflict between the first and second sensors. This indicates a conflict between the second and third sensors. This indicates a conflict between the first and third sensors.

[0069] When the intersection of two propositional subsets is empty (i.e., a complete contradiction, such as "having life" and "empty seat"), their product is accumulated. Local conflict factors The closer a value is to 1, the more intense the contradiction between the sensors; for example, the first sensor strongly supports the existence of life. (High), while the second sensor, due to being completely black, considers the seat to be vacant ( (High), then the local conflict factor .

[0070] Local conflict factors Its main function is sensor self-diagnosis. For example, if the local conflict factor... Extremely high and local conflict factor If the value is low, it is determined that the second sensor may be in a malfunctioning state (such as being exposed to strong light or blocked by the lens), and the weight of the environmental adjustment coefficient of the second sensor is automatically reduced.

[0071] S308. Based on the new amount of evidence, the global joint conflict factor among the amounts of evidence is calculated.

[0072] Global Joint Conflict Factor This refers to the total conflict rate when the three sensors are considered as a whole; it is a normalization factor in the formula for calculating the joint probability of life existence. When the global joint conflict factor... When the value approaches 1, it indicates that there is an irreconcilable contradiction in the judgments of the three sensors regarding the current cockpit status (L / O / E).

[0073] Calculate the global joint conflict factor :

[0074] in, Let A, B, and C represent the amount of evidence assigned by the first, second, and third sensors to propositional subsets A, B, and C, respectively, where propositional subsets A, B, and C are one of propositional subsets L, O, and E. This represents the global joint conflict factor.

[0075] S310, based on the global joint conflict factor and the new amount of evidence, calculates the joint probability of life existence.

[0076] Using the aforementioned new amount of evidence and the global joint conflict factor K, the propositional subset is targeted... Calculate the joint probability of life forms after fusion. :

[0077] Examples are given below: , , 9, because The intersection of the three is not empty, and the global joint conflict factor. .

[0078]

[0079] Right now:

[0080] In one embodiment, the vehicle control strategy with the corresponding intervention level is determined based on the risk index range in which the joint probability of life exists.

[0081] In one example embodiment, the vehicle control strategy has three intervention levels, corresponding to different levels of core operations, as shown in Table 1.

[0082] Table 1

[0083] This application overcomes the limitations of traditional single-unit sensing schemes by using multimodal sensing fusion, achieving a closed-loop upgrade from "simple early warning" to "autonomous intervention." Compared with existing technologies, this invention has the following beneficial effects and advantages: Extremely high monitoring confidence: The decision fusion of the first, second and third sensors eliminates the uncertainty of single sensor measurement and solves the problems of missed and false alarms in "long-tail scenarios" such as blanket obstruction and direct strong light.

[0084] Precise spatiotemporal alignment: Through clustering algorithms and multi-coordinate system transformations, point cloud data and image data are deeply aligned in the spatiotemporal dimension, avoiding target position offset.

[0085] Self-diagnosis and conflict resolution capabilities: Introducing local conflict factors It can automatically identify and reduce the weight of the environmental adjustment coefficient of failed sensors (such as obstructed cameras) to ensure a stable joint probability of life existence in complex environments.

[0086] Breakthrough autonomous closed-loop intervention: When the driver is unable to respond (e.g., in a signal blind spot or due to accidental unconsciousness), it can autonomously trigger forced window lowering, air conditioning activation, and eCall emergency rescue, improving the survival environment of trapped individuals within the golden time frame.

[0087] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0088] In one embodiment, such as Figure 4 As shown, a multimodal perception fusion-based vehicle life monitoring system is provided, including: The multimodal sensing unit 100 is used to collect raw sensing data inside the cockpit; The processing and analysis unit 200 is used to execute the steps of the method in the above embodiments to generate a corresponding level of vehicle control strategy; The strategy execution unit 300 is used to execute vehicle control strategies.

[0089] The processing and analysis unit is specifically used for: filtering point cloud data according to a set neighborhood radius; clustering analysis of the filtered point cloud data using a clustering algorithm to obtain the geometric centroid coordinates of each organism; converting the geometric centroid coordinates from the radar rectangular coordinate system to the image pixel coordinate system, and using a global clock source for spatiotemporal alignment to obtain the target association region.

[0090] The processing and analysis unit is specifically used to: if the time difference between the geometric centroid coordinates and the corresponding frame of the image data is greater than a threshold, then use linear interpolation to predict and compensate for the geometric centroid coordinates.

[0091] The processing and analysis unit is specifically used for: defining a state space, which is constructed from multiple mutually exclusive proposition subsets, each representing the life existence state of a seat; calculating the amount of evidence for radar recognition probability, visual recognition probability, and pressure recognition probability based on each proposition subset and environmental adjustment coefficient; calculating the local conflict factor between the amounts of evidence, adjusting the environmental adjustment coefficient based on the local conflict factor to calculate new amounts of evidence; calculating the global joint conflict factor between the amounts of evidence based on the new amounts of evidence; and calculating the joint probability of life existence based on the global joint conflict factor and the new amounts of evidence.

[0092] In one embodiment, the state space is:

[0093] Where L represents the subset of propositions in which seats contain living beings, O represents the subset of propositions in which seats contain inanimate objects, and E represents the subset of propositions in which seats are vacant.

[0094] In one embodiment, the evidence amount by which the processing and analysis unit calculates the radar recognition probability, visual recognition probability, and pressure recognition probability includes:

[0095]

[0096]

[0097] in, This represents the amount of evidence assigned by the i-th sensor to the propositional subset L; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This represents the recognition probability corresponding to the i-th sensor; This represents the environmental conditioning coefficient corresponding to the i-th sensor.

[0098] In one embodiment, the processing analysis unit calculates the local conflict factor between evidence quantities, including:

[0099] in, This represents the local conflict factor between the i-th sensor and the j-th sensor; This represents the amount of evidence assigned by the i-th sensor to the proposition subset A; Let A represent the amount of evidence assigned by the j-th sensor to propositional subset B; propositional subsets A and B are one of propositional subsets L, O, and E.

[0100] In one embodiment, the processing and analysis unit calculates the joint probability of life existence based on the global joint conflict factor and the amount of new evidence, including: Calculate the joint probability of the existence of life for a subset L of propositions. :

[0101]

[0102] in, Let A, B, and C represent the amount of evidence assigned by the first, second, and third sensors to propositional subsets A, B, and C, respectively, where propositional subsets A, B, and C are one of propositional subsets L, O, and E. This represents the global joint conflict factor.

[0103] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the above steps.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0105] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0107] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A vehicle-mounted life monitoring method using multimodal perception fusion, characterized in that, The method includes: Acquire raw sensing data from inside the cockpit, including point cloud data and spectrum data collected by the first sensor, image data collected by the second sensor, and seat pressure data collected by the third sensor. The point cloud data is projected onto the image data inside the cockpit, and spatiotemporal alignment is performed using a global clock source to obtain the target associated region; The radar recognition probability is obtained based on the spectral data corresponding to the target association region; the visual recognition probability is obtained based on the image data corresponding to the target association region; and the pressure recognition probability is obtained based on the pressure data. The radar recognition probability, the visual recognition probability, and the pressure recognition probability are fused together to calculate the fused joint probability of life existence. Based on the joint probability of the existence of the life forms, a corresponding level of vehicle control strategy is generated.

2. The method according to claim 1, characterized in that, The step of projecting the point cloud data onto the image data within the cockpit and performing spatiotemporal alignment using a global clock source to obtain the target associated region includes: The point cloud data is filtered according to the set neighborhood radius; Clustering algorithms were used to cluster and analyze the filtered point cloud data to obtain the geometric centroid coordinates of each organism. The geometric centroid coordinates are transformed from the radar rectangular coordinate system to the image pixel coordinate system, and spatiotemporal alignment is performed using a global clock source to obtain the target associated region.

3. The method according to claim 2, characterized in that, If the time difference between the geometric centroid coordinates and the corresponding frame of the image data is greater than a threshold, then linear interpolation is used to predict and compensate the geometric centroid coordinates.

4. The method according to claim 1, characterized in that, The step of fusing the radar recognition probability, the visual recognition probability, and the pressure recognition probability to calculate the fused joint probability of life existence includes: Define a state space, which is constructed from multiple mutually exclusive subsets of propositions, which are used to represent the life state of a seat; Based on each of the aforementioned proposition subsets and environmental adjustment coefficients, calculate the amount of evidence for the radar recognition probability, the visual recognition probability, and the pressure recognition probability; Calculate the local conflict factor among the evidence quantities, and adjust the environmental adjustment coefficient according to the local conflict factor to calculate the new evidence quantity; Based on the new amount of evidence, the global joint conflict factor among the amounts of evidence is calculated; The joint probability of the existence of life is calculated based on the global joint conflict factor and the new amount of evidence.

5. The method according to claim 4, characterized in that, The state space is as follows: Where L represents the subset of propositions in which seats contain living beings, O represents the subset of propositions in which seats contain inanimate objects, and E represents the subset of propositions in which seats are vacant.

6. The method according to claim 5, characterized in that, The evidence quantity for calculating the radar recognition probability, the visual recognition probability, and the pressure recognition probability based on each of the proposition subsets and the environmental adjustment coefficient includes: in, This represents the amount of evidence assigned by the i-th sensor to the propositional subset L; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This indicates that the i-th sensor is related to the subset of propositions. The amount of evidence allocated; This represents the recognition probability corresponding to the i-th sensor; This represents the environmental conditioning coefficient corresponding to the i-th sensor.

7. The method according to claim 6, characterized in that, The calculation of the local conflict factor among the evidence quantities includes: in, This represents the local conflict factor between the i-th sensor and the j-th sensor; This represents the amount of evidence assigned by the i-th sensor to the proposition subset A; Let A represent the amount of evidence assigned by the j-th sensor to propositional subset B; propositional subsets A and B are one of propositional subsets L, O, and E.

8. The method according to claim 6, characterized in that, The calculation of the joint probability of life existence based on the global joint conflict factor and the new amount of evidence includes: Calculate the joint probability of the existence of life for a subset L of propositions. : in, Let A, B, and C represent the amount of evidence assigned by the first, second, and third sensors to propositional subsets A, B, and C, respectively, where propositional subsets A, B, and C are one of propositional subsets L, O, and E. This represents the global joint conflict factor.

9. A multimodal perception fusion vehicle-mounted life monitoring system, characterized in that, include: Multimodal sensing unit, used to collect raw sensing data inside the cockpit; The processing and analysis unit is used to perform the steps of the method according to any one of claims 1 to 8 to generate a corresponding level of vehicle control strategy; The strategy execution unit is used to execute the vehicle control strategy.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images