A multi-modal fusion cockpit collaborative recognition and intention prediction control system and method

Abstract

This invention discloses a multimodal fusion-based cockpit collaborative recognition and intent prediction control system and method, belonging to the field of vehicle cockpit intelligent control technology. The system includes at least two in-vehicle image acquisition devices, a millimeter-wave radar, an in-vehicle control module, a vehicle infotainment system, and a display terminal. The in-vehicle control module communicates with each device. A memory pre-stores a face template library and an account mapping table. The processor executes programs to achieve multimodal data synchronization, occupant recognition, and account binding. The method achieves occupant recognition through region division and multimodal fusion, combines cross-view and radar assistance for identity tracking, determines the control priority of shared devices based on dynamic state vectors and intent prediction models, and can reassess and smoothly switch device settings according to cockpit status. This approach improves the robustness of occupant recognition, solves the problem of device control conflicts in multi-occupant scenarios, and realizes proactive predictive intelligent cockpit control.

Description

Technical Field

[0001] This invention relates to the field of vehicle intelligent control technology, specifically to a multimodal fusion cockpit collaborative recognition and intent prediction control system and method. Background Technology

[0002] With the rapid development of automotive intelligent technology, the level of intelligence in the vehicle cabin has become an important indicator for measuring the competitiveness of automotive products. As a core component of cabin intelligence, the in-vehicle monitoring system (OMS) is widely used in scenarios such as facial recognition, occupant status monitoring, and personalized control. Currently, some high-end models have installed OMS cameras in the rearview mirror, B-pillar, and other locations to achieve basic occupant recognition and status detection functions.

[0003] However, existing in-vehicle cabin monitoring systems and control methods still have many technical shortcomings, making it difficult to meet the intelligent interaction needs in multi-occupant and complex usage scenarios: First, existing systems mostly rely on a single visual sensor for occupant identification. In scenarios involving changes in occupant position, drastic changes in lighting, or facial occlusion, the recognition success rate drops significantly. Furthermore, in completely dark environments, the visual sensor completely fails, resulting in incomplete recognition coverage and an inability to achieve stable identity verification. Second, in multi-occupant scenarios, especially when multiple occupants are in the rear seats and logged into different vehicle systems, the in-vehicle air conditioning, audio system, rear screens, and other devices... First, the logic for switching personalized settings is unclear, which can easily lead to setting conflicts and affect the user experience. Second, the existing system lacks a dynamic intelligent priority sorting mechanism for the control permissions of multiple occupants, and cannot automatically predict and determine the control subject of shared devices when multiple users coexist, resulting in chaotic device control in multi-occupant scenarios. Third, the existing cockpit control methods are mostly "reactive" controls, which only respond to the user's explicit operation or simple status detection, lacking a deep understanding and prediction ability of the user's control intentions, and cannot achieve natural and proactive cockpit interaction, making it difficult to meet the user's experience needs for an intelligent cockpit. Summary of the Invention

[0004] To address the aforementioned technical challenges, this invention proposes a multimodal fusion-based cockpit collaborative recognition and intent prediction control system and method. By integrating multimodal fusion of visual sensors and millimeter-wave radar with cross-view collaborative recognition, intent prediction models, and priority arbitration mechanisms, it achieves stable occupant identification in complex scenarios and intelligent device control in multi-occupant scenarios. This upgrades cockpit control from passive response to proactive perception and prediction, enhancing the intelligent interactive experience of the cockpit.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a multimodal fusion cockpit collaborative recognition and intent prediction control system, comprising at least two in-vehicle image acquisition devices, a millimeter-wave radar, an in-vehicle control module, a vehicle infotainment system, and a display terminal; the in-vehicle control module is communicatively connected to the in-vehicle image acquisition devices, the millimeter-wave radar, the vehicle infotainment system, and the display terminal, respectively;

[0007] The millimeter-wave radar is used to cover the rear seat area of ​​the vehicle and collect information on the presence status, vital signs, body posture, and micro-movements or gestures of the occupants in each rear seat area; the vital signs include heart rate and respiratory rate.

[0008] The in-vehicle control module includes: a processor and a memory;

[0009] The memory stores program instructions that can be executed by the processor, and pre-stores a face template library and an account mapping table; the face template library is used to store occupant face feature vectors; the account mapping table is used to associate face identity identifiers with vehicle system accounts;

[0010] When the processor executes program instructions, it is used to synchronize the video stream acquired by the in-vehicle image acquisition device and the data acquired by the millimeter-wave radar in time.

[0011] Establish an in-vehicle coordinate system and map the facial targets captured by the in-vehicle image acquisition device and the targets detected by the millimeter-wave radar to the corresponding seat areas;

[0012] Multimodal feature fusion is performed based on the video stream and the millimeter-wave radar data to complete the collaborative identification of occupants.

[0013] The system queries the account mapping table based on the identified facial identity to obtain the corresponding vehicle system account, and then establishes a binding relationship between the vehicle system account and the display terminal associated with the corresponding seat.

[0014] Personalized settings matching the corresponding occupants are loaded onto the seat-specific devices, and the control priority of the shared devices is determined and control is executed based on the dynamic state vector and intention prediction results of the rear occupants.

[0015] When the confidence level decreases, the target is occluded, the occupant's posture changes, the occupant's position changes, or the local recognition fails, the continuity of occupant identification tracking and equipment control is maintained based on the collaborative recognition between different in-vehicle image acquisition devices and the auxiliary presence confirmation of the millimeter-wave radar.

[0016] Optionally, the in-vehicle image acquisition device includes a left-side OMS camera mounted on the left B-pillar of the vehicle and a right-side OMS camera mounted on the right B-pillar of the vehicle.

[0017] The millimeter-wave radar is installed in the rear roof of the vehicle;

[0018] The display terminal includes a left rear position display terminal and a right rear position display terminal.

[0019] Optionally, when the processor executes the program instructions, it is further configured to:

[0020] A dynamic state vector sequence is constructed from the multimodal perception data of each rear passenger over a preset time period.

[0021] The dynamic state vector sequence is input into a pre-trained temporal intent prediction model, which outputs the probability of each occupant's control intent on each shared device within a future preset time window.

[0022] The target occupant is determined based on the probability of each occupant's intention to control the same shared device.

[0023] When the difference in the probability of two or more occupants' intention to control the same shared device is less than a preset difference threshold, the target occupant is determined according to the preset arbitration rules.

[0024] The preset arbitration rules include: priority for higher identification confidence, priority for preset main user, priority for the user who most recently received manual confirmation, and priority for the actual operator of the shared device most recently.

[0025] Optionally, the seat-specific equipment includes: a left rear-position display terminal, a right rear-position display terminal, an independent seat control unit, and a partial reading light;

[0026] The shared equipment includes: rear air conditioning, interior ambient lighting, and rear audio system;

[0027] The personalized settings include: seat parameters, desktop settings, entertainment information, lighting parameters, sound parameters, and air conditioning parameters.

[0028] Secondly, the present invention provides a multimodal fusion cockpit collaborative recognition and intent prediction control method, the method comprising:

[0029] The rear seat area of ​​the vehicle is divided into identification zones and assigned responsibilities. An in-vehicle coordinate system is established, and the facial targets captured by the in-vehicle image acquisition device and the targets detected by the millimeter-wave radar are mapped to the corresponding seat areas.

[0030] The video streams from at least two in-vehicle image acquisition devices and the data from millimeter-wave radar are synchronized and preprocessed in time, and the occupant identity is collaboratively identified through multimodal feature fusion.

[0031] Passenger identification is tracked based on cross-view assisted identification and millimeter-wave radar-assisted presence confirmation, while maintaining the continuity of equipment control;

[0032] The system queries the account mapping table based on the identified facial identity, obtains the corresponding vehicle system account, and establishes a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat.

[0033] The equipment in the cabin is classified and layered personalized control is implemented. The seat-specific equipment is loaded with the personalized settings of the corresponding occupant. A dynamic state vector is constructed for each occupant in the rear row. The probability of the occupant's control intention for the shared equipment is output through the intention prediction model. The control priority of the shared equipment is determined by the arbitration rules and the control is executed.

[0034] The reassessment mechanism is triggered based on changes in the cabin status to re-determine the control priority of shared equipment and perform a smooth switch of equipment settings when the priority changes.

[0035] Optionally, the identification area division and responsibility allocation for the rear seat area of ​​the vehicle includes: dividing the rear seat area of ​​the vehicle into a left rear identification area and a right rear identification area, with the left OMS camera having the left rear identification area as the main responsibility area and the right rear identification area as the auxiliary coverage area, and the right OMS camera having the right rear identification area as the main responsibility area and the left rear identification area as the auxiliary coverage area.

[0036] The in-vehicle coordinate system is established based on the geometric position of the seat, the installation pose of the in-vehicle image acquisition device, the installation pose of the millimeter-wave radar, and preset calibration parameters.

[0037] The video stream output by the in-vehicle image acquisition device is preprocessed, including: performing face detection on the video stream to obtain candidate face bounding boxes, seat area labels and image quality parameters, calculating the overall image quality score based on the image quality parameters, and extracting the facial feature vectors of the candidate faces;

[0038] The overall image quality score is calculated using a weighted summation method, where the sum of the weight coefficients corresponding to each image quality parameter is 1.

[0039] The image quality parameters include sharpness, brightness suitability, occlusion degree, and pose alignment.

[0040] Optionally, the occupant identity collaborative recognition through multimodal feature fusion includes: when the overall image quality score of a single viewpoint is higher than that of another viewpoint and the corresponding recognition confidence score is higher than a preset threshold, the facial feature vector of that viewpoint is selected as the recognition input; when both viewpoints meet the recognition conditions, a feature-level weighted fusion method based on the overall image quality score and the feature discriminative score is used to obtain the fused feature vector.

[0041] The fused feature vector is compared with a pre-stored face template library for similarity. When the highest similarity is not lower than a preset threshold, the identity of the corresponding passenger is determined and a face identity identifier is generated.

[0042] Optionally, determining the corresponding occupant's identity and generating a facial identity identifier includes: when the in-vehicle image acquisition device fails to detect the target face in its main responsibility area, loses tracking, or has a recognition confidence level lower than a preset threshold, it calls on image data acquired by another in-vehicle image acquisition device at the same or adjacent time to perform auxiliary recognition.

[0043] When all the in-vehicle image acquisition devices fail to effectively recognize a face, the occupant's presence status and vital signs information acquired by millimeter-wave radar are used to maintain the user identity already bound to that seat.

[0044] When the assisted identification is successful and the identity result is consistent with the historically tracked identity, the original identity binding is maintained; when the identification result is inconsistent with the historically tracked identity, the system enters a reconfirmation state, and the account switching is performed only after obtaining the same identity result in multiple consecutive frames of detection.

[0045] After successful occupant identification, an identity tracking record is generated or updated for the target occupant. The identity tracking record includes: occupant identification, current seat position, most recent identification timestamp, identification confidence level, number of consecutive stable frames, and device binding status.

[0046] Optionally, the step of querying the account mapping table based on the identified facial identity to obtain the corresponding vehicle system account includes: querying the local or cloud-based account mapping table based on the identified facial identity to obtain the bound vehicle system account, and establishing a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat;

[0047] When a first-time passenger without a linked account is detected, the system enters temporary visitor mode, assigns a visitor ID, and loads default settings.

[0048] Optionally, determining the control priority of the shared device and executing control in conjunction with arbitration rules includes: ranking the probability of each occupant's control intention for a specific shared device, and selecting the occupant with the highest score as the target occupant for control of the shared device;

[0049] When the difference in the probability of control intent of two or more occupants is less than a preset difference threshold, the final priority occupant is determined according to the preset arbitration rules.

[0050] The cabin status changes that trigger the reassessment mechanism include: occupants getting on or off the vehicle, changing seats, removal of obstructions, opening and closing of doors, and changes in vehicle operating status.

[0051] The smooth switching method is as follows: the operating parameters of the shared equipment are gradually adjusted within a preset transition time, and the key safety settings of the previous priority occupant are retained during the switching period to prevent unauthorized overwriting.

[0052] Compared with the closest prior art, the present invention has the following beneficial effects:

[0053] This invention proposes a multimodal fusion-based cockpit collaborative recognition and intent prediction control system and method. By setting up at least two in-vehicle image acquisition devices in conjunction with millimeter-wave radar, it achieves the fusion of visual and non-visual multimodal perception data. Combined with a cross-view collaborative recognition mechanism, it effectively solves the problem of poor recognition performance of a single visual sensor in complex scenarios such as changes in light, target occlusion, and changes in occupant posture / position. The presence confirmation function of millimeter-wave radar can also maintain occupant identity binding in scenarios where visual information is completely lost, such as in the absence of light, which greatly improves the all-weather robustness and continuity of occupant identity recognition.

[0054] This invention constructs a dynamic state vector and temporal intent prediction model based on multimodal perception data, which can accurately predict the probability of each rear passenger's control intent on shared devices. Combined with preset arbitration rules, it realizes intelligent determination of the control priority of shared devices, solving the problems of conflicting personalized settings and chaotic control logic of shared devices in multi-passenger scenarios.

[0055] This invention categorizes in-cabin equipment into seat-specific devices and shared devices, and implements a hierarchical personalized control strategy. Seat-specific devices directly load the personalized settings of the corresponding occupant, while shared devices are controlled based on intent prediction and priority arbitration results. At the same time, a smooth switching mechanism for device settings is set up, which gradually adjusts the device operating parameters when the priority changes, retains key safety settings, avoids poor experience caused by parameter abrupt changes, and improves the cabin interaction experience in multi-occupant scenarios.

[0056] This invention can enter a reconfirmation state when the identification result is inconsistent with the historical identity. It requires multiple consecutive frames to detect the same identity result before the account switching is performed, which effectively reduces the device control errors caused by short-term misidentification. At the same time, it sets a temporary visitor mode for passengers who have not bound an account, ensuring the system's compatibility and stability and not affecting the normal use of other passengers.

[0057] The system of this invention adopts at least two in-vehicle image acquisition devices, which not only supports the core solution of dual B-pillar OMS cameras, but also leaves sufficient protection space for expansion solutions such as three cameras and multiple cameras. At the same time, it can combine infrared sensors, voiceprint recognition, Bluetooth device matching and other methods to enhance recognition capabilities. The models used in the method are all lightweight time-series models suitable for in-vehicle edge computing platforms, which can realize fast inference on the vehicle terminal, adapt to the in-vehicle computing platforms of different models, and have good scalability and engineering application value. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0059] Figure 1 This is a structural block diagram of the multimodal fusion cockpit collaborative recognition and intent prediction control system provided by the present invention;

[0060] Figure 2 This is a flowchart of a multimodal fusion cockpit collaborative recognition and intent prediction control method provided by the present invention. Detailed Implementation

[0061] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention.

[0062] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0063] This invention provides a multimodal fusion-based cockpit collaborative recognition and intent prediction control system and method. It addresses issues such as poor recognition performance of single visual sensors, conflicts in multi-occupant scenario settings, lack of dynamic priority ranking, and insufficient intent prediction capabilities, thereby improving the accuracy and all-weather robustness of cockpit occupant recognition and achieving proactive predictive intelligent personalized cockpit control.

[0064] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0065] Example 1: See Figure 1 This embodiment 1 provides a multimodal fusion cockpit collaborative recognition and intent prediction control system, including at least two in-vehicle image acquisition devices; as well as millimeter-wave radar, in-vehicle control module, vehicle system and display terminal.

[0066] The in-vehicle image acquisition device is preferably configured as an OMS camera. In this embodiment, the in-vehicle image acquisition device includes a left-side OMS camera mounted on the left B-pillar of the vehicle and a right-side OMS camera mounted on the right B-pillar of the vehicle, used to acquire image information of the left rear seat area and the right rear seat area, respectively.

[0067] Millimeter-wave radar is installed in the rear roof of the vehicle to cover the rear area and collect information on the presence, vital signs, body posture, and subtle movements or gestures of the occupants in each rear seat area. The vital signs include heart rate and respiratory rate.

[0068] The in-vehicle control module is communicatively connected to the in-vehicle image acquisition device, millimeter-wave radar, vehicle infotainment system, and display terminal. The display terminal may include a left rear-position display terminal and a right rear-position display terminal.

[0069] The in-vehicle control module includes a processor and a memory. The memory stores program instructions executable by the processor and pre-stores a face template library and an account mapping table. The face template library stores occupant facial feature vectors, and the account mapping table associates facial identification with the vehicle's account information.

[0070] In one embodiment, for example, the face template library stores the facial feature vectors of the father, mother, and child, and the account mapping table associates the facial identity of each occupant with the corresponding personal vehicle system account. Each vehicle system account has preset personalized settings: the father's personalized settings include driving preferences, news channels, air conditioning at 26°C, and medium volume audio; the mother's personalized settings include rest seat mode, soft music, air conditioning at 24°C, and soft ambient lighting; the child's personalized settings include child seat mode, cartoons, a child-specific desktop, and a constantly lit left rear partial reading light.

[0071] The processor executes program instructions to achieve all functions, including multimodal data time synchronization, in-vehicle coordinate system establishment, occupant identity collaborative recognition, vehicle-mounted account binding, device hierarchical control, intent prediction and priority arbitration, and identity tracking continuity maintenance.

[0072] Specifically, when a processor executes program instructions, it can perform the following processes:

[0073] Time synchronization is performed between the video stream acquired by the in-vehicle image acquisition device and the data acquired by the millimeter-wave radar.

[0074] Establish an in-vehicle coordinate system and map visual and radar targets to the corresponding seating areas;

[0075] Multimodal feature fusion based on video stream and millimeter-wave radar data is used to complete collaborative identification of occupants.

[0076] The system queries the account mapping table based on the identified facial identity, obtains the corresponding vehicle system account, and establishes a binding relationship between the vehicle system account and the display terminal associated with the corresponding seat.

[0077] Personalized settings matching the corresponding occupants are loaded onto the seat-specific devices, and the control priority of the shared devices is determined and control is executed based on the dynamic state vector and intention prediction results of the rear occupants.

[0078] When the confidence level of recognition decreases, the target is occluded, the occupant's posture changes, the occupant's position changes, or the local recognition fails, the continuity of occupant identification tracking and equipment control is maintained based on the collaborative recognition between different in-vehicle image acquisition devices and the presence confirmation assisted by millimeter-wave radar.

[0079] Accordingly, Embodiment Two: Based on the same technical concept, this embodiment also provides a multimodal fusion cockpit collaborative recognition and intent prediction control method corresponding to Embodiment One above, such as... Figure 2 As shown, the method includes:

[0080] S101 identifies and divides the rear seat area of ​​the vehicle and assigns responsibility, establishes an in-vehicle coordinate system, and maps the facial targets captured by the in-vehicle image acquisition device and the targets detected by the millimeter-wave radar to the corresponding seat area.

[0081] S102 performs time synchronization and preprocessing of video streams from at least two in-vehicle image acquisition devices and data from millimeter-wave radar, and completes collaborative identification of occupants through multimodal feature fusion;

[0082] S103 achieves occupant identification tracking and maintains the continuity of equipment control based on cross-view assisted identification and millimeter-wave radar assisted presence confirmation;

[0083] S104 queries the account mapping table based on the identified facial identity to obtain the corresponding vehicle system account, and establishes a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat;

[0084] S105 classifies in-cabin equipment and performs hierarchical personalized control, with seat-specific equipment loading the corresponding occupant's personalized settings; a dynamic state vector is constructed for each occupant in the rear row, and the probability of the occupant's control intention for shared equipment is output through an intent prediction model. Combined with arbitration rules, the control priority of shared equipment is determined and control is executed.

[0085] S106 triggers a reassessment mechanism based on changes in the cabin status, re-determines the control priority of shared equipment, and performs a smooth switch of equipment settings when the priority changes.

[0086] In step S101 above, the identification area division and responsibility allocation for the rear seat area of ​​the vehicle includes: dividing the rear seat area of ​​the vehicle into a left rear identification area and a right rear identification area, with the left OMS camera having the left rear identification area as the main responsibility area and the right rear identification area as the auxiliary coverage area, and the right OMS camera having the right rear identification area as the main responsibility area and the left rear identification area as the auxiliary coverage area.

[0087] The in-vehicle coordinate system is established based on the geometric position of the seat, the installation pose of the in-vehicle image acquisition device, the installation pose of the millimeter-wave radar, and preset calibration parameters.

[0088] The video stream output by the in-vehicle image acquisition device is preprocessed, including: performing face detection on the video stream to obtain candidate face bounding boxes, seat area labels and image quality parameters, calculating the overall image quality score based on the image quality parameters, and extracting the facial feature vectors of the candidate faces;

[0089] The overall image quality score is calculated using a weighted summation method, where the sum of the weight coefficients corresponding to each image quality parameter is 1.

[0090] The image quality parameters include sharpness, brightness suitability, occlusion degree, and pose alignment.

[0091] In step S102 above, the occupant identity collaborative recognition through multimodal feature fusion includes: when the overall image quality score of a single viewpoint is higher than that of another viewpoint and the corresponding recognition confidence is higher than a preset threshold, the face feature vector of that viewpoint is selected as the recognition input; when both viewpoints meet the recognition conditions, a feature-level weighted fusion method based on the overall image quality score and the feature discriminative score is used to obtain the fused feature vector.

[0092] The fused feature vector is compared with a pre-stored face template library for similarity. When the highest similarity is not lower than a preset threshold, the identity of the corresponding passenger is determined and a face identity identifier is generated.

[0093] In the above embodiments, determining the corresponding occupant's identity and generating a facial identity identifier includes: when the in-vehicle image acquisition device fails to detect the target face in its main responsibility area, loses tracking, or has a recognition confidence level lower than a preset threshold, it calls on image data acquired by another in-vehicle image acquisition device at the same or adjacent time to perform auxiliary recognition.

[0094] When all the in-vehicle image acquisition devices fail to effectively recognize a face, the occupant's presence status and vital signs information acquired by millimeter-wave radar are used to maintain the user identity already bound to that seat.

[0095] When the assisted identification is successful and the identity result is consistent with the historically tracked identity, the original identity binding is maintained; when the identification result is inconsistent with the historically tracked identity, the system enters a reconfirmation state, and the account switching is performed only after obtaining the same identity result in multiple consecutive frames of detection.

[0096] After successful occupant identification, an identity tracking record is generated or updated for the target occupant. The identity tracking record includes: occupant identification, current seat position, most recent identification timestamp, identification confidence level, number of consecutive stable frames, and device binding status.

[0097] In step S104 above, the step of querying the account mapping table based on the identified facial identity to obtain the corresponding vehicle system account includes: querying the local or cloud account mapping table based on the identified facial identity to obtain the bound vehicle system account, and establishing a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat.

[0098] When a first-time passenger without a linked account is detected, the system enters temporary visitor mode, assigns a visitor ID, and loads default settings.

[0099] In the above embodiments, determining the control priority of the shared device and executing control in conjunction with arbitration rules includes: ranking the probability of each occupant's control intention on a specific shared device, and selecting the occupant with the highest score as the control target occupant of the shared device;

[0100] When the difference in the probability of control intent of two or more occupants is less than a preset difference threshold, the final priority occupant is determined according to the preset arbitration rules.

[0101] The cabin status changes that trigger the reassessment mechanism include: occupants getting on or off the vehicle, changing seats, removal of obstructions, opening and closing of doors, and changes in vehicle operating status.

[0102] The smooth switching method is as follows: the operating parameters of the shared equipment are gradually adjusted within a preset transition time, and the key safety settings of the previous priority occupant are retained during the switching period to prevent unauthorized overwriting.

[0103] Example 3: Establishing the Rear Seat Area Division and In-Vehicle Coordinate System

[0104] In this embodiment, the rear area of ​​the vehicle is divided into a left rear-view recognition area and a right rear-view recognition area. The left OMS camera has the left rear-view recognition area as its primary responsibility area and the right rear-view recognition area as its secondary coverage area; the right OMS camera has the right rear-view recognition area as its primary responsibility area and the left rear-view recognition area as its secondary coverage area.

[0105] The in-vehicle control module establishes a unified in-vehicle coordinate system based on the seat geometry, the installation orientation of the in-vehicle image acquisition device, the installation orientation of the millimeter-wave radar, and preset calibration parameters. This coordinate system allows for the mapping of facial and radar targets detected by different sensing devices onto a unified spatial framework, and the determination of the target's seating area.

[0106] For example, the in-vehicle coordinate system can be based on the center floor area of ​​the rear seats as the origin, with the lateral direction as the X-axis, the longitudinal direction as the Y-axis, and the vertical direction as the Z-axis. The camera imaging coordinate system and the radar detection coordinate system are converted into the in-vehicle coordinate system through calibration parameters, thereby achieving multi-source data alignment.

[0107] Example 4: Video Stream Preprocessing and Image Quality Scoring

[0108] In this embodiment, the video stream output by the in-vehicle image acquisition device is preprocessed, including:

[0109] Perform face detection on the video stream to obtain candidate face bounding boxes;

[0110] Determine seat area labels based on target location;

[0111] Calculate image quality parameters;

[0112] Calculate the overall image quality score based on image quality parameters;

[0113] Extract the facial feature vectors of the candidate faces.

[0114] The image quality parameters include sharpness, brightness suitability, occlusion, and pose alignment. The overall image quality score is calculated using a weighted summation method, where the sum of the weight coefficients for each image quality parameter is 1. For example, the overall image quality score Q can be expressed as:

[0115] Q=w1×S1+w2×S2+w3×S3+w4×S4

[0116] Where Q∈[0,1], S1 represents the sharpness score, S2 represents the brightness suitability score, S3 represents the occlusion score, S4 represents the pose alignment score, and w1, w2, w3, and w4 are the corresponding weight coefficients, satisfying the following:

[0117] w1 + w2 + w3 + w4 = 1.

[0118] In one example, sharpness can be evaluated by image gradient, Laplacian variance, etc.; brightness suitability can be evaluated by the degree of matching between the image grayscale distribution and the preset brightness range; occlusion can be evaluated by the visible proportion of key face areas; and pose alignment can be evaluated by the difference between the head pose angle and the facing angle.

[0119] Example 5: Multimodal Feature Fusion and Collaborative Identity Recognition

[0120] In this embodiment, when the overall image quality score of a single viewpoint is higher than that of another viewpoint and the corresponding recognition confidence score is higher than a preset threshold, the facial feature vector of that viewpoint is directly selected as the recognition input; when both viewpoints meet the recognition conditions, a feature-level weighted fusion method based on the overall image quality score and the feature discriminative score is used to obtain the fused feature vector.

[0121] For example, for a candidate face of the same occupant captured from two camera viewpoints, facial feature vectors FL and FR are extracted respectively. When the overall image quality score Q of one viewpoint is higher than that of the other viewpoint and the corresponding recognition confidence C is higher than a preset threshold T2, the feature vector of that viewpoint is selected as the current recognition input; when both viewpoints meet the recognition conditions, a feature-level weighted fusion based on quality and feature discriminability is used, and the fused feature vector F can be expressed as:

[0122] ;

[0123] Among them, D L and D R These are the discriminative scores for the feature vectors from the left and right perspectives, respectively, used to measure the uniqueness and information content of the features; Q L Q is the overall quality score for the face image from the left-hand perspective. R The overall quality score for the face image from the right-side view is given by α and β, which are adjustable weighting coefficients.

[0124] Next, the fused feature vector is compared with a pre-stored face template library for similarity. When the highest similarity is not lower than a preset threshold, the identity of the corresponding passenger is determined and a face identity identifier is generated.

[0125] In this embodiment, multimodal feature fusion not only includes dual-view image feature fusion, but also utilizes the target presence status, seat area location and attitude information of millimeter-wave radar as auxiliary constraints to eliminate unreasonable face-seat matching results and improve the accuracy of identity recognition.

[0126] Example 6: Cross-view assisted identification and millimeter-wave radar assisted confirmation

[0127] During actual vehicle travel, passengers may turn their heads, look down, obstruct their view, get on or off the vehicle, or change seats, which could cause the camera in the primary responsible area to fail to recognize the passenger. Therefore, this embodiment introduces a cross-view auxiliary recognition and millimeter-wave radar-assisted confirmation mechanism.

[0128] When the in-vehicle image acquisition device fails to detect, loses tracking, or has a recognition confidence level below a preset threshold in the main responsibility area of ​​this side, it calls on image data acquired by another in-vehicle image acquisition device at the same or adjacent time to perform auxiliary recognition.

[0129] When all in-vehicle image acquisition devices fail to effectively recognize faces, the occupant's presence status and vital signs information collected by millimeter-wave radar are used to maintain the user identity already bound to that seat, without immediately triggering account switching or device configuration rollback.

[0130] When the assisted identification is successful and the identity result is consistent with the historically tracked identity, the original identity binding is maintained; when the identification result is inconsistent with the historically tracked identity, the system enters a reconfirmation state, and the account switching is performed only after obtaining the same identity result in multiple consecutive frames of detection, so as to reduce the erroneous switching caused by short-term misidentification.

[0131] Simultaneously, after successful occupant identification, an identity tracking record is generated or updated for the target occupant. This identity tracking record includes: occupant identification identifier, current seat location, most recent identification timestamp, identification confidence level, number of consecutive stable frames, and device binding status. The system can determine the stability of the current binding status based on the identity tracking record, serving as a crucial basis for whether to trigger device switching.

[0132] Example 7: Automatic Account Binding and Guest Mode

[0133] In this embodiment, the system queries the local or cloud account mapping table based on the identified facial identity to obtain the bound vehicle system account, and establishes a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat.

[0134] For example, when the passenger in the left rear seat is identified as user A, the system can automatically bind user A's corresponding vehicle infotainment account to the left rear display terminal and load user A's desktop settings, entertainment preferences, volume parameters, and other content. When the passenger in the right rear seat is identified as user B, the right rear display terminal loads user B's corresponding configuration.

[0135] When a first-time passenger without a linked account is detected, the system enters temporary visitor mode, assigns a visitor ID, and loads default settings to ensure that unregistered users can still use the rear-seat devices normally.

[0136] Example 8: Hierarchical Personalized Control of Equipment

[0137] In this embodiment, the system categorizes in-cabin equipment into two types: seat-specific equipment and shared equipment.

[0138] Among them, seat-specific equipment includes a left rear-side display terminal, a right rear-side display terminal, an independent seat control unit, and partial reading lights; shared equipment includes rear air conditioning, interior ambient lighting, and rear audio systems.

[0139] Once occupant identification and account binding are complete, the system directly loads the corresponding occupant's personalized settings onto the seat-specific devices. These personalized settings include seat parameters, desktop settings, entertainment information, lighting parameters, sound parameters, and air conditioning parameters.

[0140] For example, when the occupant in the left rear seat is a frequent user, the left rear seat display terminal can automatically restore its personalized desktop, frequently used application entries and viewing history, the independent seat control unit can restore its frequently used sitting posture parameters, and the local reading light can restore its preferred brightness level.

[0141] Example 9: Dynamic State Vector Construction and Intent Prediction

[0142] In this embodiment, a dynamic state vector is constructed for each rear passenger. The dynamic state vector can be generated based on multimodal sensing data from a preset time period, including but not limited to the following parameters:

[0143] Passenger identification, recognition confidence level, historical interaction activity, real-time gaze direction, body posture, vital signs status, micro-movement or gesture information, current seat position, and recent shared device interaction records.

[0144] Then, the dynamic state vector sequence is input into the pre-trained temporal intent prediction model, which outputs the probability of each occupant's control intent on each shared device within a future preset time window.

[0145] The time-series intention prediction model can be a recurrent neural network model, a long short-term memory network model, a gated recurrent unit model, a time-series convolutional network model, or other models capable of processing time-series data.

[0146] For example, if the system detects that the passenger in the left rear seat is continuously looking at the rear air conditioning control interface, leaning forward, and making slight movements with their hands toward the control area, and combines this with historical interaction activity to determine that the passenger is likely to operate the rear air conditioning, then the probability of their intention to control the rear air conditioning can be increased.

[0147] Specifically, multimodal state vector construction: When there are multiple occupants in the back row, the system constructs a dynamic state vector V that changes over time for each occupant. t This vector integrates multimodal sensing data:

[0148] V t =[ID, C, U, G] t P st P ht ];

[0149] Where ID is the identity identifier, C is the identification confidence level, U is the historical interaction activity level, and G is the identity identifier. t P is the real-time gaze direction vector estimated by the camera. st For body postures (such as leaning forward, leaning back, turning sideways) sensed by millimeter-wave radar, P ht Physiological states (such as resting or active) are sensed by millimeter-wave radar.

[0150] Intent Prediction Model: The system has a built-in pre-trained intent prediction model (such as a Temporal Convolutional Network (TCN) or a Recurrent Neural Network (RNN)). This model receives the occupant's state vector sequence {V} over a past period of time. t-k ,...,V t As input, output the occupant's interactions with various shared devices within a short future time window. i The probability P of the control intention (such as rear-seat pads, air conditioning) intent As shown in the following formula:

[0151] P intent (device i ) = Model({V t-k ,...,V t}).

[0152] Example 10: Priority Determination and Arbitration for Shared Device Control

[0153] In this embodiment, the system sorts the probability of each occupant's intention to control a specific shared device and selects the occupant with the highest score as the target occupant for controlling the shared device.

[0154] When the difference in the probability of control intent between two or more occupants is less than a preset difference threshold, the system determines the final priority occupant according to preset arbitration rules. These preset arbitration rules include: priority based on higher identification confidence, priority based on the preset primary user, priority based on the user who most recently provided manual confirmation, and priority based on the actual operator of the most recently shared device.

[0155] For example, if both the left rear passenger and the right rear passenger show a high degree of control over the rear audio system and their scores are close, the system can further compare their identification confidence, whether they are the primary user, whether they have recently obtained control through manual confirmation, and whether they are the most recent actual operator, and determine the priority passenger accordingly.

[0156] Example 11: In this example, the system is equipped with a reassessment mechanism to re-determine the control priority of shared equipment when the cabin status changes. The cabin status changes include: passenger boarding / alighting, seat changes, removal of obstructions, door opening / closing, and changes in vehicle operating status.

[0157] Once the above state changes are detected, the system re-acquires multimodal data, updates the dynamic state vector, and recalculates the control intent probability of each occupant and the control priority of shared devices.

[0158] When priorities change, the system performs a smooth transition of device settings. For example, the operating parameters of shared devices are gradually adjusted over a preset transition time, rather than changing abruptly. If the device being switched is the rear air conditioning, the temperature or airflow can be gradually adjusted over several seconds; if the device being switched is the rear audio system, the volume, sound effect preferences, or recommended content parameters can be gradually adjusted.

[0159] In addition, critical safety settings for the previous priority occupant are preserved from unauthorized overwriting during the handover period. For example, parameters related to child safety, emergency calls, or specific safety alerts remain in their original settings to avoid introducing safety risks due to a change in general control.

[0160] Example 12: In a typical application scenario, User A sits on the left rear seat and User B sits on the right rear seat. After the vehicle starts, the left and right OMS cameras respectively capture images of the rear occupants, while millimeter-wave radar simultaneously senses the presence, posture, and vital signs of the rear occupants. After completing time synchronization and target mapping, the system identifies the left and right occupants and binds User A's and User B's vehicle system accounts to the left and right rear display terminals respectively.

[0161] Subsequently, the system loads the personalized settings for users A and B onto the dedicated devices for the left and right rear seats, respectively. If user A continuously focuses on the rear air conditioning adjustment area and exhibits obvious gesturing tendencies, while user B does not show similar behavior, the system predicts that user A has a higher intention to control the rear air conditioning and assigns it control priority.

[0162] When user A briefly turns their head, causing a decrease in the recognition confidence of the left OMS camera, the system calls the right OMS camera to perform auxiliary recognition. If neither camera can recognize the face temporarily, the system maintains the identity of user A, who is already bound to the left rear position, based on the presence status of the millimeter-wave radar and vital signs information, without immediately switching the display terminal account or air conditioning control priority, thereby maintaining the continuous and stable operation of the system.

[0163] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A multi-modal fusion cockpit collaborative recognition and intent prediction control system, characterized in that, It includes at least two in-vehicle image acquisition devices, a millimeter-wave radar, an in-vehicle control module, a vehicle infotainment system, and a display terminal; the in-vehicle control module is communicatively connected to the in-vehicle image acquisition devices, the millimeter-wave radar, the vehicle infotainment system, and the display terminal, respectively; The millimeter-wave radar is used to cover the rear seat area of ​​the vehicle and collect information on the presence status, vital signs, body posture, and micro-movements or gestures of the occupants in each rear seat area; the vital signs include heart rate and respiratory rate. The in-vehicle control module includes: a processor and a memory; The memory stores program instructions that can be executed by the processor, and pre-stores a face template library and an account mapping table; the face template library is used to store occupant face feature vectors; the account mapping table is used to associate face identity identifiers with vehicle system accounts; When the processor executes program instructions, it is used to synchronize the video stream acquired by the in-vehicle image acquisition device and the data acquired by the millimeter-wave radar in time. Establish an in-vehicle coordinate system and map the facial targets captured by the in-vehicle image acquisition device and the targets detected by the millimeter-wave radar to the corresponding seat areas; Multimodal feature fusion is performed based on the video stream and the millimeter-wave radar data to complete the collaborative identification of occupants. The system queries the account mapping table based on the identified facial identity to obtain the corresponding vehicle system account, and then establishes a binding relationship between the vehicle system account and the display terminal associated with the corresponding seat. Personalized settings matching the corresponding occupants are loaded onto the seat-specific devices, and the control priority of the shared devices is determined and control is executed based on the dynamic state vector and intention prediction results of the rear occupants. When the confidence level decreases, the target is occluded, the occupant's posture changes, the occupant's position changes, or the local recognition fails, the continuity of occupant identification tracking and equipment control is maintained based on the collaborative recognition between different in-vehicle image acquisition devices and the auxiliary presence confirmation of the millimeter-wave radar.

2. The system of claim 1, wherein, The in-vehicle image acquisition device includes a left-side OMS camera installed on the left B-pillar of the vehicle and a right-side OMS camera installed on the right B-pillar of the vehicle. The millimeter-wave radar is installed in the rear roof of the vehicle; The display terminal includes a left rear position display terminal and a right rear position display terminal.

3. The system of claim 1, wherein, When the processor executes the program instructions, it is also used to: A dynamic state vector sequence is constructed from the multimodal perception data of each rear passenger over a preset time period. The dynamic state vector sequence is input into a pre-trained temporal intent prediction model, which outputs the probability of each occupant's control intent on each shared device within a future preset time window. The target occupant is determined based on the probability of each occupant's intention to control the same shared device. When the difference in the probability of two or more occupants' intention to control the same shared device is less than a preset difference threshold, the target occupant is determined according to the preset arbitration rules. The preset arbitration rules include: priority for higher identification confidence, priority for preset main user, priority for the user who most recently received manual confirmation, and priority for the actual operator of the shared device most recently.

4. The system of claim 1, wherein, The seat-specific equipment includes: a left rear-position display terminal, a right rear-position display terminal, an independent seat control unit, and a partial reading light; The shared equipment includes: rear air conditioning, interior ambient lighting, and rear audio system; The personalized settings include: seat parameters, desktop settings, entertainment information, lighting parameters, sound parameters, and air conditioning parameters.

5. A multi-modal fusion cockpit collaborative recognition and intention prediction control method, characterized in that, The method includes: The rear seat area of ​​the vehicle is divided into identification zones and assigned responsibilities. An in-vehicle coordinate system is established, and the facial targets captured by the in-vehicle image acquisition device and the targets detected by the millimeter-wave radar are mapped to the corresponding seat areas. The video streams from at least two in-vehicle image acquisition devices and the data from millimeter-wave radar are synchronized and preprocessed in time, and the occupant identity is collaboratively identified through multimodal feature fusion. Passenger identification is achieved through cross-view assisted identification and millimeter-wave radar-assisted presence confirmation, while maintaining the continuity of equipment control. The system queries the account mapping table based on the identified facial identity, obtains the corresponding vehicle system account, and establishes a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat. The equipment in the cabin is classified and layered personalized control is implemented. The seat-specific equipment is loaded with the personalized settings of the corresponding occupant. A dynamic state vector is constructed for each occupant in the rear row. The probability of the occupant's control intention for the shared equipment is output through the intention prediction model. The control priority of the shared equipment is determined by the arbitration rules and the control is executed. The reassessment mechanism is triggered based on changes in the cabin status to re-determine the control priority of shared equipment and perform a smooth switch of equipment settings when the priority changes.

6. The method of claim 5, wherein, The process of dividing and assigning responsibility for the recognition area of ​​the rear seat of the vehicle includes: dividing the rear seat area of ​​the vehicle into a left rear recognition area and a right rear recognition area; the left OMS camera takes the left rear recognition area as its primary responsibility area and the right rear recognition area as its secondary coverage area; and the right OMS camera takes the right rear recognition area as its primary responsibility area and the left rear recognition area as its secondary coverage area. The in-vehicle coordinate system is established based on the geometric position of the seat, the installation pose of the in-vehicle image acquisition device, the installation pose of the millimeter-wave radar, and preset calibration parameters. The video stream output by the in-vehicle image acquisition device is preprocessed, including: performing face detection on the video stream to obtain candidate face bounding boxes, seat area labels and image quality parameters, calculating the overall image quality score based on the image quality parameters, and extracting the facial feature vectors of the candidate faces; The overall image quality score is calculated using a weighted summation method, where the sum of the weight coefficients corresponding to each image quality parameter is 1. The image quality parameters include sharpness, brightness suitability, occlusion degree, and pose alignment.

7. The method of claim 5, wherein, The method of achieving collaborative occupant identification through multimodal feature fusion includes: when the overall image quality score of a single viewpoint is higher than that of another viewpoint and the corresponding recognition confidence score is higher than a preset threshold, the facial feature vector of that viewpoint is selected as the recognition input; when both viewpoints meet the recognition conditions, a feature-level weighted fusion method based on the overall image quality score and the feature discriminative score is used to obtain the fused feature vector. The fused feature vector is compared with a pre-stored face template library for similarity. When the highest similarity is not lower than a preset threshold, the identity of the corresponding passenger is determined and a face identity identifier is generated.

8. The method of claim 7, wherein, The process of determining the corresponding passenger identity and generating a facial identity identifier includes: when the in-vehicle image acquisition device fails to detect the target face in the main responsibility area on this side, loses tracking, or the recognition confidence level is lower than a preset threshold, it calls on image data acquired by another in-vehicle image acquisition device at the same time or adjacent time to perform auxiliary recognition. When all the in-vehicle image acquisition devices fail to effectively recognize a face, the occupant's presence status and vital signs information acquired by millimeter-wave radar are used to maintain the user identity already bound to that seat. When the assisted identification is successful and the identity result is consistent with the historically tracked identity, the original identity binding is maintained; when the identification result is inconsistent with the historically tracked identity, the system enters a reconfirmation state, and the account switching is performed only after obtaining the same identity result in multiple consecutive frames of detection. After successful occupant identification, an identity tracking record is generated or updated for the target occupant. The identity tracking record includes: occupant identification, current seat position, most recent identification timestamp, identification confidence level, number of consecutive stable frames, and device binding status.

9. The method of claim 8, wherein, The step of querying the account mapping table based on the identified facial identity to obtain the corresponding vehicle system account includes: querying the local or cloud account mapping table based on the identified facial identity to obtain the bound vehicle system account, and establishing a binding relationship between the vehicle system account and the display terminal associated with the passenger's seat; When a first-time passenger without a linked account is detected, the system enters temporary visitor mode, assigns a visitor ID, and loads default settings.

10. The method of claim 9, wherein, The process of determining the control priority of shared devices and executing control by combining arbitration rules includes: ranking the probability of each occupant's control intention for a specific shared device, and selecting the occupant with the highest score as the control target occupant of that shared device; When the difference in the probability of control intentions of two or more occupants is less than a preset difference threshold, the final priority occupant is determined according to the preset arbitration rules. The cabin status changes that trigger the reassessment mechanism include: occupants getting on or off the vehicle, changing seats, removal of obstructions, opening and closing of doors, and changes in vehicle operating status. The smooth switching method is as follows: the operating parameters of the shared equipment are gradually adjusted within a preset transition time, and the key safety settings of the previous priority occupant are retained during the switching period to prevent unauthorized overwriting.

Images