State awareness method and apparatus, electronic device, and storage medium

Abstract

The application provides a state perception method and device, electronic equipment and storage medium. The method comprises: acquiring an environment data stream, a visual image stream and a thermal imaging data stream of a vehicle; determining at least one target passenger in the vehicle based on the visual image stream and the thermal imaging data stream; mapping an environment parameter corresponding to the environment data stream to a high-dimensional semantic space to obtain an environment feature vector; for any target passenger, generating a thermal attribute feature vector corresponding to the target passenger according to the visual image stream and the thermal imaging data stream; and performing state perception on the target passenger based on the environment feature vector and the thermal attribute feature vector corresponding to the target passenger. In this way, the accuracy of passenger state perception can be improved, and the technical problem that the prior art only focuses on environment parameters and facial features, resulting in insufficient accuracy of thermal comfort level or thermal state evaluation results, is solved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more particularly to a state perception method, device, electronic device, and storage medium. Background Technology

[0002] As automotive cabins evolve towards intelligence and personalization, passenger thermal comfort control has gradually shifted from traditional automatic air conditioning based on set temperatures to human-centered intelligent thermal management. Accurate, real-time, and personalized perception of the thermal state and comfort level of cabin passengers is the prerequisite and core for achieving collaborative intelligent temperature control across multiple terminals, including air conditioning, seats, steering wheel, and air vents.

[0003] In related technologies, existing state perception methods mainly collect in-vehicle environmental data (such as air temperature and humidity) through onboard sensors, while simultaneously capturing passenger facial images through cockpit cameras. Convolutional neural networks are then used to extract facial visual features to derive thermal comfort levels or thermal state assessments based on the environmental data and visual characteristics. However, this method focuses solely on environmental parameters and facial features, leading to insufficient accuracy in the thermal comfort level or thermal state assessment results. Summary of the Invention

[0004] The purpose of this application is to provide a state-sensing method, device, electronic device, and storage medium to solve the technical problem that existing technologies, which only focus on environmental parameters and facial features, result in insufficient accuracy of thermal comfort levels or thermal state evaluation results. The specific technical solution is as follows: In a first aspect of this application, a state-aware method is provided, the method comprising: Acquire vehicle environmental data stream, visual image stream, and thermal imaging data stream; Based on the visual image stream and thermal imaging data stream, at least one target passenger inside the vehicle is identified; The environmental parameters corresponding to the environmental data stream are mapped to a high-dimensional semantic space to obtain an environmental feature vector; For any of the target passengers, a thermal attribute feature vector corresponding to the target passenger is generated based on the visual image stream and the thermal imaging data stream; Based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger, the state of the target passenger is perceived.

[0005] In an optional implementation, mapping the environmental parameters corresponding to the environmental data stream to a high-dimensional semantic space to obtain an environmental feature vector includes: The environmental parameters corresponding to the environmental data stream are normalized to generate normalized environmental parameters; The normalized environmental parameters are mapped to the high-dimensional semantic space to obtain the environmental feature vector.

[0006] In an optional implementation, generating the thermal attribute feature vector corresponding to the target passenger based on the visual image stream and the thermal imaging data stream includes: Extract the target region image corresponding to the target passenger from the visual image stream; The target region image corresponding to the target passenger is processed to obtain the face region image and clothing region image corresponding to the target passenger; Based on the face region image corresponding to the target passenger, determine the implicit face feature vector and passenger attribute label; The implicit clothing feature vector and the estimated thermal resistance of the clothing are determined based on the clothing area image corresponding to the target passenger. Based on the thermal imaging data stream and the target area image corresponding to the target passenger, an implicit thermal distribution feature vector, a first thermal sensing value, a second thermal sensing value, and a thermal sensing difference are determined, wherein the first thermal sensing value is the average temperature value of the forehead area of ​​the target passenger, and the second thermal sensing value is the average temperature value of the nose tip area of ​​the target passenger. Based on the implicit face feature vector, the passenger attribute label, the implicit clothing feature vector, the estimated thermal resistance of the clothing, the implicit thermal distribution feature vector, the first thermal perception value, the second thermal perception value, and the thermal perception difference, a thermal attribute feature vector corresponding to the target passenger is generated.

[0007] In an optional implementation, determining the implicit facial feature vector and passenger attribute label based on the facial region image corresponding to the target passenger includes: Determine the basic facial feature map based on the facial region image corresponding to the target passenger; Global average pooling is performed on the basic facial feature map to obtain the implicit facial feature vector; The passenger attribute label is determined based on the implicit facial feature vector.

[0008] In an optional implementation, determining the implicit clothing feature vector and the estimated clothing thermal resistance based on the clothing region image corresponding to the target passenger includes: Determine the basic clothing feature map based on the clothing area image corresponding to the target passenger; Global average pooling is performed on the clothing basic feature map to obtain the implicit clothing feature vector; The estimated thermal resistance of the clothing is determined based on the implicit clothing feature vector.

[0009] In an optional implementation, determining the implicit thermal distribution feature vector, the first thermal sensing value, the second thermal sensing value, and the thermal sensing difference based on the thermal imaging data stream and the target area image corresponding to the target passenger includes: Obtain the temperature distribution map corresponding to the target passenger from the thermal imaging data stream; The implicit thermal distribution feature vector is determined based on the temperature distribution map corresponding to the target passenger; Determine the coordinates of facial key points based on the target area image corresponding to the target passenger; Based on the facial key point coordinates, a first distribution area and a second distribution area are determined from the temperature distribution map corresponding to the target passenger, wherein the first distribution area is the area centered on the forehead key point, and the second distribution area is the area centered on the nose tip key point. The average temperature corresponding to the first distribution area in the temperature distribution map is taken as the first thermal sensing value. The average temperature corresponding to the second distribution area in the temperature distribution map is used as the second thermal sensing value; The difference between the first thermal sensing value and the second thermal sensing value is determined as the thermal sensing difference.

[0010] In an optional implementation, the step of sensing the state of the target passenger based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger includes: The attention weight matrix is ​​determined by using the thermal attribute feature vector as the query term and the environmental feature vector as the key term. The environmental feature vector is weighted using the attention weight matrix to obtain a weighted environmental feature vector; The thermal attribute feature vector corresponding to the target passenger is fused with the weighted environmental feature vector to obtain a multimodal fusion feature vector; Based on the multimodal fusion feature vector, the thermal state evaluation result of the target passenger is generated.

[0011] In a second aspect of this application, a state sensing device is also provided, the device comprising: The data stream acquisition module is used to acquire the vehicle's environmental data stream, visual image stream, and thermal imaging data stream; The target passenger determination module is used to determine at least one target passenger inside the vehicle based on the visual image stream and the thermal imaging data stream. An environment feature vector generation module is used to map the environment parameters corresponding to the environment data stream to a high-dimensional semantic space to obtain an environment feature vector. A thermal attribute feature vector generation module is used to generate a thermal attribute feature vector corresponding to any of the target passengers based on the visual image stream and the thermal imaging data stream. The state perception module is used to perceive the state of the target passenger based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger.

[0012] In a third aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the state-aware method described in any one of the first aspects above.

[0013] In a fourth aspect of the embodiments of this application, a storage medium is also provided, wherein the storage medium stores instructions that, when run on a computer, cause the computer to execute any of the state-aware methods described in the first aspect above.

[0014] In a fifth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the state-aware methods described in the first aspect above.

[0015] The technical solution provided in this application acquires environmental data streams, visual image streams, and thermal imaging data streams from a vehicle; based on the visual image streams and thermal imaging data streams, it identifies at least one target passenger within the vehicle; it maps the environmental parameters corresponding to the environmental data streams to a high-dimensional semantic space to obtain environmental feature vectors; for any target passenger, it generates a thermal attribute feature vector corresponding to the target passenger based on the visual image streams and thermal imaging data streams; and it performs state perception on the target passenger based on the environmental feature vectors and the thermal attribute feature vectors corresponding to the target passenger. By generating thermal attribute feature vectors corresponding to any target passenger based on the visual image streams and thermal imaging data streams, and then performing state perception on the target passenger based on the environmental feature vectors and the thermal attribute feature vectors corresponding to the target passenger, the accuracy of passenger state perception can be improved, solving the technical problem that existing technologies only focus on environmental parameters and facial features, resulting in insufficient accuracy of thermal comfort levels or thermal state evaluation results. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 A schematic diagram illustrating the implementation process of a state-aware method provided in this application embodiment; Figure 2 A schematic diagram illustrating the implementation process of another state-awareness method provided in this application embodiment; Figure 3 A schematic diagram illustrating the implementation process of another state-aware method provided in this application embodiment; Figure 4 A schematic diagram illustrating the implementation process of a method for generating thermal attribute feature vectors provided in this application embodiment; Figure 5 This is a schematic diagram of the structure of a state sensing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] To address the technical problem that existing technologies, which focus solely on environmental parameters and facial features, result in insufficient accuracy in assessing thermal comfort levels or thermal states, this application provides a state perception method, apparatus, electronic device, and storage medium. The method acquires environmental data streams, visual image streams, and thermal imaging data streams from a vehicle; identifies at least one target passenger within the vehicle based on the visual image streams and thermal imaging data streams; maps the environmental parameters corresponding to the environmental data streams to a high-dimensional semantic space to obtain environmental feature vectors; for any target passenger, generates a thermal attribute feature vector corresponding to the target passenger based on the visual image streams and thermal imaging data streams; and performs state perception on the target passenger based on the environmental feature vectors and the corresponding thermal attribute feature vectors. By generating thermal attribute feature vectors for any target passenger based on the visual image streams and thermal imaging data streams, and then performing state perception on the target passenger based on the environmental feature vectors and the corresponding thermal attribute feature vectors, the accuracy of passenger state perception can be improved.

[0023] like Figure 1 The diagram shown illustrates the implementation flow of a state-awareness method provided in this application, which may specifically include the following steps: S101, acquires the vehicle's environmental data stream, visual image stream, and thermal imaging data stream.

[0024] The aforementioned environmental data stream refers to multi-dimensional sensor data describing the physical environment inside the cabin, used to characterize external conditions affecting human thermal comfort, such as in-vehicle air temperature (e.g., 24°C), relative humidity (e.g., 50%), airflow velocity at the air conditioning vents (e.g., 0.3 m / s), and solar radiation intensity (e.g., 200 W / m²). 2 ).

[0025] The aforementioned visual image stream refers to the sequence of visible light images of passengers captured by in-cabin cameras, used to identify information such as the passenger's facial area, clothing area, posture and movements, providing a visual basis for subsequent attribute feature extraction and clothing feature extraction.

[0026] The aforementioned thermal imaging data stream refers to the sequence of images of passenger facial temperature distribution collected by infrared thermal imaging sensors. It is used to obtain physiological indicators such as forehead temperature and nose tip temperature, reflecting the human body's real hot and cold physiological responses (such as peripheral cooling caused by vasoconstriction).

[0027] In the embodiments of this application, environmental data streams, visual image streams, and thermal imaging data streams of the vehicle can be acquired to achieve multimodal data acquisition.

[0028] Specifically, environmental data streams can be obtained by collecting data from in-vehicle temperature, humidity, wind speed, and sunlight sensors at a frequency of 1Hz via the CAN (Controller Area Network) bus. A visual image stream with a resolution of 1920×1080 is obtained by capturing images at 30 frames per second via the cockpit monitoring system camera. A facial thermal image stream with a resolution of 80×60 is obtained by capturing facial thermal maps at 10 frames per second via an infrared thermal imaging sensor.

[0029] To ensure the time synchronization of environmental data streams, visual image streams, and thermal imaging data streams, timestamp alignment can be performed on these three types of data streams. Specifically, using the timestamp of the visual image stream as a reference, the environmental data frame and thermal imaging data frame that are closest in time are found to form a multimodal data triple (environmental data frame, RGB image frame, thermal imaging frame) for subsequent synchronization processing.

[0030] Furthermore, to ensure the spatial alignment of the environmental data stream, visual image stream, and thermal imaging data stream, the cockpit monitoring system's camera and infrared thermal imaging sensor can be jointly calibrated in advance, and the homography matrix between them can be calculated. During runtime, this homography matrix is ​​used to map the coordinates of facial key points detected in the image to the heatmap coordinate system, achieving precise positioning of the facial region between the two images.

[0031] Furthermore, the sensor data in the aforementioned environmental data stream requires normalization preprocessing after acquisition. For example, temperature data is normalized to the [-1, 1] interval according to a preset range (-20℃ to 60℃), humidity data is normalized to the [0, 1] interval according to the (0% to 100%) interval, wind speed data is normalized to the [0, 1] interval according to the (0m / s to 5m / s) interval, and solar radiation intensity data is normalized to the [0, 1] interval according to the (0W / m² to 1000W / m²) interval. The normalized environmental parameters are used in subsequent feature mapping steps.

[0032] Furthermore, the aforementioned visual image streams require denoising and brightness equalization preprocessing after acquisition to reduce image differences under different lighting conditions and improve the robustness of subsequent feature extraction.

[0033] Furthermore, the aforementioned thermal imaging data stream needs to undergo temperature value calibration conversion after acquisition. This converts the original digital quantity output by the sensor (such as a grayscale value of 0-255) into the actual temperature value. The conversion formula is: T=a×DN+b, where DN is the original digital output of the sensor, a and b are calibration coefficients (such as 0.1), and T is the actual temperature value.

[0034] S102, based on the visual image stream and thermal imaging data stream, identify at least one target passenger inside the vehicle.

[0035] In this embodiment, at least one target passenger inside the vehicle is identified based on a visual image stream and a thermal imaging data stream. The target passenger refers to an individual passenger identified based on the visual image stream and thermal imaging data stream and used as the object of thermal state assessment, such as the driver in the driver's seat.

[0036] S103, map the environmental parameters corresponding to the environmental data stream to a high-dimensional semantic space to obtain the environmental feature vector.

[0037] The aforementioned environmental parameters refer to low-dimensional scalar data collected from the CAN bus, including in-vehicle air temperature, relative humidity, air conditioning vent wind speed, and solar radiation intensity. Since this low-dimensional data cannot be directly used for computation with high-dimensional image features, it needs to be mapped to a high-dimensional semantic space.

[0038] The aforementioned high-dimensional semantic space refers to the vector space obtained after nonlinearly upscaling low-dimensional input vectors through neural networks (such as multilayer perceptrons). In this space, feature vectors have a pre-defined high dimension (e.g., 128 or 256 dimensions), aligned with the image feature dimensions. This space can encode the nonlinear coupling relationships between environmental parameters, such as the synergistic effect of temperature and humidity, and the correction effect of wind speed on perceived temperature.

[0039] The aforementioned environmental feature vectors are used to characterize the numerical representation of environmental parameters in a high-dimensional semantic space, providing quantitative features of the external physical environment for passenger thermal state perception.

[0040] In this embodiment of the application, the environmental parameters corresponding to the environmental data stream are mapped to a high-dimensional semantic space to obtain an environmental feature vector.

[0041] S104: For any target passenger, generate a thermal attribute feature vector corresponding to the target passenger based on the visual image stream and the thermal imaging data stream.

[0042] The aforementioned thermal attribute feature vector refers to a multi-dimensional comprehensive feature vector describing the thermal state of a single target passenger. It can include the target passenger's basic attribute features (such as age and gender), clothing features, etc., and is used to comprehensively characterize the target passenger's individual physiological, clothing and thermal response state.

[0043] In this embodiment of the application, for any target passenger, a thermal attribute feature vector corresponding to the target passenger is generated based on the visual image stream and the thermal imaging data stream.

[0044] S105, based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger, performs state perception on the target passenger.

[0045] In this embodiment, the target passenger's state is perceived based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger. State perception refers to the identification and quantitative assessment of the target passenger's current thermal comfort state.

[0046] Based on the above description of the technical solution provided in the embodiments of this application, environmental data streams, visual image streams, and thermal imaging data streams of a vehicle are acquired; based on the visual image streams and thermal imaging data streams, at least one target passenger inside the vehicle is identified; environmental parameters corresponding to the environmental data streams are mapped to a high-dimensional semantic space to obtain environmental feature vectors; for any target passenger, a thermal attribute feature vector corresponding to the target passenger is generated based on the visual image streams and thermal imaging data streams; and the state of the target passenger is perceived based on the environmental feature vectors and the thermal attribute feature vectors corresponding to the target passenger. By generating thermal attribute feature vectors corresponding to any target passenger based on the visual image streams and thermal imaging data streams, and then perceiving the state of the target passenger based on the environmental feature vectors and the thermal attribute feature vectors corresponding to the target passenger, the accuracy of passenger state perception can be improved, solving the technical problem that existing technologies only focus on environmental parameters and facial features, resulting in insufficient accuracy of thermal comfort levels or thermal state evaluation results.

[0047] like Figure 2 The diagram shown is a schematic representation of the implementation process of another state-awareness method provided in this application. Figure 2 exist Figure 1 Building upon this foundation, the paper details how to identify at least one target passenger based on visual image streams and thermal imaging data streams, and how to map environmental parameters corresponding to environmental data streams to a high-dimensional semantic space to obtain environmental feature vectors. Specifically, this may include the following: S201, acquire the vehicle's environmental data stream, visual image stream, and thermal imaging data stream.

[0048] In this embodiment of the application, this step is similar to step S101 above, and will not be described in detail here.

[0049] S202, determine at least one candidate passenger inside the vehicle based on the visual image stream.

[0050] In this embodiment, at least one candidate passenger within the vehicle is determined based on a visual image stream. A candidate passenger refers to a target object that is preliminarily identified as a passenger in the cabin by a human detection algorithm (such as a face detection algorithm) in the visual image stream. Candidate passengers have not yet undergone liveness verification and may include non-live targets (such as human models, stacks of clothing, etc.), requiring further screening in subsequent steps.

[0051] S203: For any candidate passenger, obtain the temperature distribution map corresponding to the candidate passenger from the thermal imaging data stream.

[0052] In this embodiment of the application, for any candidate passenger, a temperature distribution map corresponding to the candidate passenger is obtained from the thermal imaging data stream. The temperature distribution map refers to a two-dimensional temperature matrix extracted from the thermal imaging data stream that corresponds to the facial region of the candidate passenger. It is used to quantify and characterize the spatial temperature distribution of the candidate passenger's face, providing a data source for subsequent calculations of physiological indicators such as forehead temperature and nose tip temperature. For example, it is an 80×60 pixel grayscale image, where each pixel value corresponds to a specific temperature value (e.g., 36.2℃).

[0053] Specifically, based on the position of the candidate passenger's face bounding box in the visual image stream detected in step S202 above, the bounding box coordinates can be mapped to the coordinate system of the thermal imaging data stream using a pre-calibrated homography transformation matrix, thereby cropping out the facial temperature distribution map corresponding to the candidate passenger from the thermal imaging data stream.

[0054] S204. Based on the temperature distribution map corresponding to the candidate passenger, perform liveness verification on the candidate passenger and determine the candidate passenger who passes the verification as the target passenger.

[0055] In this embodiment, candidate passengers are verified for liveness based on the temperature distribution map corresponding to them, and those who pass the verification are identified as target passengers. Liveness verification refers to a method that uses the unique physiological characteristics of living organisms (such as a constant body temperature range) to determine the authenticity of candidate passengers. This is used to eliminate interference from non-living targets (such as mannequins, stacks of clothing, seat headrests, etc.) in subsequent state perception, such as determining whether the temperature distribution map contains a continuous area with a stable temperature value between 36℃ and 37℃. Target passengers refer to candidate passengers who successfully pass liveness verification and are confirmed as real living passengers, serving as the final object for subsequent thermal attribute feature extraction and state perception analysis, such as drivers confirmed as real passengers.

[0056] Specifically, the average temperature of the facial region in the temperature distribution map corresponding to the candidate passenger can be calculated. If the average temperature is within the normal human body temperature range (e.g., 35.5℃-37.5℃) and the temperature distribution conforms to the physiological characteristics of the human face (e.g., the temperature of the tip of the nose is slightly lower than that of the forehead), then the candidate passenger is determined to have passed the liveness verification and is identified as the target passenger.

[0057] S205, normalize the environmental parameters corresponding to the environmental data stream to generate normalized environmental parameters.

[0058] In this embodiment, the environmental parameters corresponding to the environmental data stream are normalized to generate normalized environmental parameters. These environmental parameters refer to the individual physical quantities parsed from the environmental data stream, used as specific input values ​​for subsequent feature mapping, such as in-vehicle air temperature (24℃), relative humidity (50%), wind speed (0.3m / s), and solar radiation intensity. Normalization is a mathematical transformation method that scales environmental parameters with different dimensions and numerical ranges to a unified interval (e.g., [0, 1] or [-1, 1]) to eliminate dimensional differences between different physical quantities, making subsequent neural network training more stable. For example, mapping the temperature range from -20℃ to 60℃ to the interval [0, 1].

[0059] Specifically, the min-max normalization method can be used to pre-calculate the historical maximum and minimum values ​​of each environmental parameter, perform a linear transformation on each environmental parameter collected in real time, and generate normalized environmental parameters.

[0060] S206 maps the normalized environmental parameters to a high-dimensional semantic space to obtain environmental feature vectors.

[0061] In this embodiment of the application, the normalized environmental parameters are mapped to a high-dimensional semantic space to obtain environmental feature vectors.

[0062] For example, normalized environmental parameters (such as temperature, humidity, wind speed, and solar radiation intensity) can be concatenated to obtain a 4-dimensional vector, which can then be input into an MLP network with two hidden layers (64-dimensional and 128-dimensional, respectively) to obtain a 256-dimensional environmental feature vector.

[0063] After obtaining the environmental feature vector, it can be further modified based on the current time period (such as night, spring, winter, etc.). This can be achieved through the following steps: Step 11: Determine the target environment range based on the environmental data stream.

[0064] In this embodiment, a target environmental interval is determined based on the environmental data stream. The target environmental interval refers to the category or time period of the external environment in which the cabin is located, determined based on the current time or environmental parameters. This interval is used to select a corresponding correction vector to adjust environmental characteristics to adapt to differences in thermal perception under different seasons or weather conditions, such as intervals like "winter," "summer," "night," and "noon."

[0065] Specifically, the current time period can be determined by the clock (e.g., 18:00-6:00 is considered nighttime), or the season can be determined by the ambient temperature (e.g., an average temperature below 10℃ for 7 consecutive days is considered winter), thus determining the target environmental range as "winter nighttime".

[0066] Step 12: Obtain the environmental correction vector corresponding to the target environment range.

[0067] In this embodiment, an environmental correction vector corresponding to the target environmental interval is obtained. The environmental correction vector refers to a pre-stored feature adjustment vector bound to a specific environmental interval. Its dimension is the same as the environmental feature vector, and it is used to perform targeted scaling or bias adjustment on environmental features. For example, the correction vector corresponding to the winter interval has a lower weight on the solar radiation-related dimension and a higher weight on the temperature-related dimension.

[0068] Specifically, from the pre-generated correction vector lookup table, the environmental correction vector corresponding to the target environmental interval "winter night" index is selected. For example, a 256-dimensional vector is selected, where the dimension related to temperature features is 1.2 (enhancing temperature weight) and the dimension related to sunshine features is 0.3 (reducing sunshine weight).

[0069] Step 13: Multiply the environmental feature vector and the environmental correction vector element by element to obtain the updated environmental feature vector.

[0070] In this embodiment, the environmental feature vector and the environmental correction vector are multiplied element-wise to obtain the updated environmental feature vector. Element-wise multiplication refers to multiplying the elements at the same position in two vectors of the same dimension to obtain a new vector of the same dimension. This operation applies the adjustment effect of the correction vector to each dimension of the original environmental feature vector. The updated environmental feature vector refers to the new environmental feature representation obtained after adjustment by the environmental correction vector. For example, the original feature vector [0.5, 0.8, 0.2] is multiplied element-wise with the correction vector [1.2, 1.0, 0.5] to obtain [0.6, 0.8, 0.1].

[0071] S207: For any target passenger, generate a thermal attribute feature vector corresponding to the target passenger based on the visual image stream and the thermal imaging data stream.

[0072] In this embodiment of the application, this step is similar to step S104 above, and will not be described in detail here.

[0073] S208, based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger, performs state perception of the target passenger.

[0074] In this embodiment of the application, this step is similar to step S105 above, and will not be described in detail here.

[0075] like Figure 3 The diagram shown is a schematic representation of the implementation process of another state-awareness method provided in this application. Figure 3 exist Figure 1 , Figure 2 Building upon this foundation, the paper details how to generate thermal attribute feature vectors corresponding to target passengers based on visual image streams and thermal imaging data streams, and how to perform state perception of target passengers based on environmental feature vectors and thermal attribute feature vectors corresponding to target passengers. Specifically, this may include the following: S301 acquires the vehicle's environmental data stream, visual image stream, and thermal imaging data stream.

[0076] In this embodiment of the application, this step is similar to step S101 above, and will not be described in detail here.

[0077] S302, based on the visual image stream and thermal imaging data stream, identify at least one target passenger inside the vehicle.

[0078] In this embodiment of the application, this step is similar to step S102 above, and will not be described in detail here.

[0079] S303 maps the environmental parameters corresponding to the environmental data stream to a high-dimensional semantic space to obtain the environmental feature vector.

[0080] In this embodiment of the application, this step is similar to step S103 above, and will not be described in detail here.

[0081] S304, for any target passenger, extract the target area image corresponding to the target passenger from the visual image stream.

[0082] In this embodiment of the application, for any target passenger, a target region image corresponding to the target passenger is extracted from the visual image stream. The target region image refers to a local image region cropped from the visual image stream that contains key body parts of the target passenger, used for subsequent further segmentation of the face region and clothing region to reduce background interference, such as a 500×400 pixel image of the passenger's upper body cropped from a 1920×1080 original image.

[0083] Specifically, based on the bounding box position (e.g., a rectangle containing the head and upper body) of the target passenger detected in step S202 in the RGB image of the visual image stream, the image region corresponding to the rectangle can be cropped from the visual image stream of the current frame and used as the target region image of the target passenger.

[0084] S305, generate a thermal attribute feature vector corresponding to the target passenger based on the target area image and thermal imaging data stream corresponding to the target passenger.

[0085] In this embodiment of the application, a thermal attribute feature vector corresponding to the target passenger is generated based on the target area image and thermal imaging data stream corresponding to the target passenger.

[0086] For details on how to generate a thermal attribute feature vector for a target passenger based on the target area image and thermal imaging data stream, please refer to [reference needed]. Figure 4 The method shown. (As illustrated) Figure 4 The diagram shown illustrates the implementation flow of a method for generating thermal attribute feature vectors according to an embodiment of this application, which may specifically include the following steps: S401, process the target area image corresponding to the target passenger to obtain the face area image and clothing area image corresponding to the target passenger.

[0087] In this embodiment, the target area image corresponding to the target passenger is processed to obtain a face area image and a clothing area image corresponding to the target passenger. The face area image refers to a sub-image cropped from the target area image, containing only the target passenger's face. It is used to extract passenger attribute features such as age and gender, as well as implicit features such as complexion and skin texture, to further infer the target passenger's body type. For example, a 224×224×3 RGB image may only show the driver's face. The clothing area image refers to a sub-image cropped from the target area image, mainly containing the target passenger's upper body clothing (usually located below the face area). It is used to extract visual features such as clothing texture, material, collar shape, and sleeve length, to estimate the clothing's thermal resistance. For example, a 224×224×3 RGB image may show the texture of the down jacket worn by the passenger.

[0088] Specifically, a face detection model (such as a multi-task cascaded convolutional network) can be used to locate the face bounding box in the target region image and crop the face region image; then, based on the detection results of human body key points (such as shoulder and chest key points), the body region below the face region can be cropped to obtain the clothing region image.

[0089] S402, Based on the face region image corresponding to the target passenger, determine the implicit face feature vector and passenger attribute label.

[0090] In this embodiment, an implicit facial feature vector and passenger attribute labels are determined based on the facial region image corresponding to the target passenger. The implicit facial feature vector refers to a high-dimensional feature representation without explicit semantic labels, automatically extracted from the facial region image using a deep convolutional neural network. It is used to encode subtle structural information related to individual differences in the face (such as complexion rosiness, skin luster, facial fat distribution, and wrinkle texture). The passenger attribute labels refer to demographic attribute values ​​with explicit semantic meanings predicted from the facial image through supervised learning. These are used as explicit features to directly participate in subsequent thermal state perception calculations, such as age (35 years old), gender (male), and body type (e.g., underweight, normal weight, overweight, obese).

[0091] Specifically, the face region image can be input into a ResNet-50 network pre-trained on a large face dataset (such as the Internet Movie Database - Wikipedia Face Dataset), and the output of the penultimate layer of the network can be extracted as a latent face feature vector. At the same time, three independent fully connected branches are connected to the network to output the age regression value, gender classification probability, and body type classification probability, respectively, so as to obtain the passenger attribute label (such as age 35, gender male, body type normal).

[0092] Determining the implicit facial feature vector and passenger attribute labels based on the facial region image corresponding to the target passenger may specifically include the following steps: Step 21: Determine the basic facial feature map based on the facial region image corresponding to the target passenger.

[0093] In this embodiment, a basic facial feature map is determined based on the facial region image corresponding to the target passenger. The basic facial feature map refers to a two-dimensional feature tensor (e.g., height × width × number of channels) obtained after inputting the facial region image into the first few convolutional layers of a deep convolutional neural network (e.g., ResNet-50). It is used to preserve the spatial structure and local texture information of the face, providing rich intermediate representations for subsequent global pooling operations, such as a 7×7×2048 feature map.

[0094] For example, a 224×224×3 face region image can be input into a ResNet-50 network. After calculation by the first four residual blocks (excluding the global average pooling layer and the fully connected layer), a basic face feature map with a size of 7×7×2048 can be obtained.

[0095] Step 22: Perform global average pooling on the basic facial feature map to obtain the implicit facial feature vector.

[0096] In this embodiment, global average pooling is performed on the basic facial feature map to obtain an implicit facial feature vector. Global average pooling refers to calculating the average pixel value (height × width) at all spatial locations in each channel of the basic facial feature map, compressing the three-dimensional feature tensor into a one-dimensional vector. This is used to aggregate spatial information into global statistics, reduce the number of parameters, and enhance the translation invariance of the model, such as compressing a 7×7×2048 feature map into a 2048-dimensional vector.

[0097] Step 23: Determine passenger attribute labels based on implicit facial feature vectors.

[0098] In this embodiment, passenger attribute labels are determined based on implicit facial feature vectors. Passenger attribute labels can be understood as explicit attribute values ​​obtained by mapping and classifying / regressing implicit facial feature vectors in a fully connected layer.

[0099] Specifically, the implicit facial feature vector can be input into three independent fully connected branches: the age branch outputs a scalar value (activated by ReLU); the gender branch outputs a binary classification probability via Sigmoid; and the body type branch outputs the probability distribution of the three categories via Softmax. The category corresponding to the highest probability (such as "normal") is then selected to obtain the passenger attribute label.

[0100] S403, determine the implicit clothing feature vector and clothing thermal resistance estimate based on the clothing area image corresponding to the target passenger.

[0101] In this embodiment, an implicit clothing feature vector and an estimated clothing thermal resistance value are determined based on the clothing area image corresponding to the target passenger. The implicit clothing feature vector refers to a high-dimensional feature representation extracted from the clothing area image using a deep convolutional neural network, used to encode the visual semantic information of the clothing (such as fabric thickness, loft, weave density, and coverage area), such as a 1024-dimensional floating-point vector. The estimated clothing thermal resistance value refers to a continuous numerical value obtained by regressing from the clothing image features, quantifying the thermal insulation capacity of a single garment or the overall outfit, such as 0.5clo (summer shirt) or 2.5clo (thick down jacket).

[0102] Determining implicit clothing feature vectors and estimated clothing thermal resistance values ​​based on images of clothing regions corresponding to target passengers can specifically include the following steps: Step 31: Determine the basic feature map of clothing based on the clothing area image corresponding to the target passenger.

[0103] In this embodiment, a basic clothing feature map is determined based on the clothing region image corresponding to the target passenger. The basic clothing feature map refers to a two-dimensional feature tensor obtained by inputting the clothing region image into the first few layers of a convolutional neural network, used to preserve spatial structural information such as the texture, wrinkles, and boundaries of the clothing. Specifically, the clothing region image can be input into a MobileNetV3 network to obtain the basic clothing feature map.

[0104] Step 32: Perform global average pooling on the basic clothing feature map to obtain the implicit clothing feature vector.

[0105] In this embodiment of the application, global average pooling is performed on the clothing basic feature map to obtain implicit clothing feature vectors.

[0106] Step 33: Determine the estimated value of clothing thermal resistance based on implicit clothing feature vectors.

[0107] In this embodiment, the thermal resistance estimate of clothing is determined based on implicit clothing feature vectors. Specifically, the implicit clothing feature vectors can be input into a fully connected network containing two hidden layers (128-dimensional and 32-dimensional), and the output is a scalar value (which can be scaled to the range of 0~2.5clo via Sigmoid) as the estimated thermal resistance of the clothing.

[0108] S404, based on the thermal imaging data stream and the target area image corresponding to the target passenger, determine the implicit thermal distribution feature vector, the first thermal sensing value, the second thermal sensing value, and the thermal sensing difference.

[0109] In this embodiment, based on the thermal imaging data stream and the target area image corresponding to the target passenger, an implicit thermal distribution feature vector, a first thermal sensing value, a second thermal sensing value, and a thermal sensing difference are determined. The implicit thermal distribution feature vector refers to a high-dimensional feature representation extracted from the facial heatmap (temperature distribution map), used to encode the global spatial distribution pattern of facial temperature (such as temperature asymmetry between the left and right cheeks, temperature gradient between the forehead and cheeks, etc.). The first thermal sensing value refers to the average temperature value of a specific area of ​​the face (such as the forehead area), used as a basic physiological indicator reflecting the core body temperature or vasomotor state. For example, if the average temperature value of the target passenger's forehead area is 36.8℃, then the first thermal sensing value is 36.8℃. The second thermal sensing value refers to the average temperature value of another specific area of ​​the face (such as the tip of the nose area), used for comparison with the first thermal sensing value. The tip of the nose, as a peripheral area, is more sensitive to vasoconstriction. For example, if the average temperature value of the target passenger's nose tip area is 34.2℃, then the second thermal sensing value is 34.2℃. The thermal sensing difference is the difference between the first thermal sensing value and the second thermal sensing value.

[0110] Based on the thermal imaging data stream and the target area image corresponding to the target passenger, the implicit thermal distribution feature vector, the first thermal sensing value, the second thermal sensing value, and the thermal sensing difference are determined. Specifically, this may include the following steps: Step 41: Obtain the temperature distribution map corresponding to the target passenger from the thermal imaging data stream.

[0111] In this embodiment, a temperature distribution map corresponding to the target passenger is obtained from the thermal imaging data stream. Specifically, based on the facial bounding box of the target passenger in the RGB image in step S302 above, a homography matrix is ​​used to map it to the thermal imaging coordinate system, and the facial temperature distribution map of the target passenger is cropped from the thermal imaging data stream, i.e., the temperature distribution map, with a resolution of 80×60 pixels.

[0112] Step 42: Determine the implicit thermal distribution feature vector based on the temperature distribution map corresponding to the target passenger.

[0113] In this embodiment, an implicit thermal distribution feature vector is determined based on the temperature distribution map corresponding to the target passenger. Specifically, the temperature distribution map corresponding to the target passenger can be downsampled and input into a lightweight CNN consisting of three convolutional layers (each followed by a pooling layer), and the implicit thermal distribution feature vector is obtained after flattening.

[0114] Step 43: Determine the coordinates of facial key points based on the target area image corresponding to the target passenger.

[0115] In this embodiment, the coordinates of facial key points are determined based on the target area image corresponding to the target passenger. Here, the facial key point coordinates refer to the pixel positions in the image coordinate system of facial feature points (such as the tip of the nose, the center of the forehead, the corners of the eyes, and the corners of the mouth) located from the RGB image using a facial key point detection algorithm. These coordinates are used to map key areas in the RGB image onto a temperature distribution map, such as the coordinates of the tip of the nose (x=540, y=320) and the coordinates of the center of the forehead (x=520, y=240).

[0116] Specifically, a pre-trained PFLD or HRNet model can be used to detect 68 or 106 facial key points in the target region image (or face region image), and the coordinates of the nose tip and the center point of the forehead can be extracted as the facial key point coordinates.

[0117] Step 44: Determine the first distribution area and the second distribution area from the temperature distribution map corresponding to the target passenger based on the facial key point coordinates.

[0118] In this embodiment, a first distribution region and a second distribution region are determined from the temperature distribution map corresponding to the target passenger based on the facial key point coordinates. The first distribution region refers to a region of interest (ROI) defined on the temperature distribution map based on the facial key point coordinates, corresponding to the forehead region, such as a circular region with a radius of 8 pixels centered on the forehead key point. The second distribution region refers to another ROI defined on the temperature distribution map, corresponding to the tip of the nose region, such as a circular region with a radius of 5 pixels centered on the tip of the nose key point.

[0119] Step 45: Take the average temperature of the first distribution area in the temperature distribution map as the first thermal sensing value.

[0120] In this embodiment, the average temperature corresponding to the first distribution area in the temperature distribution map is used as the first thermal sensing value. The first thermal sensing value refers to the arithmetic mean of the temperature values ​​of all pixels within the first distribution area, representing the skin temperature of the forehead region.

[0121] Specifically, the temperature values ​​of all pixels within the first distribution area (the circular area on the forehead) can be extracted, and their average value can be calculated, for example, 36.8℃, as the first thermal sensing value.

[0122] Step 46: Take the average temperature of the second distribution area in the temperature distribution map as the second thermal sensing value.

[0123] In this embodiment, the average temperature corresponding to the second distribution area in the temperature distribution map is used as the second thermal sensing value. The second thermal sensing value refers to the arithmetic mean of the temperature values ​​of all pixels within the second distribution area, representing the skin temperature of the nose tip region.

[0124] Specifically, the temperature values ​​of all pixels within the second distribution area (the circular area at the tip of the nose) can be extracted, and their average value can be calculated, for example, 34.2℃, as the second thermal sensing value.

[0125] Step 47: Determine the difference between the first thermal sensing value and the second thermal sensing value as the thermal sensing difference.

[0126] In this embodiment of the application, the difference between the first thermal sensing value and the second thermal sensing value is defined as the thermal sensing difference.

[0127] S405 generates a thermal attribute feature vector corresponding to the target passenger based on implicit face feature vector, passenger attribute label, implicit clothing feature vector, clothing thermal resistance estimation value, implicit thermal distribution feature vector, first thermal perception value, second thermal perception value and thermal perception difference.

[0128] In this embodiment of the application, a thermal attribute feature vector corresponding to the target passenger is generated based on the implicit face feature vector, passenger attribute label, implicit clothing feature vector, clothing thermal resistance estimation value, implicit thermal distribution feature vector, first thermal perception value, second thermal perception value and thermal perception difference.

[0129] Specifically, the implicit face feature vector, passenger attribute label, implicit clothing feature vector, clothing thermal resistance estimate, implicit thermal distribution feature vector, first thermal perception value, second thermal perception value, and thermal perception difference can be concatenated in order to obtain the thermal attribute feature vector corresponding to the target passenger.

[0130] S306, using the thermal attribute feature vector as the query term and the environmental feature vector as the key term, determines the attention weight matrix.

[0131] In this embodiment, the thermal attribute feature vector is used as the query term, and the environmental feature vector is used as the key term to determine the attention weight matrix. The query term refers to the feature vector used to actively match the key term in the cross-attention mechanism, and the key term refers to the feature vector matched by the query term in the cross-attention mechanism, used to calculate the relevance score with the query term. The attention weight vector is used to quantify the degree of attention passengers pay to different environmental factors.

[0132] Specifically, each environmental parameter (temperature, humidity, wind speed, and sunshine) in the environmental feature vector can be mapped to a sub-vector of a uniform dimension and stacked into a key matrix. Specifically, each environmental parameter is normalized to obtain a temperature scalar T, a humidity scalar H, a wind speed scalar W, and a sunshine scalar S, which are then input into four multilayer perceptrons (MLPs) with identical structures but independent parameters. Each MLP contains two hidden layers (e.g., a 64-dimensional first layer and a 128-dimensional second layer), outputting a fixed-dimensional D environmental sub-vector (D being the same dimension as the thermal attribute feature vector). This yields four D-dimensional vectors: KT, KH, KW, and KS. These four D-dimensional vectors are stacked row-wise to form the key matrix K. The attention weight matrix is ​​then calculated using the matrix formula: ; in, Here is the attention weight matrix. For the query item (hot attribute feature vector). The key matrix, Let T represent the total dimension of the environmental features, and let T denote the transpose of a matrix or vector, converting a column vector into a row vector for dot product operations. It is a normalized exponential function.

[0133] S307 uses the attention weight matrix to weight the environmental feature vector to obtain the weighted environmental feature vector.

[0134] In this embodiment, the environmental feature vector is weighted using an attention weight matrix to obtain a weighted environmental feature vector.

[0135] S308, the thermal attribute feature vector corresponding to the target passenger is fused with the weighted environmental feature vector to obtain a multimodal fusion feature vector.

[0136] In this embodiment, the thermal attribute feature vector corresponding to the target passenger is fused with the weighted environmental feature vector to obtain a multimodal fusion feature vector. The multimodal fusion feature vector refers to the final feature representation that integrates passenger attributes, clothing, physiological state, and environmental information after state weighting, and is used as input to a subsequent temporal network (such as LSTM) for dynamic analysis.

[0137] To fuse the thermal attribute feature vector corresponding to the target passenger with the weighted environmental feature vector to obtain a multimodal fusion feature vector, the thermal attribute feature vector can be used as the query vector Q, and the weighted environmental feature vector as the value vector V. Q and V are concatenated to obtain the concatenated vector [Q; V]. This concatenated vector is then input into a gating unit, which consists of a fully connected layer and a sigmoid activation function, outputting a fusion coefficient g between 0 and 1. Finally, the multimodal fusion feature vector F = g × [Q; V] + (1-g) × Q is calculated, or the concatenated vector [Q; V] can be directly mapped to the target dimension through a fully connected layer. Through the gating unit, more weighted environmental information can be adopted when the environmental influence is significant, while the passenger's own physiological state information is preserved when the environmental influence is weak (such as when the passenger is in an extreme emotional state or sick), achieving adaptive multimodal fusion.

[0138] S309 generates thermal status evaluation results for target passengers based on multimodal fusion feature vectors.

[0139] In this embodiment, a thermal state evaluation result for the target passenger is generated based on a multimodal fusion feature vector. The thermal state evaluation result refers to the final output quantitative index used to describe the passenger's current thermal comfort level.

[0140] Specifically, the multimodal fusion feature vectors of each frame can be stacked into a temporal feature matrix in chronological order (time window length T=30 frames), and input into an LSTM network to extract temporal dependencies (such as temperature decrease trend). The output of the last time step is then taken. This output is then mapped to a scalar value through a fully connected layer. Finally, this scalar value is weighted and summed with the physical baseline value calculated based on the Fanger PMV formula to obtain the final thermal state evaluation result (e.g., output thermal sensation vote TSV=-0.2).

[0141] Furthermore, the state awareness method provided in this application is illustrated with specific examples in the embodiments of this application: To address the following problems existing in the current technology: (1) Single dimension of environmental parameters: The existing solution only relies on the in-vehicle air temperature sensor and fails to fully consider the nonlinear coupling effect of environmental factors such as solar radiation, local wind speed and air humidity on human thermal sensation; (2) Lack of adaptability to individual differences: The traditional Fanger PMV (Predicted Mean Vote) model is a static model based on statistics, which is difficult to adapt to individual metabolic differences among passengers of different ages, genders, and body mass indexes (BMI). For example, under the same environmental conditions, there are significant differences in thermal comfort needs between the elderly and young people; (3) Simple multimodal fusion mechanism: Existing deep learning solutions mostly use simple feature splicing to fuse environmental data and visual data, ignoring the physical nature of "human-environment interaction". For example, when passengers are wearing heavy clothing, the influence of ambient temperature and wind speed on the human body's thermal state should be significantly reduced, but simple feature splicing cannot dynamically adjust such weight relationships, resulting in insufficient model generalization ability; (4) Lack of physiological feedback mechanism: Relying solely on visual images cannot obtain real information on the human body's physiological responses to cold and heat (such as the drop in peripheral temperature caused by vasoconstriction), thus resulting in a lag in response to dynamic changes in the thermal environment.

[0142] This application provides a state perception method. This method extracts occupant attribute features, clothing features, and thermophysiological features through a deep learning network, and introduces an attention-based module to simulate the modulating effect of clothing insulation performance and physiological state on environmental perception. This achieves more accurate, robust, and physically interpretable cabin occupant thermal state perception. Specifically, it may include the following: Step 1: Acquire multimodal data synchronously.

[0143] Acquire real-time multimodal data streams from within the cockpit, specifically including: Environmental data stream: Obtain environmental parameters such as in-vehicle air temperature, relative humidity, air conditioning vent wind speed, and sunlight intensity via the vehicle CAN bus; Visual image stream: Visible light images of occupants are captured by the RGB camera of the Occupant Monitoring System (OMS); Thermal imaging data stream: Infrared thermal images of the occupant's face are acquired using an infrared thermal imaging sensor.

[0144] Step 2: Heterogeneous feature extraction and alignment.

[0145] For the physical characteristics of different modal data, feature extraction and spatial alignment are performed separately. The specific process is as follows: (1) Upscaling of environmental features: Since environmental data is a low-dimensional scalar and cannot directly interact with high-dimensional image features, a multilayer perceptron (MLP) encoder can be constructed. The normalized environmental parameters are input into the encoder and mapped to a high-dimensional semantic space to obtain the environmental feature vector. This process can effectively capture complex environmental features such as the nonlinear coupling between temperature and humidity.

[0146] (2) Preprocessing and input of visual data: The original RGB images are acquired through the cockpit monitoring system. The images are preprocessed using object detection algorithms (such as YOLO or RetinaFace) to crop out the face area image and the upper body clothing area image, and the images are normalized to a preset size (e.g., 224×224×3 pixels) for subsequent network input.

[0147] (3) Extraction of occupant attribute features: A deep convolutional neural network (e.g., ResNet-50 or MobileNetV3) is used as the first shared backbone network, specifically including: 1. Backbone feature extraction: The face region image is input into the first shared backbone network, and after multiple convolution and pooling operations, the basic feature map of the second to last layer of the network is extracted. 2. Construction of implicit attribute feature vector: Global average pooling is performed on the above basic feature map to obtain a one-dimensional high-dimensional vector, namely the implicit face feature vector. This implicit vector retains the unstructured subtle features of the face (such as complexion rosiness, skin texture, and micro-geometric structure of the face), containing richer individual difference information than a single label. 3. Regression and classification of explicit attribute labels: Three independent fully connected layer branches are connected after the implicit face feature vector, which are used to predict explicit labels respectively: the age branch outputs the classification probability distribution or regression value to obtain the age label; the gender branch outputs the binary classification probability to obtain the gender label; the body shape branch outputs the probability distribution of body shape categories (underweight, normal, overweight, obese) to obtain the BMI level label.

[0148] (4) Clothing feature extraction process: A convolutional neural network pre-trained on the clothing dataset is used as the second shared backbone network. Specifically, it includes: 1. Inputting the upper body clothing area image into the second shared backbone network. Similarly, the output of the penultimate layer of the network is extracted and pooled to obtain the implicit clothing feature vector. This vector encodes visual semantic information such as the texture density, material fluffiness, neckline opening and closing status, and sleeve length of the clothing. This information cannot be completely covered by simple numerical labels and is crucial for judging the true thermal insulation effect of clothing. 2. Estimation of explicit clothing thermal resistance: A regression branch is connected after the implicit clothing feature vector. This branch consists of a fully connected layer and an activation function, which directly maps and outputs a continuous value, namely the estimated value of clothing thermal resistance. During the training phase, this regression branch uses a dataset with thermal resistance labels for supervised learning, enabling the network to infer physical thermal resistance from visual texture.

[0149] (5) Extraction process of thermophysiological features: The temperature matrix data obtained by the infrared thermal imaging sensor is processed as follows: 1. Calculation of explicit physiological indicators: Based on the coordinates of key facial points detected by the RGB image, the forehead region and the tip of the nose region are located by mapping to the infrared thermogram coordinate system through the pre-calibrated homography matrix. Forehead temperature: Calculate the average temperature of all pixels in the forehead region; Tip of the nose temperature: Calculate the average temperature of all pixels in the tip of the nose region; Temperature gradient: Calculate the difference between the forehead temperature and the tip of the nose temperature. 2. Construction of implicit thermal distribution feature vector: The localized facial thermal map (single-channel grayscale image) is cropped and downsampled to a fixed size (e.g., 32×32 pixels) and input into a lightweight convolutional network (e.g., a 3-layer convolutional neural network). The output vector of the network is extracted as the implicit thermal distribution feature vector. This implicit vector preserves the spatial distribution structure of facial temperature and can identify local thermal unevenness patterns such as "unilateral cheek heating up due to sunlight".

[0150] Step 3: Context-gated cross-attention fusion.

[0151] A context-gated cross-attention module is constructed to simulate the physical process of "occupant state determining the weight of environmental influences". The specific steps are as follows: (1) Construct query items and key-value items: The occupant attribute features, clothing features and thermophysiological features extracted in the previous step are concatenated to form a query vector Q representing the subjective state of the occupant. The extracted environmental feature vector is used as the key vector K and the value vector V.

[0152] (2) Calculate attention weights: Calculate the dot product correlation between the query vector Q and the key vector K, and normalize it using the Softmax function to obtain the attention weight matrix. This step enables the model to learn the interaction relationships between different features. For example, when the query vector Q contains the feature "thick down jacket", the model will automatically reduce the activation value corresponding to the feature "high wind speed" in the key vector K, thereby suppressing the weight influence of environmental factors on the final thermal comfort score.

[0153] (3) Context Gating: The environmental feature vector is weighted using the attention weights mentioned above to obtain a weighted environmental vector. Then, the original occupant query vector Q is concatenated with the weighted environmental vector, and a fusion coefficient is calculated using a gating unit. This gating unit consists of a fully connected layer and a sigmoid activation function, outputting a fusion coefficient ranging from 0 to 1. Finally, a multimodal fusion feature vector is output. This step ensures that even when environmental influences are minimal (e.g., when the occupant is in an extreme emotional or sick state), the model can still effectively preserve the occupant's original physiological state information.

[0154] Step 4: Timing analysis.

[0155] Set a time window of length T (e.g., 10 seconds), and stack the fused feature vectors of each frame output from step three in chronological order to construct a temporal feature matrix. Input this temporal feature matrix into a Long Short-Term Memory (LSTM) network, as follows: The LSTM unit internally uses a forget gate to determine how far back in thermal state information to discard, and an input gate to determine how much new features to accept at the current moment.

[0156] When a sudden change in the clothing feature vector is detected in a short period of time (e.g., the estimated thermal resistance of clothing jumps from high to low, indicating that the occupant is taking off their clothes), the LSTM cell state is updated rapidly to adapt to the new thermal equilibrium point.

[0157] The hidden state of a network implicitly encodes the first derivative (i.e., the rate of change) of the input features. For example, if the nasal tip temperature feature shows a continuous downward trend in a time series, then even if the current absolute temperature is still within the normal range, LSTM can output a predictive signal indicating that "I am about to feel cold and uncomfortable".

[0158] The output vector of the last time step of the LSTM is taken as the final thermal state representation vector. This time-series analysis mechanism actually acts as a "low-pass filter," which can filter out instantaneous fluctuations caused by sensor noise, ensuring that the output thermal comfort evaluation curve is smooth and stable, consistent with the physiological fact that human thermal sensation has thermal inertia.

[0159] Step 5: Physically guided residual output.

[0160] A dual-stream output architecture was used for the final thermal state evaluation, as detailed below: (1) Physical reference flow: Using the extracted clothing thermal resistance estimate, environmental parameters, and human metabolic rate calculated based on age and BMI, the theoretical thermal comfort value is calculated by substituting it into the Fanger PMV (Predicted Mean Vote) formula, which serves as the physical reference value.

[0161] (2) Neural network flow: The time-series feature vector output from step four is passed through a fully connected layer to map and predict personalized thermal comfort deviation values.

[0162] (3) Final output: The physical baseline value and the personalized deviation value predicted by the neural network are weighted and summed to obtain the final thermal comfort evaluation result, namely thermal sensation vote (TSV).

[0163] This application's embodiments successfully simulate the moderating effects of clothing insulation and individual physical constitution on environmental perception through a cross-attention mechanism, solving the problem that traditional stitching and fusion methods cannot handle—the issue of "different clothing leading to different sensations at the same temperature." By introducing a physical PMV model as a benchmark, the lower limit of the model's output is guaranteed under data scarcity or extreme conditions, avoiding non-physical predictions that pure neural networks may produce. By incorporating the temperature difference features between the nose tip and forehead, it can capture the occupant's cold discomfort (vasoconstriction) response earlier and more sensitively than simply monitoring air temperature. By employing a pre-trained backbone network and a multi-task learning architecture, and utilizing MLP to align feature dimensions, the model can achieve good generalization ability without being trained from scratch on massive amounts of cabin data, and is suitable for deployment on in-vehicle systems.

[0164] Corresponding to the above method embodiments, this application also provides a state sensing device, such as... Figure 5 As shown, the device may include a data stream acquisition module 501, a target passenger determination module 502, an environmental feature vector generation module 503, a thermal attribute feature vector generation module 504, and a state perception module 505.

[0165] The data stream acquisition module 501 is used to acquire the vehicle's environmental data stream, visual image stream, and thermal imaging data stream. The target passenger determination module 502 is used to determine at least one target passenger inside the vehicle based on a visual image stream and a thermal imaging data stream. The environmental feature vector generation module 503 is used to map the environmental parameters corresponding to the environmental data stream to a high-dimensional semantic space to obtain the environmental feature vector. The thermal attribute feature vector generation module 504 is used to generate a thermal attribute feature vector corresponding to any target passenger based on the visual image stream and the thermal imaging data stream. The state perception module 505 is used to perceive the state of the target passenger based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger.

[0166] This application also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. Memory 603 is used to store computer programs; In one embodiment of this application, when the processor 601 executes a program stored in the memory 603, it performs the following steps: The system acquires environmental data streams, visual image streams, and thermal imaging data streams from the vehicle; identifies at least one target passenger within the vehicle based on the visual image streams and thermal imaging data streams; maps the environmental parameters corresponding to the environmental data streams to a high-dimensional semantic space to obtain environmental feature vectors; for any target passenger, generates a thermal attribute feature vector corresponding to the target passenger based on the visual image streams and thermal imaging data streams; and performs state perception on the target passenger based on the environmental feature vectors and the thermal attribute feature vectors corresponding to the target passenger.

[0167] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0169] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0170] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0171] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute any of the state-aware methods described in the above embodiments.

[0172] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the state-aware methods described in the above embodiments.

[0173] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0174] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0176] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the protection scope of this application.

Claims

1. A state-aware method, characterized in that, The method includes: Acquire vehicle environmental data streams, visual image streams, and thermal imaging data streams; Based on the visual image stream and thermal imaging data stream, at least one target passenger inside the vehicle is identified; The environmental parameters corresponding to the environmental data stream are mapped to a high-dimensional semantic space to obtain environmental feature vectors; For any of the target passengers, a thermal attribute feature vector corresponding to the target passenger is generated based on the visual image stream and the thermal imaging data stream; Based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger, the state of the target passenger is perceived.

2. The method according to claim 1, characterized in that, The step of mapping the environmental parameters corresponding to the environmental data stream to a high-dimensional semantic space to obtain an environmental feature vector includes: The environmental parameters corresponding to the environmental data stream are normalized to generate normalized environmental parameters; The normalized environmental parameters are mapped to the high-dimensional semantic space to obtain the environmental feature vector.

3. The method according to claim 1, characterized in that, The step of generating a thermal attribute feature vector corresponding to the target passenger based on the visual image stream and the thermal imaging data stream includes: Extract the target region image corresponding to the target passenger from the visual image stream; The target region image corresponding to the target passenger is processed to obtain the face region image and clothing region image corresponding to the target passenger; Based on the face region image corresponding to the target passenger, determine the implicit face feature vector and passenger attribute label; Based on the clothing area image corresponding to the target passenger, the implicit clothing feature vector and the estimated value of clothing thermal resistance are determined; Based on the thermal imaging data stream and the target area image corresponding to the target passenger, an implicit thermal distribution feature vector, a first thermal sensing value, a second thermal sensing value, and a thermal sensing difference are determined, wherein the first thermal sensing value is the average temperature value of the forehead area of ​​the target passenger, and the second thermal sensing value is the average temperature value of the nose tip area of ​​the target passenger. Based on the implicit face feature vector, the passenger attribute label, the implicit clothing feature vector, the estimated thermal resistance of the clothing, the implicit thermal distribution feature vector, the first thermal perception value, the second thermal perception value, and the thermal perception difference, a thermal attribute feature vector corresponding to the target passenger is generated.

4. The method according to claim 3, characterized in that, The step of determining the implicit facial feature vector and passenger attribute label based on the facial region image corresponding to the target passenger includes: Determine the basic facial feature map based on the facial region image corresponding to the target passenger; Global average pooling is performed on the basic facial feature map to obtain the implicit facial feature vector; The passenger attribute label is determined based on the implicit facial feature vector.

5. The method according to claim 3, characterized in that, The step of determining the implicit clothing feature vector and the estimated clothing thermal resistance value based on the clothing area image corresponding to the target passenger includes: Determine the basic clothing feature map based on the clothing area image corresponding to the target passenger; Global average pooling is performed on the clothing basic feature map to obtain the implicit clothing feature vector; The estimated thermal resistance of the clothing is determined based on the implicit clothing feature vector.

6. The method according to claim 3, characterized in that, The step of determining the implicit thermal distribution feature vector, the first thermal sensing value, the second thermal sensing value, and the thermal sensing difference based on the thermal imaging data stream and the target area image corresponding to the target passenger includes: Obtain the temperature distribution map corresponding to the target passenger from the thermal imaging data stream; The implicit thermal distribution feature vector is determined based on the temperature distribution map corresponding to the target passenger; Determine the coordinates of facial key points based on the target area image corresponding to the target passenger; Based on the facial key point coordinates, a first distribution area and a second distribution area are determined from the temperature distribution map corresponding to the target passenger, wherein the first distribution area is the area centered on the forehead key point, and the second distribution area is the area centered on the nose tip key point. The average temperature corresponding to the first distribution area in the temperature distribution map is taken as the first thermal sensing value. The average temperature corresponding to the second distribution area in the temperature distribution map is used as the second thermal sensing value; The difference between the first thermal sensing value and the second thermal sensing value is determined as the thermal sensing difference.

7. The method according to claim 1, characterized in that, The process of perceiving the state of the target passenger based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger includes: The attention weight matrix is ​​determined by using the thermal attribute feature vector as the query term and the environmental feature vector as the key term. The environmental feature vector is weighted using the attention weight matrix to obtain a weighted environmental feature vector; The thermal attribute feature vector corresponding to the target passenger is fused with the weighted environmental feature vector to obtain a multimodal fusion feature vector; Based on the multimodal fusion feature vector, the thermal state evaluation result of the target passenger is generated.

8. A state sensing device, characterized in that, The device includes: The data stream acquisition module is used to acquire the vehicle's environmental data stream, visual image stream, and thermal imaging data stream; A target passenger determination module is used to determine at least one target passenger inside the vehicle based on the visual image stream and the thermal imaging data stream. An environment feature vector generation module is used to map the environment parameters corresponding to the environment data stream to a high-dimensional semantic space to obtain an environment feature vector. A thermal attribute feature vector generation module is used to generate a thermal attribute feature vector corresponding to any of the target passengers based on the visual image stream and the thermal imaging data stream. The state perception module is used to perceive the state of the target passenger based on the environmental feature vector and the thermal attribute feature vector corresponding to the target passenger.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images