A vehicle virtual interior display method, device, medium, equipment and product

Abstract

The present application relates to a kind of vehicle virtual interior display method, device, medium, equipment and product, main technical scheme includes: the visual image and inertial measurement data of acquisition augmented reality AR glasses collection;Based on visual image and inertial measurement data, the head pose of the user wearing AR glasses is determined;The pose of target dynamic component in vehicle is detected;Based on the pose of target dynamic component, head pose and interior theme package, the computer aided design CAD model of target dynamic component is rendered, to generate composite picture;Synthetic picture is transmitted to AR glasses and is shown.By wearing AR glasses, user can flexibly select the interior effect that wants to present according to mood, travel scene or external environment, greatly improve the experience of user, and, by detecting the pose of target dynamic component, accurate display of vehicle virtual interior is realized, and the display effect of vehicle virtual interior is improved.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, specifically to a method, apparatus, medium, equipment, and product for displaying virtual interiors of vehicles. Background Technology

[0002] In the field of automotive personalization, the visual style of the vehicle's interior is a crucial factor influencing the user's perceived experience. Currently, the main way to change the exterior of the vehicle's interior is by physically replacing or adding interior components (such as seat covers, dashboard trim, door panel covers, etc.). However, this process usually requires professional disassembly and assembly, which is cumbersome, time-consuming, and costly. Furthermore, the style of physically replaced or added interior components is relatively fixed, making it difficult for users to flexibly adjust the visual presentation of the vehicle's interior according to their mood, travel scenarios, or changes in the external environment, thus limiting their pursuit of a diverse and personalized interior style experience. Summary of the Invention

[0003] The purpose of this invention is to provide a method, device, medium, equipment and product for displaying virtual interior of a vehicle, so as to overcome the limitations of traditional physical replacement of interior parts, such as complicated operation, high cost and fixed style.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for displaying a vehicle's virtual interior, the method comprising: Acquire visual images and inertial measurement data captured by augmented reality (AR) glasses; Based on the visual image and the inertial measurement data, the head position of the user wearing the AR glasses is determined; Detect the pose of the target dynamic component inside the vehicle; Based on the pose of the target dynamic component, the pose of the head, and the interior theme package, the computer-aided design CAD model of the target dynamic component is rendered to generate a composite image. The synthesized image is transmitted to the AR glasses for display.

[0005] Based on the aforementioned technical means, the user's head pose is obtained through visual images and inertial measurement data collected by AR glasses, and the pose of target dynamic components inside the vehicle is detected. Based on the pose of the target dynamic components, the head pose, and the interior theme package, the CAD model of the target dynamic components is rendered to generate a composite image, which is then transmitted to the AR glasses for display. This method solves the problems of complex operation, high cost, and fixed style associated with traditional physical replacement of interior parts. By wearing AR glasses, users can flexibly choose the desired interior effect according to their mood, travel scenario, or external environment, greatly improving the user experience. Furthermore, by detecting the pose of the target dynamic components, the problem of inconsistency between the virtual and real scenes caused by adjustments to dynamic components such as seats is solved, achieving accurate display of the vehicle's virtual interior and improving the display effect of the virtual interior.

[0006] Furthermore, detecting the pose of the target dynamic component within the vehicle includes: Acquire the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment; Based on the pose change data and the pose of the target dynamic component at the reference time, the pose of the target dynamic component at the current time is determined.

[0007] Based on the above technical means, the pose of the target dynamic component at the current moment is determined by the pose and pose change data of the target dynamic component at the reference time. In other words, the present invention does not require global relocalization every time. It only needs to track the pose change data and combine it with the known pose to efficiently and continuously calculate the accurate pose at the current moment, providing a reliable input for real-time rendering.

[0008] Furthermore, acquiring the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment includes: Obtain component adjustment signals from the vehicle's CAN bus; The adjustment signal of the component is analyzed to obtain the pose change data of the target dynamic component in the vehicle at the current time relative to the reference time.

[0009] Based on the above technical means, by directly parsing the component adjustment signals from the CAN bus, it is possible to directly obtain the precise mechanical motion commands or status feedback issued by the seat control unit, providing a highly deterministic and real-time data source for determining the dynamic component pose changes, and improving the accuracy of pose change data.

[0010] Furthermore, acquiring the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment includes: Based on the visual image and the inertial measurement data, a three-dimensional point cloud corresponding to the target dynamic component at the current moment is generated; The three-dimensional point cloud corresponding to the target dynamic component at the current time is matched with the three-dimensional point cloud corresponding to the target dynamic component at the reference time to obtain the point cloud matching result; Based on the point cloud matching results, the pose change data of the target dynamic component inside the vehicle at the current time relative to the reference time is determined.

[0011] Based on the above technical means, by generating and comparing the three-dimensional point clouds of the target dynamic parts at different times, the actual pose change data of the target dynamic parts can be calculated based on visual geometric changes, thereby improving the authenticity of the pose change data.

[0012] Furthermore, acquiring the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment includes: The component adjustment signal is obtained from the vehicle's CAN bus, and the component adjustment signal is analyzed to obtain the first pose change data of the target dynamic component in the vehicle at the current time relative to the reference time. Based on the visual image and the inertial measurement data, a three-dimensional point cloud corresponding to the target dynamic component at the current moment is generated. The three-dimensional point cloud corresponding to the target dynamic component at the current moment is matched with the three-dimensional point cloud corresponding to the target dynamic component at the reference moment to obtain the point cloud matching result. Based on the point cloud matching result, the second pose change data of the target dynamic component in the vehicle at the current moment relative to the reference moment is determined. Determining the pose of the target dynamic component at the current moment based on the pose change data and the pose of the target dynamic component at the reference time includes: Based on the first pose change data, the second pose change data, and the pose of the target dynamic component at the reference time, the pose of the target dynamic component at the current time is determined.

[0013] Based on the above technical means, by simultaneously acquiring and processing the first pose change data from the CAN bus and the second pose change data calculated based on visual point cloud matching, the advantages of the two perception modes can be combined to make up for the limitations of a single sensor, thereby obtaining a more accurate and stable pose estimation of the target dynamic parts and effectively suppressing visual noise and mechanical signal drift.

[0014] Further, determining the pose of the target dynamic component at the current time based on the first pose change data, the second pose change data, and the pose of the target dynamic component at the reference time includes: Based on the pose of the target dynamic component at the reference time and the first pose change data, the prior pose estimate of the target dynamic component at the current time and its corresponding prior covariance matrix are predicted through the kinematic model of the target dynamic component. The prior covariance matrix is ​​used to characterize the degree of uncertainty of the prior pose estimate. Based on the prior pose estimation and the preset observation model, the theoretical observation value is determined, and the difference between the theoretical observation value and the second pose change data is determined as the observation residual. The Kalman gain is calculated based on the prior covariance matrix and the preset observation noise matrix. The observation residual is weighted using the Kalman gain, and based on the prior pose estimation and the weighted observation residual, the posterior pose estimation of the target dynamic component at the current time is obtained. The posterior pose estimation is then determined as the pose of the target dynamic component at the current time.

[0015] Based on the above technical means, by adopting the extended Kalman filter algorithm, the CAN signal can be predicted based on the kinematic model to obtain the prior pose estimate. Then, the prediction result is corrected based on visual observation (i.e., the second pose change data), which realizes more accurate weighted fusion, thereby outputting the pose of the target dynamic component in a continuous, smooth and high-precision manner during the dynamic process.

[0016] Furthermore, the composite image includes image content corresponding to the area inside the cockpit and image content corresponding to the area outside the cockpit; The process of rendering the computer-aided design (CAD) model of the target dynamic component based on its pose, head pose, and interior theme package to generate a composite image includes: Acquire illumination data for the area inside the cockpit and illumination data for the area outside the cockpit; The in-cabin rendering parameters for the image content of the in-cabin area are determined based on the lighting data of the in-cabin area. The vehicle cabin exterior rendering parameters for the image content of the area outside the cabin are determined based on the lighting data of the area outside the cabin. Based on the pose of the target dynamic component, the head pose, the interior theme package, the interior rendering parameters, and the exterior rendering parameters, the computer-aided design CAD model of the target dynamic component is rendered to generate a composite image.

[0017] Based on the aforementioned technical means, by logically dividing the composite image into two areas, the interior and exterior of the cabin, and determining independent rendering parameters for each, the problem of overexposure of the virtual interior of the vehicle under strong light or dimness under low light caused by using globally unified rendering parameters is avoided. This ensures the global visual consistency and realism of the virtual content inside and outside the cabin under complex and ever-changing real lighting conditions.

[0018] Furthermore, the step of obtaining the illumination data of the cabin area includes: obtaining a seat surface image of the seat surface area collected by the AR glasses, extracting the RGB mean from the seat surface image, and determining the illuminance value and color temperature based on the RGB mean, and determining the illuminance value and the color temperature as the illumination data of the cabin area; The step of acquiring the illumination data of the area outside the cockpit includes: acquiring a windshield image of the windshield area collected by the AR glasses, extracting the external environment brightness distribution and sky color shift based on the windshield image using high dynamic range (HDR) imaging technology, and determining the external environment brightness distribution and sky color shift as the illumination data of the area outside the cockpit.

[0019] Based on the aforementioned technical methods, the illuminance and color temperature inside the cabin are calculated using the RGB mean value of the seat surface image, while the external brightness distribution and sky color shift are extracted from the windshield image. This method of image acquisition and parameterization, which distinguishes different regions, can accurately reflect the lighting characteristics of object surfaces inside the cabin and the external environment, providing precise input data for subsequent differentiated rendering and ensuring the accuracy and consistency of adaptive lighting adjustment.

[0020] Furthermore, the in-cabin rendering parameters include physically based rendering (PBR) parameters for the virtual interior of the vehicle for the target dynamic component, and the PBR parameters include at least metallicity, roughness, and self-illumination intensity. The rendering parameters for the exterior of the vehicle include white balance for the virtual scenery outside the window.

[0021] Based on the aforementioned technical methods, by specifying PBR parameters (including attributes such as metallicity, roughness, and self-illumination intensity) for the interior areas of the vehicle cabin, it is possible to accurately simulate the appearance of different virtual interior materials (such as leather and metal) under realistic lighting, based on the principles of physical optics. Simultaneously, specifying white balance parameters for the exterior areas of the vehicle cabin can correct the color cast of the sky in the virtual landscape outside the window. This parameter division and definition allows the rendering process to specifically and precisely adjust the visual effects of different parts of the image, improving the realism and blending of the composite image.

[0022] A virtual interior display device for vehicles, the device comprising: The acquisition unit is configured to acquire visual images and inertial measurement data captured by augmented reality (AR) glasses; The determining unit is configured to determine the head position of the user wearing the AR glasses based on the visual image and the inertial measurement data. The detection unit is configured to detect the pose of a target dynamic component within the vehicle; The generation unit is configured to render the computer-aided design CAD model of the target dynamic component based on the pose of the target dynamic component, the pose of the head, and the interior theme package, so as to generate a composite image. The display unit is configured to transmit the synthesized image to the AR glasses for display.

[0023] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0024] An electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in the embodiments of the present invention.

[0025] A computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0026] The beneficial effects of this invention are: This invention uses visual images and inertial measurement data collected by AR glasses to obtain the user's head pose and detect the pose of target dynamic components inside the vehicle. Based on the pose of the target dynamic components, the head pose, and interior theme packages, the CAD model of the target dynamic components is rendered to generate a composite image, which is then transmitted to the AR glasses for display. This method solves the problems of complex operation, high cost, and fixed style associated with traditional physical replacement of interior parts. By wearing AR glasses, users can flexibly choose the desired interior effect according to their mood, travel scenario, or external environment, greatly improving the user experience. Furthermore, by detecting the pose of the target dynamic components, the problem of inconsistency between the virtual and real scenes caused by the adjustment of dynamic components such as seats is solved, achieving accurate display of the vehicle's virtual interior and improving the display effect of the virtual interior. Attached Figure Description

[0027] Figure 1 This is a system architecture diagram to which the present invention applies.

[0028] Figure 2 A flowchart of the vehicle virtual interior display method provided by the present invention.

[0029] Figure 3 This is a flowchart illustrating the overall steps of an embodiment of a virtual interior display for a vehicle provided by the present invention.

[0030] Figure 4 A schematic block diagram of the vehicle virtual interior display device provided by the present invention.

[0031] Figure 5 A schematic block diagram of the electronic device provided by the present invention. Detailed Implementation

[0032] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0033] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0034] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0035] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0036] In the field of vehicle interior personalization, to achieve dynamic switching of interior styles and an immersive experience, related technologies typically employ two approaches: one is to adjust the theme of the central control screen and the color of ambient lighting through software; the other is to customize the surface of interior parts through physical processes such as printing and bonding during the manufacturing process. The former relies on changes in the output of the in-vehicle display unit and light source, creating different atmospheres by changing the user interface (UI) theme package or adjusting the parameters of the red-green-blue (RGB) LED beads; the latter solidifies patterns and textures onto the surface of the interior substrate during the production stage, allowing users to use this fixed appearance permanently after purchasing the vehicle.

[0037] However, solutions that adjust the theme of the central control screen or the color of the ambient lighting only apply to the vehicle display unit and the light source itself, and cannot cover physical surfaces such as seats, door panels, and roofs; solutions based on printing technology have their appearance fixed at the factory, and users cannot switch them in real time according to the scenario or mood during use.

[0038] In view of this, this application provides a new approach. To facilitate understanding of this application, the system architecture on which this application is based will first be described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture may include: Augmented Reality (AR) glasses, vehicle domain controller, and cloud server.

[0039] Among them, AR glasses, as terminal devices worn on the user's head, can be equipped with binocular RGB cameras, time-of-flight depth sensors, six-axis inertial measurement units, microphone arrays, optical display units, and wireless communication modules, etc., to collect visual images, inertial measurement data, depth information, and raw perception data such as user voice and gestures, and to optically overlay and display the composite images sent by the vehicle domain controller.

[0040] The vehicle infotainment domain controller is a high-performance in-vehicle computing unit integrated into the vehicle's center console. It runs a real-time operating system and can be configured with an on-chip system processor, graphics processing unit, local memory, and wireless communication module. The vehicle infotainment domain controller interacts bidirectionally with AR glasses via a wireless link, and also connects to electronic control units such as seat control units and body control units within the vehicle via a Controller Area Network (CAN) bus interface located in the vehicle, in order to obtain component adjustment signals for dynamic parts.

[0041] In some embodiments, the vehicle cabin is equipped with dynamic components and static components. Dynamic components include components with one or more degrees of freedom of movement, such as seat backrests, seat cushions, leg rests, headrests, and lumbar supports, whose motion states are collected by the seat control unit and output via the CAN bus; static components include components whose positions remain substantially unchanged during vehicle use, such as the dashboard, center console, door panels, and headliner.

[0042] The vehicle's domain controller has pre-stored component 3D sub-models generated from the vehicle's original computer-aided design (CAD) model. These component 3D sub-models can refer to the CAD models of each dynamic component and each static component (or the CAD model of the overall static component formed by the static components) separated from the vehicle's original CAD model. It also has pre-stored several interior theme packages, which include material textures, animation effects (such as flowing water and snow particles), and sound effects (such as ocean waves and birdsong).

[0043] In addition, auxiliary sensors such as surround-view cameras and millimeter-wave radar can be deployed in the vehicle to assist AR glasses in obtaining spatial information of occluded areas during the initial cabin scanning phase, so as to obtain a more accurate initial 3D point cloud of the vehicle cabin.

[0044] Regarding communication connectivity, the AR glasses and the vehicle's infotainment system (VIS) can be connected via a wired connection. Preferably, a point-to-point connection is established between the AR glasses and the VIS via a high-speed wireless direct link. This link carries the raw perception data stream from the AR glasses and the composite image stream from the vehicle's infotainment system, supporting millisecond-level end-to-end latency. The VIS exchanges data with various electronic control units within the vehicle via a CAN bus to acquire adjustment signals from dynamic components such as seats. Furthermore, the VIS can connect to a cloud server via a cellular network to automatically detect whether new interior theme packages are available or if resources within existing interior theme packages have been updated. If so, incremental downloads and differential updates can be performed to avoid the time-consuming process of transmitting the entire interior theme package.

[0045] As a specific implementation, the raw perception data, such as visual images and inertial measurement data collected in real time by the AR glasses, are transmitted to the vehicle domain controller via a wireless uplink; the component adjustment signals generated by the seat control unit and other components inside the vehicle are transmitted to the vehicle domain controller via the CAN bus; the vehicle domain controller generates a composite image based on the above-mentioned raw perception data and component adjustment signals, and in combination with the locally stored CAD model and interior theme package; the composite image is transmitted back to the AR glasses via a wireless downlink, and presented to the user by the optical display unit of the AR glasses.

[0046] Figure 2 This is a flowchart of a vehicle virtual interior display method provided in an embodiment of this application. The method can be... Figure 1 The vehicle-mounted domain controller in the system shown executes this. For example... Figure 2 As shown, the method may include the following steps: Step 201: Obtain visual images and inertial measurement data captured by augmented reality (AR) glasses.

[0047] Step 202: Determine the head pose of the user wearing AR glasses based on visual images and inertial measurement data.

[0048] Step 203: Detect the pose of the target dynamic component inside the vehicle.

[0049] Step 204: Based on the pose of the target dynamic part, the head pose, and the interior theme package, render the computer-aided design CAD model of the target dynamic part to generate a composite image.

[0050] Step 205: Transmit the synthesized image to the AR glasses for display.

[0051] As can be seen from the above process, this invention obtains the user's head pose by collecting visual images and inertial measurement data through AR glasses, and detects the pose of target dynamic components inside the vehicle. Based on the pose of the target dynamic components, the head pose, and the interior theme package, the CAD model of the target dynamic components is rendered to generate a composite image, which is then transmitted to the AR glasses for display. This method solves the problems of complex operation, high cost, and fixed style associated with traditional physical replacement of interior parts. By wearing AR glasses, users can flexibly choose the desired interior effect according to their mood, travel scenario, or external environment, greatly improving the user experience. Furthermore, by detecting the pose of the target dynamic components, the problem of inconsistency between the virtual and real scene spatial relationships caused by the adjustment of dynamic components such as seats is solved, achieving accurate display of the vehicle's virtual interior and improving the display effect of the vehicle's virtual interior.

[0052] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments. It should be noted that the terms "first" and "second" involved in this disclosure do not have limitations in terms of size, order, or quantity, but are only used to distinguish them in name. For example, "first pose change data" and "second pose change data" are used to distinguish between two pose changes.

[0053] First, the above step 201, namely "acquiring visual images and inertial measurement data collected by augmented reality (AR) glasses", will be described in detail with reference to the embodiments.

[0054] The application acquires visual images and inertial measurement data collected by AR glasses. In this application, visual images can refer to image information reflecting the three-dimensional geometry and texture of the cockpit surrounding the user; these can be binocular RGB images or depth maps acquired using a time-of-flight depth sensor. Inertial measurement data can refer to physical quantities that describe the motion state of the AR glasses in space, and can include triaxial acceleration and triaxial angular velocity.

[0055] As a specific implementation method, AR glasses, as a terminal device worn on the user's head, can collect visual images within the user's field of vision in real time through its configured binocular RGB camera. At this time, the visual images can be a sequence of binocular visual frames continuously output by the AR glasses' binocular RGB camera. At the same time, it can collect inertial measurement data including three-axis acceleration and three-axis angular velocity in real time through its configured six-axis inertial measurement unit. At this time, the inertial measurement data can be the three-axis acceleration and three-axis angular velocity output by the inertial measurement unit at a sampling frequency of approximately 1000Hz.

[0056] The acquired visual images and inertial measurement data can be lightly compressed and then uploaded to the vehicle's domain controller via a high-speed wireless direct link through the wireless communication module in the AR glasses. In some embodiments, the high-speed wireless direct link can establish a peer-to-peer (P2P) connection using the WiFi 7 protocol (IEEE 802.11be). P2P specifically refers to a wireless communication mode that does not rely on traditional wireless access points (such as in-vehicle routers or home wireless routers) for data relay. In short, it refers to establishing a direct and dedicated wireless connection between the AR glasses and the vehicle's domain controller. The operating frequency band for the above communication process can be 6GHz, and the end-to-end uplink latency can be controlled to within approximately 5 milliseconds, thereby providing timely data input for subsequent real-time positioning and rendering.

[0057] Optionally, AR glasses can also include components such as microphone arrays and optical display units. The microphone array can be used to collect user voice commands, and the optical display unit can employ waveguide display technology with a field of view (FOV) of no less than 55° (FOV refers to the angular range of space that a human eye or device can observe in an optical or display system, usually expressed in degrees. In AR glasses, the larger the FOV, the wider the real field of view covered by the virtual image seen through the lenses, resulting in a stronger sense of immersion). The resolution per eye can be 1080p, used to present the composited image transmitted from the vehicle's domain controller to the user. It should be noted that while AR glasses can handle all image processing and rendering tasks, in practical applications, to ensure the lightweight design of AR glasses, their primary function is perception and display, while computational tasks such as image processing and rendering are centrally handled by the vehicle's domain controller.

[0058] The following describes step 202, namely "determining the head posture of a user wearing AR glasses based on visual images and inertial measurement data", in detail with reference to the embodiments.

[0059] After receiving visual images and inertial measurement data from AR glasses, the vehicle's domain controller can estimate the head pose of the user wearing the AR glasses in real time using a Visual-Inertial Odometry (VIO) algorithm. VIO is an algorithm that fuses visual images and inertial measurement data to estimate the motion trajectory and posture of devices (such as AR glasses, mobile phones, and robots) in three-dimensional space in real time. Its working principle is as follows: the visual component perceives environmental motion by tracking feature points in consecutive image frames, but is susceptible to changes in lighting and rapid movement; the inertial component calculates motion using three-dimensional acceleration and angular velocity, but suffers from drift and error accumulation. VIO tightly couples (i.e., mutually corrects) the two, using visual information to correct inertial measurement drift and inertial measurement information to fill in motion predictions when visual tracking is lost, thus achieving high-precision and robust pose estimation in complex dynamic environments. As a specific implementation method, head pose can include information on the user's head in six degrees of freedom, including the three-dimensional position (x, y, z) and three-dimensional attitude (pitch, yaw, roll) in a preset reference coordinate system. This fully represents the position and orientation of the user's head in the cockpit space and provides the necessary geometric reference for accurately presenting virtual content to the corresponding position in the user's field of vision.

[0060] In some embodiments, the user's head pose may be determined based solely on visual images or solely on inertial measurement data.

[0061] The following describes step 203, namely "detecting the pose of the target dynamic component inside the vehicle," in detail with reference to the embodiments.

[0062] In a vehicle cabin setting, users can view virtual interior scenes using AR glasses for an immersive experience. However, the vehicle cabin contains various adjustable dynamic components (such as seat backs, leg rests, and cushions), which undergo positional changes during use. If the virtual interior scene fails to match the actual positions of these dynamic components in real time, a misalignment will occur between the virtual interior scene and the real cabin structure, severely impacting the user's visual experience.

[0063] Therefore, this invention can detect the pose of a target dynamic component within a vehicle in real time. Here, the target dynamic component can refer to a component within the vehicle cabin that has one or more degrees of freedom of motion, and whose position or orientation can change during use. Examples include seat sub-components with independent degrees of freedom of motion, such as the backrest, cushion, leg rest, headrest, and lumbar support of the front seats, or other cabin components with mechanical adjustment capabilities. Correspondingly, the pose of the target dynamic component can refer to its position and orientation within a preset reference coordinate system.

[0064] Specifically, the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment is obtained, and the pose of the target dynamic component at the current moment is determined based on the pose change data and the pose of the target dynamic component at the reference moment.

[0065] The pose change data mentioned here is used to characterize the pose shift of the target dynamic component from the reference time to the current time. For example, the pose change data may include the amount of translational and / or rotational change of the target dynamic component within this time interval (from the reference time to the current time).

[0066] It should be noted that the reference time can refer to the initialization time, i.e., the moment when the user first activates the AR glasses and completes the initial cabin scan. In other embodiments, the reference time can also refer to the moment corresponding to the previous frame, i.e., pose estimation is performed in a frame-by-frame recursive manner, which helps to reduce the impact of long-term cumulative drift.

[0067] The pose of the target dynamic component at the reference time can refer to the pose of the target dynamic component at the time corresponding to the previous frame, or it can refer to the pose at the initialization time. For example, if the pose of the target dynamic component at the reference time refers to the pose of the target dynamic component at the initialization time, the AR glasses can scan the cabin environment to obtain an initial 3D point cloud of the cabin (if the vehicle is also equipped with auxiliary sensors such as surround-view cameras and millimeter-wave radar, the initial 3D point cloud of the cabin can be determined based on the data jointly collected by the AR glasses, surround-view cameras, and millimeter-wave radar, etc.), and match and register the initial 3D point cloud of the cabin with the pre-stored CAD models of each dynamic component to determine the pose of each dynamic component at the initialization time.

[0068] The above method eliminates the need for global relocalization of the target dynamic component during pose estimation. Instead, it efficiently and continuously calculates the precise pose of the target dynamic component at the current moment by overlaying pose change data onto the known pose at a reference time. As a specific implementation, the vehicle domain controller can record the pose of the target dynamic component at a reference time and store it in local memory. Then, it acquires pose change data at fixed intervals (e.g., every 50 milliseconds). Based on the target dynamic component's pose at the reference time and the pose change data, it determines the target dynamic component's pose at the current moment and updates the target dynamic component's pose at the reference time in the local memory using the current pose.

[0069] Next, the method for acquiring pose change data will be described. This application does not limit the method for acquiring pose change data, but includes, but is not limited to, the following three implementation methods.

[0070] The first implementation method is to obtain the component adjustment signal from the vehicle's CAN bus, parse the component adjustment signal, and obtain the pose change data of the target dynamic component in the vehicle at the current moment relative to the reference moment.

[0071] Among them, component adjustment signals can refer to signals output by electronic control units such as the vehicle seat control unit via the CAN bus, used to describe the current mechanical adjustment state of the target dynamic component. As a specific implementation, the vehicle's domain controller can connect to the seat control unit via the CAN bus interface to acquire the current component adjustment signals of the seat in real time, including the fore-and-aft displacement of the seat cushion, the tilt angle of the backrest, the extension length of the leg rest, and the position of the lumbar support, etc. Its displacement accuracy can reach ±1mm, and its angle accuracy can reach ±0.5°. That is to say, the above component adjustment signals can be represented by displacement (e.g., the fore-and-aft displacement of the seat cushion from zero to 50cm) or by angle (e.g., the rotation angle of the backrest around the hinge axis), and the specific form can be determined according to the motion characteristics of different dynamic components.

[0072] The vehicle's domain controller can parse received component adjustment signals. For example, it can map message fields to specific mechanical physical quantities (such as displacement and angle) using a pre-agreed CAN message protocol, and then determine the pose change data based on these mechanical physical quantities. This design can directly acquire precise mechanical motion commands or status feedback generated by the seat control unit, thus avoiding potential problems with computational load and environmental interference in pure vision solutions. Its advantages include low latency, high reliability, and independence from the AR glasses' field of view coverage; even if the target dynamic component is not within the AR glasses' field of view, its pose change data can still be acquired normally.

[0073] It should be noted that the source of component adjustment signals is not limited to the seat control unit, but can also come from other electronic control units related to movable components in the cabin, such as the steering wheel adjustment control unit and the door control unit.

[0074] The second implementation method is to generate a 3D point cloud corresponding to the target dynamic component at the current moment based on visual images and inertial measurement data; match the 3D point cloud corresponding to the target dynamic component at the current moment with the 3D point cloud corresponding to the target dynamic component at the reference moment to obtain the point cloud matching result; and determine the pose change data of the target dynamic component in the vehicle at the current moment relative to the reference moment based on the point cloud matching result.

[0075] Specifically, the Simultaneous Localization and Mapping (SLAM) engine in the vehicle's domain controller can reconstruct sparse or dense feature maps of local areas of the cabin in real time based on visual images and inertial measurement data uploaded from AR glasses, using a tightly coupled visual-inertial SLAM algorithm, and extract 3D point clouds belonging to the surfaces of target dynamic components. As a specific implementation, the SLAM engine can first determine the approximate area of ​​the target dynamic component in the visual image based on the current user's head pose and a pre-stored CAD model, and then perform triangulation reconstruction on the image features within that area to obtain the 3D point cloud corresponding to the target dynamic component at the current moment. For example, for cabin components such as the steering wheel, the corresponding 3D point cloud can be presented as cylindrical, ring-shaped, or other structures; for the seat back and cushion, the corresponding 3D point cloud can be presented as a set of points with corresponding curved surface features.

[0076] Subsequently, the vehicle domain controller can match the 3D point cloud corresponding to the target dynamic component at the current moment with the 3D point cloud corresponding to the same target dynamic component at the reference moment. As a specific implementation, the Iterative Closest Point (ICP) algorithm can be used to register the two sets of 3D point clouds. By iteratively solving for the optimal rigid body transformation matrix, the sum of the distances between corresponding points in the current moment's point cloud and the reference moment's point cloud is minimized, thus obtaining the point cloud matching result. The point cloud matching result can be represented as a rigid body transformation matrix containing translational and rotational components. Based on the point cloud matching result, the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment can be determined.

[0077] This design can calculate the pose changes of target dynamic components based on visual geometric changes, thereby providing positioning capabilities for components that cannot be connected to the CAN bus. It can also provide redundant visual observation channels for fusion schemes, enhancing the robustness and versatility of the system. Its advantage lies in its low dependence on hardware interfaces, requiring only the visual and inertial sensing capabilities of the AR glasses themselves. Optionally, the 3D point cloud corresponding to the target dynamic component at the reference time can be the initial cockpit 3D point cloud obtained from the initialization phase scan, or it can be the 3D point cloud obtained based on the visual image at the time corresponding to the previous frame; this application does not impose any restrictions on this.

[0078] The third implementation method involves acquiring component adjustment signals from the vehicle's CAN bus, parsing these signals to obtain the first pose change data of the target dynamic component within the vehicle at the current moment relative to the reference moment. Based on visual images and inertial measurement data, a 3D point cloud corresponding to the target dynamic component at the current moment is generated. This 3D point cloud is then matched with the 3D point cloud corresponding to the target dynamic component at the reference moment to obtain the point cloud matching result. Based on the point cloud matching result, the second pose change data of the target dynamic component within the vehicle at the current moment relative to the reference moment is determined. Finally, based on the first and second pose change data, the pose change data of the target dynamic component within the vehicle at the current moment relative to the reference moment is determined. For example, the first and second pose change data are weighted and summed according to preset weight values ​​to obtain the pose change data of the target dynamic component within the vehicle at the current moment relative to the reference moment.

[0079] This design combines the high determinism of mechanical signals with the directness of visual observation, overcoming the limitations of a single sensor. This results in more accurate and stable pose change data than either source, effectively suppressing visual noise and mechanical signal drift. Its advantage lies in the fact that even with minor deviations in the CAN bus data or brief interference with visual observation, this solution can still maintain high pose estimation accuracy through cross-verification of the two data sources.

[0080] As a preferred implementation, after obtaining the first pose change data and the second pose change data, an extended Kalman filter can be used to fuse the first pose change data and the second pose change data to obtain the final pose of the target dynamic component at the current moment.

[0081] As a specific implementation, the state variables of the extended Kalman filter include the pose of the target dynamic component, with both the first pose change data and the second pose change data serving as observation inputs. Specifically, based on the pose of the target dynamic component at a reference time, the prior pose estimate of the target dynamic component at the current time and its corresponding prior covariance matrix are predicted using the kinematic model of the target dynamic component. The prior covariance matrix is ​​used to characterize the uncertainty of the prior pose estimate. Based on the prior pose estimate and a preset observation model, a first theoretical observation value is determined. The first theoretical observation value is the theoretical pose change data expected to be obtained by parsing the component adjustment signal in the CAN bus when the dynamic component is assumed to be in the pose indicated by the prior pose estimate. The difference between the first theoretical observation value and the first pose change data is determined as the first observation residual. Based on the prior pose estimate... Based on a preset observation model, a second theoretical observation value is determined. The second theoretical observation value is the theoretical pose change data expected to be obtained through point cloud matching when the target dynamic component is assumed to be in the pose indicated by the prior pose estimation. The difference between the second theoretical observation value and the second pose change data is determined as the second observation residual. Based on the prior covariance matrix and the preset observation noise matrix, the Kalman gain is calculated. The first observation residual and the second observation residual are weighted and fused using the Kalman gain. Based on the weighted and fused observation residual and the prior pose estimation, the posterior pose estimation of the target dynamic component at the current time is obtained. The posterior pose estimation is determined as the pose of the target dynamic component at the current time.

[0082] As another specific embodiment, the state variables of the extended Kalman filter include the spatial pose of the dynamic component, with the first pose change data serving as the control input for state prediction and the second pose change data serving as the observation input. Specifically, based on the pose and first pose change data of the target dynamic component at the reference time, the prior pose estimate of the target dynamic component at the current time and its corresponding prior covariance matrix are predicted through the kinematic model of the target dynamic component. The prior covariance matrix is ​​used to characterize the uncertainty of the prior pose estimate. Based on the prior pose estimate and a preset observation model, theoretical observation values ​​are determined. The theoretical observation values ​​are the theoretical pose change data expected to be obtained through point cloud matching when the target dynamic component is assumed to be in the pose indicated by the prior pose estimate. The difference between the theoretical observation value and the second pose change data is determined as the observation residual. Based on the prior covariance matrix and a preset observation noise matrix, the Kalman gain is calculated. The observation residual is weighted using the Kalman gain, and based on the prior pose estimate and the weighted observation residual, the posterior pose estimate of the target dynamic component at the current time is obtained. The posterior pose estimate is determined as the pose of the target dynamic component at the current time.

[0083] The kinematic model of the target dynamic component can be constructed based on its degrees of freedom. For example, a linear translation model can be used for a seat cushion that translates along the slide rail; a kinematic model of rotation about a fixed axis can be used for a backrest that rotates about a hinge axis. The observation model can map the prior pose estimate to theoretical observations, and the form of the theoretical observations can be consistent with the expression of the first or second pose change data. The specific value of the observation noise matrix is ​​not limited to a fixed value and can be adjusted within a reasonable range according to the actual sensor noise characteristics and operating scenario.

[0084] For example, when the seat cushion is detected to have slid forward 10cm, the vehicle domain controller can first use the first pose change data to predict the prior pose estimate of the seat cushion at the current moment under the kinematic model, and then combine the second pose change data obtained based on point cloud matching to correct the prior pose estimate, so that the seat cushion in the synthesized image keeps synchronous translation, ensuring that the virtual image is completely consistent with the real image.

[0085] Furthermore, those skilled in the art will understand that, in addition to extended Kalman filtering, alternative algorithms that achieve the same purpose may include unscented Kalman filtering (UKF), particle filtering, graph optimization, etc., all of which can achieve fusion estimation of pose state under multi-source observation conditions.

[0086] The following describes in detail step 204, namely, "rendering the computer-aided design CAD model of the target dynamic component based on the pose, head pose, and interior theme package of the target dynamic component to generate a composite image," with reference to the embodiments.

[0087] Based on the pose, head pose, and interior theme package of the target dynamic component, the CAD model of the target dynamic component is rendered to generate a composite image. Specifically, the real-time rendering engine deployed in the vehicle domain controller can call the pre-stored 3D sub-models of the components and the interior theme package in the local memory, and combine them with the pose and head pose of the target dynamic component at the current moment to render the CAD model of the target dynamic component, thereby generating a composite image corresponding to the user's viewpoint.

[0088] The CAD models of the target dynamic components can be obtained as follows: The cockpit structure is divided into multiple independent sub-components based on function and motion characteristics, including static components such as the dashboard, center console, door panels, headliner, and windshield, as well as dynamic components such as seat backs, cushions, leg rests, and headrests. The original vehicle CAD model is obtained, and based on this model, refined mesh modeling is performed on each independent sub-component, creating 3D sub-models with textured coordinates. These sub-models can be pre-stored in the local memory of the vehicle's domain controller using a lightweight GL Transmission Format (glTF) or Universal Scene Description (USDZ) format. It is important to note that static components can be modeled as a whole, while dynamic components can be modeled separately, allowing for independent pose updates and rendering for specific dynamic components undergoing pose changes during seat adjustment.

[0089] In this application, an interior theme package can refer to a collection of resources used to define the overall visual style of a virtual interior, which may include resources such as texture maps, animation effects, and sound effects. As a specific implementation, an interior theme package may include stylized themes such as a bamboo forest, a beach, or a starry night sky. Taking the bamboo forest theme as an example, the center console, seats, roof, and doors inside the vehicle can use a soft and elegant Chinese-style green color scheme, while the view outside the windows can showcase a bamboo forest swaying in the wind. Optionally, the vehicle's domain controller can pre-store 5 to 10 interior theme packages locally at the factory. Each interior theme package can include resources such as texture maps, animation effects, and sound effects, with the total capacity controlled within 32GB to ensure efficient data transmission. When a vehicle connects to a cloud server via a cellular network (such as a 6G cellular link), the vehicle's domain controller can automatically detect if a new interior theme package is available on the cloud server. If so, the new interior theme package can be downloaded (e.g., via Over-The-Air (OTA) technology) so that the new virtual interior can be displayed to the user later. To avoid consuming excessive bandwidth and time, the interior theme package update uses a differential transmission method, that is, only the changes of the new interior theme package compared to the existing interior theme package (such as newly added textures) are downloaded, and then merged locally in the vehicle's domain controller, thereby significantly shortening the download time. Users can switch the virtual interior style of the cabin according to their personal preferences or scenario needs (i.e., select different interior theme packages) to enhance the personalized cabin experience, while also supporting continuous updates and expansions of themes via OTA.

[0090] It's worth noting that the real-time rendering engine can be custom-developed based on a graphics engine (such as Unity or Unreal Engine) when generating composite images. As a specific implementation, the vehicle's domain controller can first update the position and orientation of the corresponding CAD model in a preset reference coordinate system based on the detected pose of the target dynamic component, ensuring that the CAD model's pose in the preset reference coordinate system matches its pose in the real scene. Then, based on the user's head pose, the position and orientation of the virtual camera in the preset reference coordinate system are determined. Subsequently, resources such as textures and animation effects from the interior theme package are loaded to give the CAD model a corresponding visual style. Finally, the rendering engine generates the composite image according to the virtual camera's perspective, with the frame rate set to 120Hz.

[0091] Similarly, for static components, rendering can be done by directly obtaining their poses. For example, based on the initial 3D point cloud of the cockpit, the poses of the static components can be obtained. When generating the composite image, the CAD models of the target dynamic components and the static components are rendered based on the poses of the target dynamic components, the poses of the static components, the head pose, and the interior theme package to generate the composite image.

[0092] Next, to address the visual disconnect and distortion between the virtual interior and the exterior scenery caused by heterogeneous and asynchronous changes in lighting conditions inside and outside the vehicle cabin, this application further provides the following preferred solution. When generating the composite image, lighting data for the interior area and the exterior area of ​​the cabin are acquired. Interior rendering parameters for the interior area are determined based on the interior lighting data, and exterior rendering parameters for the exterior area are determined based on the exterior lighting data. Based on the pose, head pose, interior theme package, interior rendering parameters, and exterior rendering parameters of the target dynamic component, the CAD model of the target dynamic component is rendered to generate the composite image.

[0093] In other words, the vehicle domain controller can analyze and parameterize the two sets of acquired lighting data. The cabin interior area refers to the image regions in the composite image corresponding to the cabin's internal structure and interior design, such as the content corresponding to the center console, door panels, and seat surfaces. Generally, cabin interior rendering parameters are used when rendering the content of the cabin interior area. The cabin exterior area refers to the image regions in the composite image corresponding to light-transmitting structures such as windows and sunroofs, as well as their external virtual scenery, such as virtual bamboo forests, beaches, and starry skies. Generally, cabin exterior rendering parameters are used when rendering the content of the cabin exterior area. As a specific implementation method, the vehicle domain controller can sample and update the lighting data approximately every 50 milliseconds, supporting gradual transitions to prevent visual discomfort caused by sudden changes in lighting.

[0094] This design can independently adjust the visual attributes of different areas of the screen content (such as virtual interior and virtual scenery) based on the actual differences in lighting conditions inside and outside the cabin. This avoids the problem of cabin materials being overexposed in strong light or dim in low light caused by using globally uniform lighting parameters. Its advantage lies in maintaining global visual consistency and realism of virtual content inside and outside the cabin in various complex scenarios such as strong midday sun, dim evening light, and backlighting.

[0095] To further clarify the specific methods for acquiring cabin interior and exterior lighting data, this application also provides the following preferred solutions. In a preferred embodiment, the lighting data for the cabin interior area can be obtained by: acquiring a seat surface image of the seat surface area captured using AR glasses; extracting the RGB mean value from the seat surface image; and determining the illuminance value and color temperature based on the RGB mean value, thus defining the illuminance value and color temperature as the lighting data for the cabin interior area. The lighting data for the cabin exterior area can be obtained by: acquiring a windshield image of the windshield area captured using AR glasses; and extracting the external environment brightness distribution and sky color shift based on the windshield image using High Dynamic Range (HDR) imaging technology, thus defining the external environment brightness distribution and sky color shift as the lighting data for the cabin exterior area.

[0096] It should be noted that the external ambient brightness distribution can refer to the brightness variations of the cabin's external environment in different directions or areas. For example, the external ambient brightness distribution can characterize the difference in brightness between the left and right sides of the vehicle when sunlight shines from the left, or the overall uniform brightness without strong directionality in a cloudy environment. The external ambient brightness distribution can be used to provide directional light intensity input for the lighting effects in the virtual scene (such as the contrast between light and dark caused by the sun's position and the direction of shadow projection) when rendering virtual scenery outside the window, ensuring that the lighting distribution of the virtual scenery is consistent with the lighting distribution of the actual external environment.

[0097] Sky color cast refers to the color shift characteristics of light in the external sky. For example, the sky at midday on a sunny day tends to be bluish (color temperature approximately 5500K-6500K), while at sunset it tends to be orange-red (color temperature approximately 2000K-3000K), and on a cloudy day it tends to be grayish-white (color temperature approximately 6000K-7000K). Sky color cast can be used to correct the overall color tone of virtual landscape images outside a window, harmonizing the color atmosphere of the virtual landscape with the actual sky light and avoiding visual conflicts between the virtual landscape and real skylight.

[0098] By employing the aforementioned method of calculating illuminance and color temperature from the RGB mean of seat surface images within the cabin, and extracting external ambient brightness distribution and sky color shift from HDR images of the windshield outside the cabin, this preferred solution can further improve the accuracy and consistency of adaptive lighting adjustment. Furthermore, those skilled in the art will understand that the sampling area for lighting data within the cabin is not limited to the seat surface area; it can also include areas such as the inner door panel and trunk surface that reflect the actual lighting conditions inside the cabin. Similarly, the sampling area for lighting data outside the cabin is not limited to the windshield area; it can also be extended to the sunroof area, etc., and this application does not impose any limitations in this regard.

[0099] As a preferred implementation, the in-cabin rendering parameters include physically based rendering (PBR) parameters for the virtual interior of the vehicle targeting dynamic components, and the PBR parameters include at least metallicity, roughness, and self-illumination intensity; the out-of-cabin rendering parameters include white balance for the virtual scenery outside the window.

[0100] Among them, metallicity describes the degree to which the material of the virtual interior of the vehicle exhibits metallic properties when it reflects incident light, and can be used to distinguish different materials such as leather, fabric, and metal trim; roughness describes the microscopic unevenness of the material surface, and can be used to control the sharpness of specular reflection; self-illumination intensity describes the intensity of the material's own light emission, and can be used to appropriately enhance the glow of the material under low light conditions. For example, under low light conditions, the real-time rendering engine can appropriately increase the self-illumination intensity to avoid the image appearing hazy.

[0101] In addition to PBR parameters, cabin rendering parameters can also include contrast and shadow depth for the virtual interior of the vehicle targeting dynamic parts. For example, under strong light conditions, the real-time rendering engine can enhance the contrast and shadow depth of the virtual interior materials to avoid overexposure or a plastic look in the overall image.

[0102] For virtual scenery outside the window, the real-time rendering engine can automatically correct the white balance based on the brightness distribution of the external environment and the color cast of the sky. For example, it can use a cool-toned bamboo forest landscape under the midday sun and switch to a warm orange beach landscape under the evening sunset, thus ensuring that the virtual scenery outside the window blends naturally with the real sky light in terms of color tone.

[0103] This design allows the rendering process to precisely adjust the visual effects of different parts of the image. For the cabin area, by introducing PBR parameters, it's possible to accurately simulate the appearance of different virtual materials (such as leather, metal, and fabric) under realistic lighting based on physical optics principles. For the area outside the cabin, by introducing white balance parameters, it's possible to correct the color cast of the virtual scenery outside the windows, making it consistent with real daylight. Thus, a high level of visual presentation is achieved in both physical realism and environmental integration.

[0104] Furthermore, those skilled in the art will understand that the rendering parameters inside the vehicle cabin are not limited to metallicity, roughness, and self-illumination intensity, but may also include other PBR parameters such as normal intensity, ambient occlusion, and subsurface scattering; the rendering parameters outside the vehicle cabin, in addition to white balance, may also include parameters such as exposure, tone mapping curve, and fog density, in order to further optimize the visual effect of the virtual scenery outside the window.

[0105] The following describes step 205, namely "transmitting the synthesized image to AR glasses for display," in detail with reference to the embodiments.

[0106] After generating the composite image, the vehicle's domain controller can perform lightweight compression to create a low-bandwidth video stream, which is then transmitted to the AR glasses via a high-speed wireless direct link. As a specific implementation, the downlink can use the same WiFi 7 protocol as the uplink, and introduce forward error correction (FEC) and frame prediction mechanisms at the link layer. FEC allows the receiver to recover some lost data packets without retransmission, while the frame prediction mechanism predicts the content of the current frame based on the motion vector of the previous frame, maintaining smooth display even in cases of extreme latency or brief frame drops. This ensures users receive a continuous augmented reality experience in dynamic scenes, with end-to-end latency controlled to within approximately 10 milliseconds. After receiving the downloaded composite image, the AR glasses can present the composite image to the user's retina through optical superposition using its optical display unit. The user can then see the effect of the virtual interior through the AR glasses. When the target dynamic parts change position, the AR glasses can render the CAD model of the target dynamic parts based on the real-time position of the target dynamic parts, thereby achieving a continuous and smooth change of the virtual interior image with the position of the target dynamic parts, avoiding the visual disharmony between the virtual image and the actual image.

[0107] It should be noted that in the process of rendering the CAD model of the target dynamic part based on the pose of the target dynamic part, the head pose, and the interior theme package to generate a composite image, the interior theme package can be determined by the theme selected by the user, or it can be determined by the theme automatically adapted to the external scene.

[0108] Specifically, for automatic theme adaptation, the vehicle's domain controller can analyze external environment images captured by AR glasses or the vehicle's surround-view cameras, combined with high-precision map information, to determine the current scene type of the vehicle (such as city, suburbs, seaside, forest, night, etc.). When it identifies an actual scene type that matches a pre-stored theme (such as "seaside beach"), it can proactively recommend or automatically switch to that theme to the user. In this case, the interior theme package used to generate the composite image is the interior theme package corresponding to that theme.

[0109] For users to manually select a theme, they can do so via pre-set physical buttons on the vehicle or AR glasses. In a preferred embodiment, this application provides an example of contactless interaction. In this embodiment, users can select the desired theme through various contactless input methods. These contactless input methods include, but are not limited to: voice commands, air gesture trajectories, and gaze duration.

[0110] Specifically, the AR glasses collect the user's voice commands through their integrated microphone array, capture the user's hand gestures in the air through their camera, and detect the user's gaze focus and duration through eye-tracking sensors. This raw data is transmitted to the vehicle's domain controller via a wireless uplink. After receiving the raw data, the vehicle's domain controller performs the following processing: For voice command processing: The natural language understanding engine parses the semantics of voice commands, such as recognizing the voice command "switch to the bamboo forest theme" and triggering the corresponding theme switching command.

[0111] For processing air gesture trajectories: The gesture trajectories are analyzed by a pre-trained 3D convolutional neural network to identify the type of gesture. For example, "slide right" indicates switching to the next topic, and "pinch to confirm" indicates selecting the current highlighted item.

[0112] Regarding the handling of gaze focus and dwell time: Through gaze timing judgment logic, when the user's gaze lingers on the virtual selection control corresponding to a certain topic for more than a preset time (e.g., 2 seconds), the topic is highlighted and a confirmation prompt pops up, waiting for the user to make a final confirmation through other means (such as gestures or voice).

[0113] This multimodal interaction method allows users to freely combine input methods according to the current scenario and personal preferences. For example, users can select a target with their gaze, issue confirmation with their voice, or switch topics by drawing circles in the air with their fingers. The entire process does not require touching any physical buttons, maintaining the continuity and immersion of the augmented reality experience. Furthermore, by concentrating the recognition and computing tasks on the more powerful vehicle domain controller, more complex models (such as large-vocabulary speech recognition and high-precision 3D gesture models) can be run, thereby improving the accuracy and response speed of the interaction.

[0114] Next, to explain this solution more clearly, a specific application scenario example will be used to illustrate it, such as... Figure 3 As shown.

[0115] Specifically, the user enters the vehicle's cabin and puts on lightweight AR glasses, then says "Switch to the beach theme" via voice. At this point, the AR glasses' built-in binocular RGB camera and inertial measurement unit begin collecting visual images and inertial measurement data from inside the cabin, and upload them in real time to the vehicle's domain controller via a high-speed Wi-Fi 7 link.

[0116] Based on visual images and inertial measurement data, the vehicle's infotainment domain controller determines the head posture of the user wearing AR glasses. Simultaneously, the controller connects to the seat control unit via the CAN bus interface to acquire component adjustment signals. These signals are then analyzed to obtain the first-order posture change data of the target dynamic component within the vehicle at the current moment relative to the reference moment.

[0117] Meanwhile, the vehicle's domain controller uses visual images and inertial measurement data uploaded from the AR glasses to perform real-time SLAM, generating a 3D point cloud of the target dynamic component at the current moment. This 3D point cloud is then iteratively matched with the 3D point cloud at the reference moment to obtain the point cloud matching result. Based on this result, the rotation and translation changes of the target dynamic component, such as the seat back and cushion, in space are determined, forming the second pose change data.

[0118] The vehicle-mounted domain controller employs an extended Kalman filter to fuse the first and second pose change data. Specifically, based on the pose of the target dynamic component at a reference time and the first pose change data from the CAN bus, the prior pose estimate and prior covariance matrix at the current time are predicted using the kinematic model corresponding to the target dynamic component. Based on the prior pose estimate and a preset observation model, the theoretical observation value is determined, and the difference between the theoretical observation value and the second pose change data is defined as the observation residual. Based on the prior covariance matrix and a preset observation noise matrix, the Kalman gain is calculated. The observation residual is weighted using the Kalman gain, and based on the prior pose estimate and the weighted observation residual, the posterior pose estimate of the target dynamic component at the current time is obtained. This posterior pose estimate is then determined as the pose of the target dynamic component at the current time.

[0119] The AR glasses capture images of the seat surface, extract the RGB mean value, and convert it into illuminance and color temperature as lighting data for the cabin area. Simultaneously, images of the windshield are captured, and HDR imaging technology is used to extract the ambient brightness distribution and sky color shift, serving as lighting data for the exterior of the cabin. Interior rendering parameters are determined based on the interior lighting data, and exterior rendering parameters are determined based on the exterior lighting data.

[0120] The rendering engine retrieves the interior theme package corresponding to the locally stored beach theme, and combines it with the CAD model of the target dynamic parts (seat backrest, seat cushion, etc.), the current pose of the target dynamic parts, the user's head pose, and the rendering parameters determined by region (including rendering parameters for the interior and exterior of the vehicle cabin), and performs real-time rendering in the vehicle's domain controller. The rendered high-frame-rate composite image is transmitted back to the AR glasses within 10 milliseconds via the Wi-Fi 7 link. Through the AR glasses, the user can see that the cabin has been transformed into a seaside resort style, with the virtual leather interior perfectly matching the shape of the real seats, and the beach scenery outside the window seamlessly blending with the real sky.

[0121] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0122] According to another embodiment, a virtual interior display device for a vehicle is provided. Figure 4 A schematic block diagram of a vehicle virtual interior display device according to one embodiment is shown, the device being disposed in... Figure 1 The vehicle-mounted domain controller in the architecture shown. For example... Figure 4 As shown, the device 400 includes an acquisition unit 401, a determination unit 402, a detection unit 403, a generation unit 404, and a display unit 405, wherein the main functions of each component unit are as follows: The acquisition unit 401 is configured to acquire visual images and inertial measurement data collected by augmented reality (AR) glasses.

[0123] The determining unit 402 is configured to determine the head pose of a user wearing AR glasses based on visual images and inertial measurement data.

[0124] The detection unit 403 is configured to detect the pose of a target dynamic component within the vehicle.

[0125] The generation unit 404 is configured to render the computer-aided design CAD model of the target dynamic part based on the pose, head pose, and interior theme package of the target dynamic part to generate a composite image.

[0126] Display unit 405 is configured to transmit the synthesized image to AR glasses for display.

[0127] As one possible implementation method, the detection unit 403, when detecting the pose of a target dynamic component inside the vehicle, can be specifically configured to: acquire pose change data of the target dynamic component inside the vehicle at the current moment relative to a reference moment; and determine the pose of the target dynamic component at the current moment based on the pose change data and the pose of the target dynamic component at the reference moment.

[0128] As one possible implementation method, when the detection unit 403 acquires the pose change data of the target dynamic component in the vehicle at the current time relative to the reference time, it can be specifically configured to: acquire the component adjustment signal from the vehicle's controller area network CAN bus; parse the component adjustment signal to obtain the pose change data of the target dynamic component in the vehicle at the current time relative to the reference time.

[0129] As one possible implementation method, when the detection unit 403 acquires the pose change data of the target dynamic component inside the vehicle at the current time relative to the reference time, it can be specifically configured to: generate a three-dimensional point cloud corresponding to the target dynamic component at the current time based on the visual image and inertial measurement data; match the three-dimensional point cloud corresponding to the target dynamic component at the current time with the three-dimensional point cloud corresponding to the target dynamic component at the reference time to obtain the point cloud matching result; and determine the pose change data of the target dynamic component inside the vehicle at the current time relative to the reference time based on the point cloud matching result.

[0130] The detection unit 403, when determining the pose of the target dynamic component at the current time based on the pose change data and the pose of the target dynamic component at the reference time, can be specifically configured to: determine the pose of the target dynamic component at the current time based on the first pose change data, the second pose change data, and the pose of the target dynamic component at the reference time.

[0131] As one possible implementation method, the detection unit 403, when determining the pose of the target dynamic component at the current time based on the first pose change data, the second pose change data, and the pose of the target dynamic component at the reference time, can be specifically configured as follows: Based on the pose of the target dynamic component at the reference time and the first pose change data, predict the prior pose estimate of the target dynamic component at the current time and its corresponding prior covariance matrix through the kinematic model of the target dynamic component. The prior covariance matrix is ​​used to characterize the uncertainty of the prior pose estimate. Based on the prior pose estimate and a preset observation model, determine the theoretical observation value, and determine the difference between the theoretical observation value and the second pose change data as the observation residual. Based on the prior covariance matrix and a preset observation noise matrix, calculate the Kalman gain. Use the Kalman gain to weight the observation residual, and based on the prior pose estimate and the weighted observation residual, obtain the posterior pose estimate of the target dynamic component at the current time, and determine the posterior pose estimate as the pose of the target dynamic component at the current time.

[0132] As one possible approach, the composite image includes both the image content corresponding to the area inside the cockpit and the image content corresponding to the area outside the cockpit.

[0133] The generation unit 404, when rendering the computer-aided design CAD model of the target dynamic component based on its pose, head pose, and interior theme package to generate a composite image, can be specifically configured to: acquire lighting data of the cabin interior area and the cabin exterior area; determine the cabin interior rendering parameters based on the lighting data of the cabin interior area; determine the cabin exterior rendering parameters based on the lighting data of the cabin exterior area; and render the computer-aided design CAD model of the target dynamic component based on its pose, head pose, interior theme package, cabin interior rendering parameters, and cabin exterior rendering parameters to generate a composite image.

[0134] As one possible implementation method, when acquiring the illumination data of the cabin area, the generation unit 404 can be specifically configured to: acquire a seat surface image of the seat surface area collected by AR glasses, extract the RGB mean from the seat surface image, determine the illuminance value and color temperature based on the RGB mean, and determine the illuminance value and color temperature as the illumination data of the cabin area.

[0135] When acquiring illumination data for the area outside the cockpit, the generation unit 404 can be specifically configured to: acquire a windshield image of the windshield area collected by AR glasses, extract the external environment brightness distribution and sky color shift based on the windshield image using high dynamic range (HDR) imaging technology, and determine the external environment brightness distribution and sky color shift as the illumination data for the area outside the cockpit.

[0136] As one possible approach, the in-cabin rendering parameters include physically based rendering (PBR) parameters for the virtual interior of the vehicle targeting dynamic components, with PBR parameters including at least metallicity, roughness, and self-illumination intensity; the out-of-cabin rendering parameters include white balance for the virtual scenery outside the window.

[0137] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0139] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0140] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0141] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0142] in, Figure 5 An exemplary architecture of an electronic device is shown, which may include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520 can communicate with each other via a communication bus 530.

[0143] The processor 510 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.

[0144] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store the operating system 521 for controlling the operation of the electronic device 500, and the basic input / output system (BIOS) 522 for controlling the low-level operations of the electronic device 500. Additionally, it can store a web browser 523, a data storage management system 524, and a vehicle virtual interior display device 400, etc. The aforementioned vehicle virtual interior display device 400 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 520 and executed by the processor 510.

[0145] Input / output interface 513 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0146] Network interface 514 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0147] Bus 530 includes a pathway for transmitting information between various components of the device, such as processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520.

[0148] It should be noted that although the above-described device only shows the processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, memory 520, bus 530, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0149] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0150] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for displaying a vehicle's virtual interior, characterized in that, The method includes: Acquire visual images and inertial measurement data captured by augmented reality (AR) glasses; Based on the visual image and the inertial measurement data, the head position of the user wearing the AR glasses is determined; Detect the pose of the target dynamic component inside the vehicle; Based on the pose of the target dynamic component, the pose of the head, and the interior theme package, the computer-aided design CAD model of the target dynamic component is rendered to generate a composite image. The synthesized image is transmitted to the AR glasses for display.

2. The method according to claim 1, characterized in that, The detection of the pose of the target dynamic component inside the vehicle includes: Acquire the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment; Based on the pose change data and the pose of the target dynamic component at the reference time, the pose of the target dynamic component at the current time is determined.

3. The method according to claim 2, characterized in that, The step of acquiring the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment includes: Obtain component adjustment signals from the vehicle's CAN bus; The adjustment signal of the component is analyzed to obtain the pose change data of the target dynamic component in the vehicle at the current time relative to the reference time.

4. The method according to claim 2, characterized in that, The step of acquiring the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment includes: Based on the visual image and the inertial measurement data, a three-dimensional point cloud corresponding to the target dynamic component at the current moment is generated; The three-dimensional point cloud corresponding to the target dynamic component at the current time is matched with the three-dimensional point cloud corresponding to the target dynamic component at the reference time to obtain the point cloud matching result; Based on the point cloud matching results, the pose change data of the target dynamic component inside the vehicle at the current time relative to the reference time is determined.

5. The method according to claim 2, characterized in that, The step of acquiring the pose change data of the target dynamic component inside the vehicle at the current moment relative to the reference moment includes: The component adjustment signal is obtained from the vehicle's CAN bus, and the component adjustment signal is analyzed to obtain the first pose change data of the target dynamic component in the vehicle at the current time relative to the reference time. Based on the visual image and the inertial measurement data, a three-dimensional point cloud corresponding to the target dynamic component at the current moment is generated. The three-dimensional point cloud corresponding to the target dynamic component at the current moment is matched with the three-dimensional point cloud corresponding to the target dynamic component at the reference moment to obtain the point cloud matching result. Based on the point cloud matching result, the second pose change data of the target dynamic component in the vehicle at the current moment relative to the reference moment is determined. Determining the pose of the target dynamic component at the current moment based on the pose change data and the pose of the target dynamic component at the reference time includes: Based on the first pose change data, the second pose change data, and the pose of the target dynamic component at the reference time, the pose of the target dynamic component at the current time is determined.

6. The method according to claim 5, characterized in that, Determining the pose of the target dynamic component at the current moment based on the first pose change data, the second pose change data, and the pose of the target dynamic component at the reference time includes: Based on the pose of the target dynamic component at the reference time and the first pose change data, the prior pose estimate of the target dynamic component at the current time and its corresponding prior covariance matrix are predicted through the kinematic model of the target dynamic component. The prior covariance matrix is ​​used to characterize the degree of uncertainty of the prior pose estimate. Based on the prior pose estimation and the preset observation model, the theoretical observation value is determined, and the difference between the theoretical observation value and the second pose change data is determined as the observation residual. The Kalman gain is calculated based on the prior covariance matrix and the preset observation noise matrix. The observation residual is weighted using the Kalman gain, and based on the prior pose estimation and the weighted observation residual, the posterior pose estimation of the target dynamic component at the current time is obtained. The posterior pose estimation is then determined as the pose of the target dynamic component at the current time.

7. The method according to claim 1, characterized in that, The composite image includes the image content corresponding to the area inside the cockpit and the image content corresponding to the area outside the cockpit. The process of rendering the computer-aided design (CAD) model of the target dynamic component based on its pose, head pose, and interior theme package to generate a composite image includes: Acquire illumination data for the area inside the cockpit and illumination data for the area outside the cockpit; The in-cabin rendering parameters for the image content of the in-cabin area are determined based on the lighting data of the in-cabin area. The vehicle cabin exterior rendering parameters for the image content of the area outside the cabin are determined based on the lighting data of the area outside the cabin. Based on the pose of the target dynamic component, the head pose, the interior theme package, the interior rendering parameters, and the exterior rendering parameters, the computer-aided design CAD model of the target dynamic component is rendered to generate a composite image.

8. The method according to claim 7, characterized in that, The step of obtaining the illumination data of the cabin area includes: obtaining a seat surface image of the seat surface area collected by the AR glasses, extracting the RGB mean from the seat surface image, and determining the illuminance value and color temperature based on the RGB mean, and determining the illuminance value and color temperature as the illumination data of the cabin area; The step of acquiring the illumination data of the area outside the cockpit includes: acquiring a windshield image of the windshield area collected by the AR glasses, extracting the external environment brightness distribution and sky color shift based on the windshield image using high dynamic range (HDR) imaging technology, and determining the external environment brightness distribution and sky color shift as the illumination data of the area outside the cockpit.

9. The method according to claim 7, characterized in that, The in-cabin rendering parameters include physically based rendering (PBR) parameters for the virtual interior of the vehicle for the target dynamic component, and the PBR parameters include at least metallicity, roughness, and self-illumination intensity. The rendering parameters for the exterior of the vehicle include white balance for the virtual scenery outside the window.

10. A virtual interior display device for vehicles, characterized in that, The device includes: The acquisition unit is configured to acquire visual images and inertial measurement data captured by augmented reality (AR) glasses; The determining unit is configured to determine the head position of the user wearing the AR glasses based on the visual image and the inertial measurement data. The detection unit is configured to detect the pose of a target dynamic component within the vehicle; The generation unit is configured to render the computer-aided design CAD model of the target dynamic component based on the pose of the target dynamic component, the pose of the head, and the interior theme package, so as to generate a composite image. The display unit is configured to transmit the synthesized image to the AR glasses for display.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images