Star projection method, device, equipment, medium and program product of vehicle

Abstract

The present application relates to the technical field of intelligent driving, and discloses a starry sky projection method, device, equipment, medium and program product of a vehicle, real-time pose information of the vehicle is acquired, and real-time environment images around the vehicle are collected; a first observation direction based on the vehicle is determined based on the real-time pose information, and celestial body coordinate information of visible celestial bodies in a celestial coordinate system is determined according to the first observation direction; the superimposed positions of the visible celestial bodies in the real-time environment images are determined by using the celestial body coordinate information; the visual features of the visible celestial bodies are rendered to the superimposed positions, an augmented reality picture is obtained, and the augmented reality picture is displayed through a display component of the vehicle. The present application calculates the corresponding positions of the visible celestial bodies in the screen by fusing the real-time pose and the real scene images of the vehicle, and the visible celestial bodies are superimposed in real time to the real scene picture outside the vehicle, so that the user can obtain stable and high-precision starry sky augmented reality navigation on the vehicle display component without holding a device.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit technology, specifically to methods, devices, equipment, media, and program products for projecting starry skies onto vehicles. Background Technology

[0002] With the integration of smart electric vehicles and outdoor camping culture, the "camping mode" of vehicles has transformed them into mobile living spaces with continuous power supply, environmental control, and audio-visual functions. In this scenario, observing the clear starry sky has become a naturally extended and frequently desired activity, currently relying mainly on augmented reality (AR) stargazing applications on smartphones. By utilizing the phone's sensors and camera, virtual star maps are overlaid on the screen to achieve basic star recognition.

[0003] However, existing smartphone AR stargazing applications cannot deeply integrate with vehicles, causing them to operate independently of the vehicle system in in-vehicle camping scenarios. They cannot utilize high-precision in-vehicle sensors, multi-camera arrays, and large screens to improve recognition accuracy and immersion, nor can they rely on the vehicle's stable power supply and seat posture support. As a result, the observation process still relies on handheld operation, resulting in a cramped experience. Furthermore, the virtual stargazing is difficult to accurately integrate with the real-world scenery outside the vehicle, failing to provide users with a stable, comfortable, and highly immersive astronomical observation experience in in-vehicle scenarios. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, medium, and program product for projecting stars into a vehicle, in order to solve the problem of how to deeply integrate users' astronomical observation needs with the vehicle, thereby providing users with a stable, comfortable, and highly immersive astronomical observation experience in the in-vehicle setting.

[0005] In a first aspect, embodiments of the present invention provide a method for projecting stars onto a vehicle, the method comprising:

[0006] Acquire the vehicle's real-time pose information and capture real-time environmental images around the vehicle; Based on the real-time pose information, a first observation orientation with the vehicle as the reference is determined, and the celestial coordinate information of the visible celestial body in the celestial coordinate system is determined according to the first observation orientation. The superposition position of the visible celestial body in the real-time environmental image is determined using the celestial coordinate information; The visual features of the visible celestial bodies are rendered onto the overlay position to obtain an augmented reality image, which is then displayed through the vehicle's display components.

[0007] This invention acquires real-time vehicle pose and real-world environmental images, determines the observation benchmark based on the real-time pose, and calculates the celestial coordinates of the corresponding celestial bodies. Then, through coordinate mapping, it associates the celestial body positions with the real-world environmental images, rendering and merging the celestial bodies at the corresponding positions in the real-world environmental images. Compared to traditional solutions that rely on mobile phone sensors and screens, this invention utilizes the vehicle's display components and cabin environment to achieve a hands-free, stable, and precisely aligned virtual and real-world immersive stargazing experience, transforming the vehicle into a comfortable and professional astronomical observation platform.

[0008] In conjunction with the first aspect, in one implementation, the real-time pose information includes the three-dimensional attitude angles of the vehicle, and determining the first observation orientation based on the vehicle using the real-time pose information includes: Analyze multiple consecutive frames of the real-time environment image to extract and match star feature points from the real-time environment image, perform visual synchronous localization processing on the star feature points, and obtain vehicle attitude assistance data; The vehicle attitude assistance data and the three-dimensional attitude angles are fused together to generate corrected attitude information; The first observation azimuth is corrected based on the corrected attitude information.

[0009] The embodiments of the present invention ensure that even when the vehicle is stationary for a long time or driving in an environment with magnetic interference, the system can still maintain a long-term stable and accurate superposition of virtual starry sky information and real night sky image, thereby improving the reliability and professionalism of the user experience.

[0010] In conjunction with the first aspect, in one embodiment, determining the superposition position of the visible celestial body in the real-time environmental image using the celestial coordinate information includes: Based on the real-time pose information and the calibration parameters of the vehicle's onboard camera, a mapping relationship is established between the visual coordinate system of the onboard camera and the celestial coordinate system. Based on the celestial coordinate information of the visible celestial body and the mapping relationship, the superimposed position of the visible celestial body in the real-time environmental image is identified.

[0011] This invention combines the vehicle's real-time pose information with the inherent calibration parameters of the onboard camera to construct a mapping relationship from the camera's viewpoint to the celestial coordinate system. Based on this mapping, the system can accurately convert celestial coordinate information obtained from astronomical calculations into specific pixel positions in the camera image. This achieves pixel-level spatial alignment between virtual information such as celestial markers and the real starry sky outside the vehicle, ensuring the stability and high precision of augmented reality overlay. It effectively overcomes the image drift or misalignment problems caused by sensor errors and model simplification in traditional solutions, providing users with a realistic, immersive, and reliable stargazing experience.

[0012] In conjunction with the first aspect, in one embodiment, after displaying the augmented reality image via the vehicle's display component, the method further includes: In response to the user selecting a first target celestial body in the augmented reality scene, it is determined whether the first target celestial body is within the field of view of the vehicle's onboard camera; If not, then based on the celestial coordinate information of the first target celestial body and the real-time pose information, determine the offset of the first target celestial body relative to the user's real-time observation direction, wherein the real-time observation direction is determined based on the orientation of the user's seat in the vehicle; Based on the offset azimuth, a multimodal guidance strategy is generated and executed to guide the user to adjust the real-time observation direction until the first target celestial body enters the field of view.

[0013] This invention combines seat orientation to estimate the user's line of sight and calculates the precise azimuth deviation of the line of sight to celestial bodies. Then, by dynamically updating directional arrows, synchronizing voice navigation, and coordinating seat posture adjustments, a multi-sensory guidance loop is formed, enabling users to intuitively and conveniently adjust their personal observation posture, significantly improving the interactivity and overall immersive experience of astronomical observation.

[0014] In conjunction with the first aspect or its corresponding implementation, in one implementation, the multimodal guidance strategy includes at least one of the following: Directional indicator graphics are overlaid on the edge area of ​​the augmented reality image, and the direction of the directional indicator graphics is updated in real time as the three-dimensional attitude angle of the vehicle changes. The vehicle's audio system broadcasts a voice guidance command containing the offset direction, and the vehicle's seats are adjusted to face the offset direction based on the voice guidance command.

[0015] After presenting an augmented reality image of the starry sky on the in-vehicle display screen, this embodiment of the invention estimates the user's line of sight by combining the seat orientation and calculates the precise azimuth deviation of the celestial bodies relative to the line of sight. Then, by dynamically updating the directional arrows, synchronizing voice navigation, and coordinating seat posture adjustments, a multi-sensory guidance loop is formed, enabling users to intuitively and conveniently adjust their personal observation posture, significantly improving the interactivity and overall immersive experience of astronomical observation.

[0016] In conjunction with the first aspect, in one implementation, the real-time pose information further includes the vehicle's geographical location and reference time, and the method further includes: Acquire ephemeris data, which is used to characterize astronomical events of at least one celestial body in the future and the corresponding prediction data of the astronomical events; Filter and obtain at least one target astronomical event that is associated with the reference time and can be observed at the geographical location from the ephemeris data, and determine the prediction data corresponding to the target astronomical event; The predicted data corresponding to the target astronomical event is integrated and displayed on the augmented reality screen.

[0017] This invention upgrades the vehicle-mounted stargazing system from a static star map display to an intelligent platform with proactive service capabilities by introducing a dynamic astronomical event prediction function based on ephemeris data. The system filters and accurately calculates dynamic events visible locally, such as satellite transits and meteor showers, based on the vehicle's real-time geographical location and time. It then intuitively integrates the predicted event trajectories and times into the AR display, expanding the spatiotemporal dimensions and content depth of the service.

[0018] In conjunction with the first aspect or its corresponding implementation, in one implementation, after integrating and displaying the predicted data corresponding to the target astronomical event on the augmented reality screen, the method further includes: In response to a user's command to pay attention to any of the target astronomical events, the vehicle-mounted camera's second observation position is switched or guidance information is generated to remind the user to observe the target astronomical event.

[0019] Through the above-mentioned response mechanism, this embodiment of the invention achieves a seamless transition from passive event forecasting to proactive observation assistance, ensuring that users can obtain clear and timely operational guidance even in the face of dynamically changing events, greatly improving the success rate of astronomical event observation and the integrity of the user experience.

[0020] In conjunction with the first aspect, in one embodiment, the vehicle seat is equipped with a posture sensor, and the method further includes: The posture data of the seat is acquired through the posture sensor, and the user's real-time sitting posture is determined based on the posture data; The user's primary gaze direction is determined based on the mapping relationship between the real-time riding posture and the gaze direction. The second target celestial body corresponding to the main line of sight is highlighted in the augmented reality image, and the information of the second celestial body corresponding to the second target celestial body is broadcast through the audio component mounted on the vehicle.

[0021] This invention utilizes an attitude sensor to intelligently perceive the user's posture and estimate their primary line of sight, thereby enabling personalized and proactive delivery of stargazing content. The system automatically highlights prominent celestial objects in the area where the user's line of sight naturally falls, and simultaneously broadcasts relevant scientific information, significantly reducing the user's exploration burden in the complex night sky and greatly enhancing the guidance and immersion of observation.

[0022] In a second aspect, embodiments of the present invention provide a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the vehicle star projection method of the first aspect or any corresponding embodiment described above.

[0023] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to execute the star projection method for a vehicle according to the first aspect or any corresponding embodiment described above.

[0024] Fifthly, embodiments of the present invention provide a program product including computer instructions, the computer instructions being used to cause a computer to execute the vehicle star projection method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the structure of a vehicle's star projection system according to some embodiments of the present invention; Figure 2 This is a schematic flowchart of a method for projecting stars onto a vehicle according to some embodiments of the present invention; Figure 3 This is a schematic diagram of star projection according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a vehicle's star projection device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1This embodiment provides a vehicle-mounted starry sky projection system. The vehicle-mounted starry sky projection system in this embodiment comprises three layers: a data acquisition layer, a data processing and fusion layer, and an application and interaction layer. The data acquisition layer includes a GNSS (Global Navigation Satellite System) receiving module, a vehicle attitude sensing module, a high-precision clock module, a multi-channel image acquisition module, and a user interaction module. The GNSS receiving module acquires the vehicle's current latitude, longitude, and altitude information; the vehicle attitude sensing module includes an onboard inertial measurement unit and an electronic compass to acquire the vehicle's pitch, roll, and yaw angles; the high-precision clock module provides accurate Coordinated Universal Time (UTC) signals; the multi-channel image acquisition module includes forward-looking, surround-view, or in-cabin cameras to acquire real-time environmental images, including the night sky; and the user interaction module includes a touchscreen, a voice microphone, and a seat attitude sensor to receive user commands and status updates.

[0029] The data processing and fusion layer includes a positioning and attitude fusion module, a star catalog database, an astronomical computing engine, and a camera calibration and sky mapping module. The positioning and attitude fusion module is used to fuse geographic location, vehicle attitude, and time information; the star catalog database is used to store reference coordinate data of celestial bodies such as stars and planets; the astronomical computing engine is used to calculate the precise coordinates of visible celestial bodies based on spatiotemporal information; and the camera calibration and sky mapping module is used to establish coordinate mapping relationships based on camera parameters and vehicle attitude.

[0030] The application and interaction layer includes an AR (Augmented Reality) rendering and overlay module, a display output module, and an intelligent guidance and event service module. The AR rendering and overlay module is used to generate and overlay augmented reality information; the display output module contains display components for outputting the final image; and the intelligent guidance and event service module is used to generate interactive guidance and astronomical event reminders.

[0031] In one embodiment, the physical form and installation position of the display component can be designed in various ways according to different vehicle configurations, user experience requirements and technical implementation paths, including but not limited to vehicle display screens, window / sunroof projections, physical screens / roller blinds, multi-screen linkage and external display devices.

[0032] In a preferred embodiment, the display component can be an existing vehicle central control display screen, such as a central control screen, instrument panel display screen, or passenger entertainment screen. Central control displays typically feature high resolution, high brightness, and wide viewing angles, enabling them to clearly present a starry sky image incorporating virtual information. The system transmits the rendered augmented reality image to this display screen in real time via an in-vehicle video interface such as LVDS or GMSL, allowing users to view it directly from a comfortable seating position. This fully utilizes existing vehicle hardware, resulting in low cost and rapid deployment.

[0033] In another preferred embodiment, the display component may comprise a combination of an in-vehicle projection device and a vehicle window, particularly a sunroof or windshield. Specifically, the system outputs augmented reality images to a micro-projector or laser projection module, which projects the image onto a specially treated vehicle window or sunroof, such as one with a diffuse reflection coating or laminated projection film. While viewing the real starry sky through the glass, the user can also see superimposed virtual constellation markers, celestial names, and other information on the glass, creating a more direct augmented reality head-up display effect.

[0034] In another preferred embodiment, for vehicles equipped with electric sunshades or rear entertainment screens, the display component may further include a retractable physical screen, such as an electric projection screen. When the user activates the stargazing mode, the system can control the sunroof sunshade or rear screen to descend, with its back or front used as a projection screen. The in-vehicle projection device projects augmented reality images onto this screen, allowing the user to view the starry sky image on the screen in a completely dark cabin, while simultaneously catching glimpses of the real night sky through the transparent windows surrounding the screen, creating a semi-enclosed, cinematic stargazing experience.

[0035] In another preferred embodiment, in high-end vehicles, the display component can be a collaborative combination of multiple displays. For example, the main augmented reality image is displayed on the central control screen, while brief information or guidance instructions for key celestial bodies are simultaneously displayed on the instrument panel or head-up display; rear passengers can independently choose to watch starry sky images from different directions or popular science explanations of the same event through the rear entertainment screen, achieving an immersive shared experience for all passengers in the vehicle.

[0036] In another preferred embodiment, the display component may further include an external display device, such as augmented reality glasses or a virtual reality headset, that is wirelessly or wiredly connected to the vehicle system. The system transmits the generated augmented reality video stream or spatial data to the head-mounted device via Wi-Fi, Bluetooth, or 5G. When the user wears the device, they can obtain a more immersive private starry sky cinema experience with full field of vision, while the built-in sensors of the device can also be fused with vehicle data to further improve the accuracy of interaction.

[0037] In one embodiment, the basic data collected by the GNSS receiving module, vehicle attitude sensing module and high-precision clock module are first transmitted to the positioning and attitude fusion module for fusion processing. The positioning and attitude fusion module outputs the spatiotemporal coordinates and spatial pointing vector with a certain point on the vehicle as the observation origin, which determines the reference point and direction of the observed celestial body for the entire system.

[0038] The astronomical computing engine receives the fused data and calls upon celestial data from the star catalog database to calculate the real-time coordinates of all visible celestial bodies (such as stars and planets). Simultaneously, the camera calibration and sky mapping module receives real-time attitude data from the vehicle attitude sensing module and, combined with the pre-calibrated optical parameters of the camera, determines the celestial coordinates of each pixel in the real-time environmental image captured by the multi-channel image acquisition module.

[0039] The AR rendering and overlay module receives celestial coordinate information from all real-time visible celestial bodies from the astronomical computing engine and coordinate mapping relationships from the camera calibration and sky mapping module. Based on these mapping relationships, it accurately renders virtual celestial markers, constellation lines, and other graphic information onto the corresponding pixel positions in the real-time video stream to synthesize augmented reality images, which are ultimately presented to the user through the display output module.

[0040] The user interaction module is responsible for receiving user commands, such as clicking on a constellation on the screen or asking "Where is Jupiter?" via voice, and then passing the commands to the intelligent guidance and event service module.

[0041] The intelligent guidance and event service module provides guidance based on user requests, the position of the target celestial body, and the vehicle's attitude. If the target celestial body is not in the field of view, it generates guidance instructions, such as "Please look to the left and behind," and coordinates with the AR rendering and overlay module to add dynamic arrows at the edge of the image, or uses the audio component to provide voice announcements. Furthermore, the intelligent guidance and event service module can handle dynamic astronomical events, proactively triggering reminders or guidance processes when such events occur.

[0042] Audio components play a crucial role in conveying astronomical information, guidance instructions, and scientific content to users through auditory means. The implementation and coordination methods of audio components can be diversely designed according to different vehicle configurations, user experience requirements, and acoustic scenarios, including but not limited to the following embodiments: In a preferred embodiment, the audio component is a multi-channel surround sound system installed in the vehicle, typically comprising speaker units, such as tweeters, midrange speakers, and woofers, distributed in multiple locations including the front doors, rear doors, dashboard, rear shelf, and A / B pillars. The system outputs celestial information in stereo or surround sound format via an onboard amplifier and audio processor. For example, when guiding a user to look to the left rear, the system can prioritize broadcasting this instruction through the left rear surround channel, enhancing directional perception using sound image localization principles, allowing the user to intuitively understand the target's location through hearing.

[0043] In another preferred embodiment, the audio components include near-field speaker units integrated within the seat headrest. This design utilizes near-field acoustic principles to create a private acoustic zone near the user's ears, undisturbed by other passengers. When celestial information involves sensitive content or when the user wishes not to disturb other passengers' rest, the system can provide personalized announcements via the headrest speakers, such as explaining the mythology of a specific constellation only to the driver or front passenger.

[0044] In another preferred embodiment, for scenarios requiring an immersive experience or continued awareness of the external environment, the audio components may include an interface and adapter module for bone conduction headphones or in-ear personal speakers. Users can wear a bone conduction device that wirelessly connects to the vehicle system, which transmits astronomical information to the device via Bluetooth or a dedicated short-range communication protocol. Bone conduction technology transmits sound through skull vibrations without blocking the ear canal, allowing users to clearly perceive sounds of the real environment outside the vehicle, such as wind and insect chirps, while listening to astronomical explanations, achieving a high degree of integration between natural sounds and electronic narration.

[0045] In another preferred embodiment, in scenarios where a special atmosphere needs to be created, such as simulating a meteor streaking across the sky or a satellite passing overhead, the audio components can utilize the vehicle's independent subwoofer unit, typically installed in the trunk or under the seats, to output specific low-frequency sound effects, enhancing the impact and sense of ceremony of the experience.

[0046] In another preferred embodiment, the system synthesizes textual celestial information, such as celestial names, scientific information, and guidance instructions, into natural and fluent speech in real time. Simultaneously, through audio partitioning technology, independent broadcasts are achieved for different seating areas. For example, the driver's area broadcasts guidance information about the celestial bodies ahead, while the right rear passenger area simultaneously broadcasts scientific stories about the constellation corresponding to their side window, without interference.

[0047] In another preferred embodiment, the audio component may further include an external audio output device that connects wirelessly or wired to the vehicle system, such as a user-provided Bluetooth headset, smart speaker, or in-vehicle outdoor speaker. When the vehicle is in camping mode and the user wishes to lie down and stargaze outside the vehicle, the system can transmit celestial information to a portable speaker placed outside the vehicle, allowing the user to enjoy synchronized astronomical interpretation services even outside the vehicle.

[0048] The vehicle's star projection system in this embodiment achieves a complete technical loop from raw data acquisition to precise spatiotemporal and coordinate calculation, and then to virtual-real fusion rendering and intelligent interaction through a layered modular design, thereby transforming the intelligent vehicle into a high-precision, highly immersive mobile astronomical observation platform.

[0049] With the integration of smart electric vehicles and outdoor camping culture, vehicle camping has transformed into a mobile living space with continuous power supply, environmental control, and audio-visual functions. In this context, observing the clear starry sky has become a naturally extended and frequent demand, currently relying mainly on augmented reality (AR) stargazing applications on smartphones. By utilizing the phone's sensors and camera, virtual star maps are overlaid on the screen to achieve basic star recognition.

[0050] However, existing smartphone AR stargazing applications cannot deeply integrate with vehicles, causing them to operate independently of the vehicle system in in-vehicle camping scenarios. They cannot utilize high-precision in-vehicle sensors, multi-camera arrays, and large screens to improve recognition accuracy and immersion, nor can they rely on the vehicle's stable power supply and seat posture support. As a result, the observation process still relies on handheld operation, resulting in a cramped experience. Furthermore, the virtual stargazing is difficult to accurately integrate with the real-world scenery outside the vehicle, failing to provide users with a stable, comfortable, and highly immersive astronomical observation experience in in-vehicle scenarios.

[0051] Based on this, the present invention provides an embodiment of a method for projecting stars onto a vehicle. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0052] This embodiment provides a method for projecting stars onto a vehicle. Figure 2 This is a flowchart of a vehicle star projection method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the real-time pose information of the vehicle and acquire real-time environmental images around the vehicle.

[0053] Step S202: Determine the first observation azimuth based on the vehicle using real-time pose information, and determine the celestial coordinate information of visible celestial bodies in the celestial coordinate system based on the first observation azimuth.

[0054] Step S203: Use celestial coordinate information to determine the superposition position of visible celestial bodies in the real-time environmental image.

[0055] In step S204, the visual features of visible celestial bodies are rendered onto the overlay position to obtain an augmented reality image, which is then displayed through the vehicle's display components.

[0056] The vehicle-based starry sky projection method provided in this embodiment acquires the vehicle's real-time pose information and collects real-time environmental images around the vehicle; based on the real-time pose information, it determines a first observation orientation with the vehicle as the reference, and determines the celestial coordinate information of visible celestial bodies in the celestial coordinate system according to the first observation orientation; it uses the celestial coordinate information to determine the superposition position of the visible celestial bodies in the real-time environmental images; it renders the visualization features of the visible celestial bodies to the superposition position to obtain an augmented reality image, and displays the augmented reality image through the vehicle's display components. This embodiment calculates the corresponding position of the visible celestial bodies on the screen by fusing the vehicle's real-time pose with the real-world image, and superimposes the visible celestial bodies onto the real-world image outside the vehicle in real time, so that users can obtain a stable and high-precision augmented reality starry sky guide on the in-vehicle large screen without holding a handheld device.

[0057] Reference Figure 3 The image illustrates a typical application scenario where starlight information is projected onto a central control display screen inside a vehicle. In the diagram, the driver and passengers have adjusted their seatbacks to a comfortable reclining angle for observation, with their lines of sight naturally directed towards the central control display screen.

[0058] The central control display screen showcases multiple augmented reality graphic elements, such as constellation lines superimposed on real stars to form easily recognizable constellation outlines. Simultaneously, the vehicle's central control display screen allows occupants to view detailed information up close or interact with it via touch, enabling them to access in-depth astronomical information through touch controls.

[0059] The steps described above are explained in detail below.

[0060] In step S201, the real-time pose information of the vehicle is acquired, and real-time environmental images around the vehicle are collected.

[0061] Specifically, real-time pose information is a complete set of states describing the vehicle's spatial position and three-dimensional orientation at a specific moment. In this embodiment, the real-time pose information includes the vehicle's geographical location, reference time, and the vehicle's three-dimensional attitude angles.

[0062] Geographic location refers to the precise geographical coordinates of a vehicle on Earth, calculated in real time by the vehicle's onboard global navigation satellite system, which typically includes longitude, latitude, and altitude.

[0063] The reference time, Coordinated Universal Time (UTC), can be provided by a high-precision clock module in the vehicle's star projection system. As a globally unified standard time for astronomical calculations, the accuracy of UTC directly determines the accuracy of calculating the positions of celestial bodies such as the sun, moon, and planets, avoiding star chart deviations caused by local time or system clock errors.

[0064] Three-dimensional attitude angles refer to the three Euler angles of the vehicle's body coordinate system relative to the ground horizontal reference coordinate system, which are measured and calculated in real time by the vehicle's inertial measurement unit (IMU, which usually includes a three-axis gyroscope and a three-axis accelerometer) and electronic compass or magnetometer. Euler angles include yaw angle, pitch angle and roll angle. The yaw angle is the angle between the direction of the vehicle's front and true north, the pitch angle is the angle of the vehicle's forward and backward tilt, and the roll angle is the angle of the vehicle's left and right tilt.

[0065] Real-time environmental images refer to video streams or single-frame images containing actual night sky scenes, synchronously captured by vehicle-mounted cameras, such as front-view cameras, surround-view cameras, or cabin-mounted cameras specifically designed for stargazing.

[0066] The vehicle's geographical location, reference time, and three-dimensional attitude angle collected in this embodiment provide a stable spatiotemporal reference far exceeding that of mobile phone sensors for astronomical calculations and augmented reality rendering.

[0067] In step S202, a first observation azimuth with the vehicle as the reference is determined based on real-time pose information, and the celestial coordinate information of visible celestial bodies in the celestial coordinate system is determined based on the first observation azimuth.

[0068] The celestial coordinate system refers to an ideal spherical coordinate system centered on the Earth's center of mass or the observer, used to describe the orientation of celestial bodies in space. In this embodiment, the celestial coordinate system is specifically implemented as a geocentric inertial coordinate system, that is, an equatorial coordinate system with the Earth's center of mass as the origin and the coordinate axes pointing to the equator, which is fixed in inertial space (pointing to the vernal equinox).

[0069] First, establish multiple spatial coordinate systems, including the vehicle carrier coordinate system, the local horizontal coordinate system, the geocentric coordinate system, and the geocentric inertial coordinate system.

[0070] The vehicle carrier coordinate system refers to a three-dimensional Cartesian coordinate system fixed to the vehicle body. The origin of this coordinate system is usually defined as a predetermined observation center point, such as the geometric center of the sunroof. In this coordinate system, the X-axis can be defined as pointing directly forward of the vehicle, the Y-axis as pointing to the left side of the vehicle, and the Z-axis as pointing vertically upwards towards the roof.

[0071] The local horizontal coordinate system refers to the local tangent plane coordinate system with the observation center point as the origin. The three axes of this coordinate system point to the geographical east, north and vertical zenith directions, respectively. This coordinate system is used to correlate the vehicle's observation direction with the local horizontal plane and due north direction.

[0072] The geocentric-fixed coordinate system is a coordinate system whose origin is at the Earth's center of mass and rotates with the Earth. The geocentric inertial coordinate system also has the Earth's center of mass as its origin, but its coordinate axes point in a fixed direction in inertial space, for example, towards distant stars, and do not rotate with the Earth.

[0073] Subsequently, the geodetic coordinates of the observation center point under the Earth reference ellipsoid model were obtained using the GNSS receiving module. ,Right now Indicates longitude, Indicate latitude and This indicates the altitude. The observation center point is a preset, fixed spatial geometric reference point located on the vehicle, such as the geometric center point of the sunroof or panoramic glass roof when it is closed. Since this point is closest to the area where the user's line of sight naturally converges in the common in-car stargazing posture, such as when the passenger is lying on their back in the seat, the geometric center point can be selected as the observation center point.

[0074] Furthermore, using standard geodetic formulas, the aforementioned geodetic coordinates are... Convert to Earth-centered and Earth-fixed rectangular coordinates This allows us to determine the spatial position of the observation center point relative to the Earth's center of mass.

[0075] Furthermore, the three-dimensional attitude angles calculated by the fusion of the vehicle-mounted inertial measurement unit and electronic compass are read. The three-dimensional attitude angles specifically include the heading angle. Pitch angle and roll angle These three angles are used to uniquely determine the rotational relationship of the vehicle's coordinate system relative to the local horizontal coordinate system.

[0076] In the vehicle's coordinate system, the user's standard line of sight is defined as a fixed vector. For example, looking upwards vertically through a skylight. First, utilize the three-dimensional attitude angle. Construct the first rotation matrix , the fixed vector The coordinates are transformed from the vehicle coordinate system to the local horizontal coordinate system. Next, a second rotation matrix is ​​constructed using the latitude and longitude of the observation center point under the Earth reference ellipsoid model. The fixed vector is then further transformed to the Earth-centered Earth-fixed coordinate system. Finally, based on the third rotation matrix calculated from the reference time and corresponding to the Earth's rotation angle... The fixed vector is transformed from a geocentric, Earth-fixed coordinate system that rotates with the Earth to a geocentric, inertial coordinate system that does not rotate with the Earth, thus obtaining the direction vector of the first observation azimuth with the vehicle as the reference. .

[0077]

[0078] In the formula, This represents the direction vector of the first observation position relative to the vehicle; Represents the third rotation matrix; Indicates the second rotation matrix; Represents the first rotation matrix; This represents a fixed vector.

[0079] In one embodiment, the real-time pose information includes the vehicle's three-dimensional attitude angles, and determining a first observation orientation based on the vehicle using the real-time pose information includes: Analyze multiple consecutive frames of real-time environmental images to extract and match stellar feature points from the real-time environmental images, perform visual synchronous localization processing on the stellar feature points, and obtain vehicle attitude assistance data; Vehicle attitude assistance data is fused with three-dimensional attitude angles to generate corrected attitude information; The first observation azimuth is corrected based on the corrected attitude information.

[0080] In this embodiment, by identifying stellar feature points in the image and matching them with a star catalog database, it is possible to assist in correcting or verifying vehicle orientation data provided by sensors such as the vehicle-mounted inertial measurement unit and electronic compass. This is especially helpful when the vehicle-mounted inertial measurement unit and electronic compass are subject to short-term interference or have accumulated errors, which can further improve the system accuracy.

[0081] Specifically, based on the vehicle's three-dimensional attitude angles directly provided by sensors such as the onboard inertial measurement unit and electronic compass, the first observation azimuth with the vehicle's observation center point as the origin is initially calculated using standard coordinate transformation relationships (i.e., the first observation azimuth in the above steps is calculated). ).

[0082] Meanwhile, the system analyzes multiple frames of real-time environmental images continuously captured by the vehicle-mounted camera. Specifically, it identifies and extracts star feature points with stable brightness and outlines from the night sky images. Subsequently, these star feature points are tracked and matched between consecutive image frames to obtain the pixel displacement sequence of the same star on the image plane at different times.

[0083] Based on the pixel displacement sequence of stellar feature points obtained through the above matching, Visual Simultaneous Localization and Mapping (Visual SLAM) processing is performed. Specifically, by solving a visual geometry problem, the relative rotation change of the vehicle camera in inertial space between adjacent frames is calculated in reverse, i.e., vehicle attitude auxiliary data. Then, using data fusion algorithms such as Kalman filtering or complementary filtering, the vehicle attitude auxiliary data is fused with the three-dimensional attitude angles to estimate and compensate for errors in the vehicle's inertial measurement unit and sensors such as the electronic compass, such as gyroscope drift, in real time, thereby outputting a more accurate and stable set of corrected attitude information. The corrected attitude information replaces the original three-dimensional attitude angles, and coordinate transformation calculations are re-performed to correct the aforementioned first observation azimuth, resulting in a more accurate first observation azimuth based on the vehicle. This embodiment ensures that even when the vehicle is stationary for a long time or traveling in an environment with magnetic interference, the system can still maintain a long-term stable and accurate overlay of virtual starry sky information and real night sky images, improving the reliability and professionalism of the user experience.

[0084] In one embodiment, after obtaining the first observation azimuth, the reference time is converted into a standard time for astronomical calculations, such as Julian Day or Earth Time, which eliminates the uniform time scale caused by the non-uniformity of Earth's rotation and provides a consistent time reference for high-precision position calculations.

[0085] For distant celestial objects such as stars, their positions are obtained by progressively correcting from the standard epoch mean position to the observed instantaneous apparent position. Specifically, the mean position of stars at the J2000.0 standard epoch is read from a preset or online-updated star catalog database. and its own data, among which Indicates the mean right ascension, The mean declination is represented by the proper motion data, which refers to the long-term inherent motion of stars in the sky. First, based on the time span from epoch J2000.0 to the observation time, the mean position is corrected using the proper motion data to obtain the intermediate mean position at the observation time.

[0086] Due to the long-term slow wobble (precession) and periodic minute wobble (nutation) of the Earth's rotation axis in space, the celestial coordinate system itself changes over time. Therefore, precession and nutation corrections are needed to transform the stellar positions from the coordinate system at epoch J2000.0 to the true celestial coordinate system at the moment of observation, thereby eliminating the systematic bias introduced by the change in the Earth's rotation axis.

[0087] Due to Earth's revolution around the Sun, the direction of light rays seen by the observer shifts, a phenomenon known as stellar aberration. The system calculates and applies stellar aberration corrections. Furthermore, for nearby stars, annual parallax corrections due to the change in viewing angle caused by Earth's revolution around the Sun must also be considered. After these corrections, the final apparent equatorial coordinates of the star relative to the Earth's position at the time of observation are obtained. And convert it into a user-centric geocentric coordinate vector. .

[0088] For celestial bodies within the solar system, such as the Sun, Moon, planets, and their satellites, their motions are more complex. Therefore, the system utilizes high-precision numerical ephemeris tables or mature analytical theories, such as the DE series ephemeris published by the Jet Propulsion Laboratory. Based on unified Earth time, the precise three-dimensional position vectors of these celestial bodies in the geocentric inertial coordinate system are obtained directly through interpolation calculations or direct analytical solutions.

[0089] Ultimately, both the apparent positions of stars obtained through complex corrections and the planetary positions directly derived from ephemeris catalogs are uniformly represented as position vectors in a geocentric inertial coordinate system. .

[0090] Calculate the relationship between each celestial body and the first observation position. The angle between them will be with Celestial bodies with an angle less than half the camera's field of view are identified as being within or near the current field of view. The identifiers of these celestial bodies and their coordinate vectors in the geocentric inertial coordinate system are output, or further converted into azimuth / altitude angles with the observer as the origin.

[0091] Furthermore, based on the first observation azimuth and the field of view parameters of the currently active vehicle-mounted camera, a three-dimensional observation field of view cone is constructed in the celestial coordinate system. This cone, with the first observation azimuth vector as its central axis and the horizontal and vertical field of view angles of the camera as its subtended angles, defines the entire sky range that the current camera can capture.

[0092] Subsequently, based on the current precise UTC time and the vehicle's geographical location, preliminary coordinate transformation and field-of-view inclusion tests were performed on all known celestial objects. Specifically, the equatorial coordinates of stars stored in the star catalog database were converted to horizontal coordinates centered on the observer, and compared with the spatial extent of the aforementioned observation field-of-view cone. Only celestial objects whose azimuth and elevation angles both fall within this cone were identified as potentially visible objects.

[0093] For the selected stars, corrections are applied sequentially to compensate for their long-term motion, precession, and nutation to eliminate the effects of the Earth's long-term precession and periodic wobble, as well as to correct for stellar aberration and annual parallax to eliminate the ray direction shift caused by the Earth's revolution. This yields the star's apparent equatorial coordinates at the current observation time. For celestial bodies within the solar system, such as planets and the Moon, the system uses a high-precision numerical ephemeris to directly obtain their position vectors in the geocentric inertial coordinate system through interpolation algorithms, converting them to horizontal coordinates for comparison.

[0094] After the above filtering and calculation, a list of visible celestial bodies is generated. The list contains a unique identifier for each visible celestial body and its coordinate information in the celestial coordinate system. This information is used to determine the specific superposition position of the celestial body in the real-time environmental image, thereby ensuring the accuracy and scientific validity of the augmented reality image.

[0095] In step S203, the superposition position of visible celestial bodies in the real-time environmental image is determined using celestial coordinate information.

[0096] In one embodiment, determining the superposition position of visible celestial bodies in a real-time environmental image using celestial coordinate information includes: Based on real-time pose information and the calibration parameters of the vehicle's onboard camera, a mapping relationship is established between the visual coordinate system of the onboard camera and the celestial coordinate system. Based on the celestial coordinates and mapping relationships of visible celestial bodies, the superimposed positions of visible celestial bodies in real-time environmental images are identified.

[0097] Specifically, camera calibration parameters refer to the internal and external parameters of the vehicle-mounted camera imaging model, obtained through prior experimental calibration. Internal parameters include focal length, principal point coordinates, and lens distortion coefficients, which define the mathematical relationships between points in three-dimensional space and the image formed on the camera's own sensor plane. External parameters describe the fixed installation position and angle of the camera's visual coordinate system relative to the vehicle's coordinate system.

[0098] Real-time pose information is a complete set of states describing the vehicle's spatial position and three-dimensional orientation at a specific moment. In this embodiment, the real-time pose information includes the vehicle's geographical location, reference time, and the vehicle's three-dimensional attitude angles.

[0099] By combining the real-time pose information with the pre-stored external parameters of the camera, the orientation of the camera's optical center in the geocentric inertial coordinate system at the current moment is determined. Then, using the camera's internal parameters, a mapping relationship is constructed from the three-dimensional direction vector in the celestial coordinate system (or geocentric inertial coordinate system) to the two-dimensional pixel coordinates on the camera's image plane. This mapping relationship can be represented by a composite transformation matrix or equivalent function that includes rotation, translation, and perspective projection.

[0100] After the mapping relationship is established, for each visible celestial object to be displayed, its coordinate information in the celestial coordinate system has been obtained through astronomical calculations. Substituting the celestial object's direction vector into the aforementioned mapping relationship, firstly, the celestial object's direction vector is transformed to the camera's visual coordinate system, obtaining a three-dimensional pointing vector relative to the camera's optical center. Then, using the camera's perspective projection model, the three-dimensional vector is projected onto the two-dimensional image sensor plane, finally calculating the corresponding pixel coordinates. These pixel coordinates represent the superimposed position of the visible celestial object in the real-time environmental image that should be observed.

[0101] Specifically, the camera calibration parameters include an intrinsic parameter matrix. With distortion coefficient The intrinsic parameter matrix is ​​used to characterize the ideal pinhole projection geometry of the camera, and the core parameters include the focal length in pixels. , and the principal point coordinates of the image , The distortion coefficient quantifies the inherent optical distortions of the camera lens, such as radial distortion and tangential distortion, and is used to mathematically correct non-ideal images.

[0102] extrinsic parameter matrix This characterizes the fixed mounting posture and position of the camera on the vehicle, specifying the rotation and translation relationship between the camera's visual coordinate system and the vehicle's coordinate system. It is the fourth rotation matrix. It is a translation vector.

[0103] Next, for any pixel in the real-time environment image By using the intrinsic parameter matrix and distortion coefficients to perform anti-distortion processing, the coordinates of the ideal projection point in the normalized visual coordinate system of the camera are obtained. Next, the direction vector in the normalized visual coordinate system of the camera is transformed to the vehicle coordinate system:

[0104] In the formula, This represents a fixed vector (i.e., the user's standard line of sight). Represents the fourth rotation matrix; This represents the matrix corresponding to the ideal projection point coordinates in the normalized visual coordinate system of the camera. Since the parallax caused by translation is negligible for distant celestial targets, this embodiment ignores translation. The impact.

[0105] Furthermore, by utilizing the vehicle's attitude angle at the current moment... and geographical location By using the same coordinate rotation chain, the direction vector in the carrier coordinate system is... Convert to unit direction vector in geocentric inertial coordinate system ,vector That is, the pixel point The corresponding celestial coordinate direction.

[0106]

[0107] In the formula, Represents the unit direction vector in the geocentric inertial coordinate system; Represents the first rotation matrix; Indicates the second rotation matrix; Represents the third rotation matrix; This represents a fixed vector.

[0108] To meet real-time requirements, the system can pre-calculate and store a lookup table or online. The lookup table stores the celestial direction vector, calculated using the inverse mapping described above, for each pixel (or gridded sampled pixel) within the effective area of ​​the image. In actual operation, the system can instantly obtain the sky direction corresponding to any pixel by querying this table, or quickly perform forward calculations from celestial positions to pixel coordinates through interpolation, thereby achieving efficient and accurate augmented reality rendering.

[0109] Furthermore, given the known orientation of a celestial body in the universe (i.e., the geocentric inertial coordinate vector) The pixel coordinates of a given element in the camera view can be determined as follows: : Specifically, by using the inverse of the coordinate rotation matrix, the direction vector of the celestial body in the geocentric inertial coordinate system is... Transform to vehicle coordinate system:

[0110] In the formula, yes The inverse / transpose of the matrix; yes The inverse / transpose of the matrix; yes The inverse / transpose of the matrix; Represents the first rotation matrix; Indicates the second rotation matrix; Represents the third rotation matrix; Represents a fixed vector; This represents the geocentric coordinate vector centered on the observer.

[0111] Next, the direction vector in the carrier coordinate system Transform to the camera's visual coordinate system:

[0112] In the formula, This represents the three-dimensional direction vector in the visual coordinate system of the camera. Represents the fourth rotation matrix; This represents a fixed vector.

[0113] The three-dimensional direction vector in the visual coordinate system where the camera is located Normalize and project to On the plane, we obtain normalized coordinates. This step requires... That is, the celestial body is located in front of the camera:

[0114] Next, based on a pre-stored distortion model (such as the Brown-Conrady model), the normalized coordinates are... Apply distortion to obtain the distorted coordinates. .

[0115] Using camera intrinsic parameter matrix Transform the distorted normalized coordinates to the image pixel coordinate system:

[0116] Received This refers to the precise pixel position where the celestial body should appear in the current graphical environment. The AR rendering module draws virtual information such as icons and text at this position, thus achieving precise alignment with the real celestial body.

[0117] In step S204, the visual features of visible celestial bodies are rendered onto the overlay position to obtain an augmented reality image, which is then displayed through the vehicle's display components.

[0118] Specifically, visual features refer to virtual graphic elements superimposed on an image to represent celestial information, such as a set of rendered objects with specific graphic and spatial attributes. These include, but are not limited to: highlights or specific icons indicating the position of celestial bodies, lines or outlines connecting stars to form constellations, text labels indicating the names or attributes of celestial bodies, and dynamic curves used to indicate planetary trajectories or satellite transit paths. Each visual feature is associated with a target celestial body identifier obtained through astronomical calculations and is endowed with visual attributes determined based on its celestial type, brightness, cultural significance, or user settings, such as color, brightness, flicker frequency, size, and transparency.

[0119] The system calls a graphics rendering engine (such as OpenGL ES, Vulkan, or a specific automotive graphics API). It renders the precise pixel-level overlay coordinates corresponding to each visual feature, which have been determined through the aforementioned mapping calculations. Specifically, using the overlay coordinates as spatial anchor points, pixels are drawn at the corresponding positions in the frame buffer, depending on the type of visual feature. For example, for a star, a circular spot with a halo effect might be drawn at its overlay position; for a constellation, line segments are drawn sequentially between the overlay positions of the stars that make up the constellation.

[0120] Next, to ensure a natural blend between virtual elements and the real background, the brightness and transparency of the visualization features are dynamically adjusted based on the local brightness of the real-time environmental image, avoiding overexposure near bright celestial bodies (such as the moon) or appearing too abrupt in the dark night sky. Simultaneously, anti-aliasing and feathering may be applied to the edges of the graphics to ensure a smooth transition with the real-world image. For moving targets (such as satellites), their visualization features (such as the satellite's trajectory) are animated frame-by-frame based on continuous updates to their coordinate information, creating a smooth dynamic effect.

[0121] After rendering, an augmented reality (AR) image is generated. An AR image is a composite image containing pixel data of the original real-time environment and overlaid pixel data of visual features. Subsequently, the system transmits this composite image as a video signal or buffers it to the vehicle's display components, such as the vehicle's infotainment display, window / sunroof projection, physical screen / blind, multi-screen setups, and external display devices, via the vehicle's video display interface. The display component's drive circuitry receives the video signal from the composite image and presents it as a visible optical image at the screen's inherent refresh rate.

[0122] Ultimately, users will see a dynamic visual experience that seamlessly blends reality and virtuality. For example, a real starry sky outside the vehicle is overlaid with clearly marked constellation patterns, planetary positions, dynamic event trajectories, and other rich virtual information, thus transforming the vehicle's cabin into an immersive astronomical observation window with real-time, accurate, and interactive navigation capabilities.

[0123] In one embodiment, after displaying the augmented reality image via the vehicle's display components, the method further includes: In response to the user selecting a first target celestial body in the augmented reality scene, determine whether the first target celestial body is within the field of view of the vehicle's onboard camera; If not, then the offset of the first target celestial body relative to the user's real-time observation direction is determined based on the celestial coordinate information and real-time pose information of the first target celestial body. The real-time observation direction is determined based on the orientation of the user's seat in the vehicle. Based on the offset azimuth, a multimodal guidance strategy is generated and executed to guide the user to adjust the real-time observation direction until the first target celestial object enters the field of view.

[0124] Specifically, when a user selects a specific first target celestial body, such as a planet or a constellation, in the displayed augmented reality scene through touch, voice, or gaze focus, and the system determines in real time that the first target celestial body is not currently within the field of view of the activated vehicle camera, that is, within the sky area that the camera can capture while it is not currently facing, the system calculates the azimuth deviation of the target relative to the user's real-time observation direction, which is not the fixed orientation of the vehicle.

[0125] Specifically, the system first determines the user's real-time observation direction, which depends on the orientation information of the user's vehicle seat. In a preferred embodiment, the system estimates the natural direction of the user's head or line of sight in the current seat posture by reading data from the seat posture sensor and combining it with a preset ergonomic model. For example, when the seat back is reclined significantly, the user's line of sight will shift from the sunroof area directly above the vehicle towards the rear windshield. Based on this estimated real-time observation direction, the system determines the offset azimuth of the celestial body relative to the user's current line of sight by comparing it with the celestial coordinates of the first target celestial body. The offset azimuth typically includes a horizontal angle and a vertical angle.

[0126] In one embodiment, the multimodal guidance strategy includes at least one of the following: Directional indicator graphics are overlaid on the edge areas of the augmented reality image, and the direction of the directional indicator graphics is updated in real time as the vehicle's three-dimensional attitude angle changes; and / or, voice guidance commands containing the offset orientation are broadcast through the vehicle's audio components, and the vehicle's seat is adjusted to the offset orientation based on the voice guidance commands.

[0127] Specifically, after obtaining the offset orientation, a directional indicator graphic, such as an arrow or curved guide line, is dynamically overlaid at the edge of the augmented reality image displayed on the display component. The direction of the directional indicator graphic corresponds to the calculated offset orientation in real time and is dynamically updated as the system continuously estimates changes in the user's real-time observation direction through sensors. And / or, a voice command containing a specific orientation description is broadcast through the vehicle's audio system, such as "Please look up to your left rear." The audio system is a multi-channel surround sound system installed in the vehicle, typically containing speaker units such as tweeters, midrange speakers, and woofers distributed in multiple locations, including the front doors, rear doors, dashboard, rear shelf, and A / B pillars.

[0128] When vehicle configuration allows, the system can coordinate vehicle comfort functions. For example, it can slightly adjust the angle of the seat back or headrest to assist the user in turning their body in the general direction of the target; or it can indirectly guide the user's attention by controlling changes in the transparency of the window glass.

[0129] This embodiment is user-centric. Users can actively adjust their head and line of sight based on the multi-sensory prompts provided by the system. The system continuously senses this adjustment through sensors and updates the guidance information in real time, such as moving the arrow position and updating the voice prompts, until the system detects or the user confirms that the first target celestial body has entered the camera's field of view and has been successfully captured in the augmented reality image.

[0130] After displaying an augmented reality image of the starry sky through the vehicle's display components, this embodiment estimates the user's line of sight by combining the seat orientation and calculates the precise azimuth deviation of celestial bodies relative to the line of sight. Then, by dynamically updating the directional arrows, synchronizing voice navigation, and coordinating seat posture adjustments, a multi-sensory guidance loop is formed, enabling users to intuitively and conveniently adjust their personal observation posture, significantly improving the interactivity and overall immersive experience of astronomical observation.

[0131] In one embodiment, the real-time pose information further includes the vehicle's geographical location and reference time, and the method further includes: Acquire ephemeris data, which is used to characterize astronomical events involving at least one celestial body in the future and the corresponding prediction data for these events; Filter and obtain at least one target astronomical event that is associated with a reference time and can be observed at a geographical location from the ephemeris data, and determine the prediction data corresponding to the target astronomical event; The predicted data corresponding to the target astronomical event is integrated and displayed in the augmented reality screen.

[0132] Specifically, the system acquires and maintains a set of ephemeris data via vehicle-mounted network connection or pre-stored data. Ephemeris data refers to a set of dynamic information describing celestial bodies or specific astronomical phenomena within the solar system over a future period. It is used to characterize at least one specific celestial body's predictable astronomical events within a future time window, as well as the prediction data for these events. The prediction data may include, but is not limited to, celestial body position sequences, occurrence times, critical moments, end times, trajectories, and visibility status. Celestial body position sequences refer to the precise celestial coordinates of the event-related celestial bodies at different times; occurrence times, critical moments, and end times refer to the start time, optimal observation time, and end time of an astronomical event, such as the rise, zenith, and setting times of a satellite transit; trajectories refer to the movement path of a celestial body in the sky described by a sequence of coordinate points or mathematical formulas; visibility status refers to whether the astronomical event is visible at the current geographical location based on criteria such as celestial body elevation angle and solar position, and the estimated brightness or observation condition rating when visible.

[0133] The system processes the acquired ephemeris data using the vehicle's real-time geographical location (latitude and longitude) and reference time (UTC) as filtering criteria. From the massive ephemeris data, it filters out astronomical events whose predicted occurrence time is correlated with the current and future reference time, and which are theoretically observable from the vehicle's geographical location, such as astronomical events that are about to occur or are currently occurring. For example, events that are not visible in the Southern Hemisphere or events that have already ended are removed.

[0134] For the initially selected events, the current geographical location, precise time, and trajectory obtained from ephemeris data are input into the preset prediction model to determine the detailed trajectory, visible time window, and elevation angle changes of the astronomical event in the local horizontal coordinate system. Finally, one or more target astronomical events and their prediction data tailored to the current vehicle position are obtained, such as the locally visible transit time, azimuth angle, elevation angle trajectory, etc.

[0135] The predicted data for the identified target astronomical events are integrated and displayed in the augmented reality screen. The integration method can include at least one of the following: timeline or list overlay, trajectory pre-plotting, and active prompts and interactive entry points.

[0136] The timeline or list overlay refers to displaying target events that will occur in the next few hours in a graphical timeline or list format at the edge of the screen, such as "International Space Station transit: 21:40-21:35, from southwest to northeast". Track pre-drawing refers to the system's ability to pre-draw the predicted sky trajectory of events with obvious movement trajectories, such as satellite transits, using dashed lines or gradient color bands within the AR screen, and marking time points on the trajectory. Active prompts and interactive entry points refer to the system's ability to proactively trigger visual prompts (such as flashing icons) or voice reminders as the start time of a target event approaches, allowing the user to click on the visual prompt. In response to the user's click, the system can directly switch the camera's direction or activate the aforementioned guidance functions to guide the user in observing the astronomical event.

[0137] This embodiment upgrades the star projection system into an intelligent platform with proactive service capabilities by introducing a dynamic astronomical event prediction function based on ephemeris data. The system filters and accurately calculates dynamic events visible locally, such as satellite transits and meteor showers, from ephemeris data based on the vehicle's real-time geographical location and time. It then intuitively integrates the predicted event trajectories and times into the AR screen, expanding the spatiotemporal dimension and content depth of the system's services.

[0138] In one embodiment, after integrating and displaying the predicted data corresponding to the target astronomical event on the augmented reality screen, the method further includes: In response to a user's command to pay attention to any target astronomical event, the vehicle camera can switch to a second observation position or generate guidance information to remind the user to observe the target astronomical event.

[0139] Specifically, when a user issues a follow command for any target astronomical event displayed in the augmented reality screen, such as a predicted satellite transit trajectory, through clicking, voice commands, or gaze confirmation, the system will immediately respond to the command and, based on the event status of the target astronomical event and the system configuration, initiate corresponding observation assistance in at least one of the following modes: For target astronomical events that have not yet begun, the observation viewpoint can be actively switched. Specifically, if the target astronomical event has not yet entered the current camera's field of view, but is judged to be about to become visible based on its prediction data, the system can actively switch the vehicle-mounted camera's second observation position. The second observation position is an optimized spatial orientation of the camera, calculated in advance or in real time based on the event characteristics. Specifically, the system calculates the azimuth and elevation angles of the starting point or optimal observation point relative to the vehicle at the current moment, based on the starting point or optimal observation point of the event trajectory, such as the rising point or zenith of a satellite transit, and then generates control commands. If the vehicle-mounted camera has a motorized gimbal or the vehicle is equipped with multiple cameras with different fixed viewing angles, the system controls the gimbal to rotate or automatically switches to the most suitable camera, so that the starting area or critical path of the target astronomical event immediately enters the center of the image, providing the user with the optimal initial field of view for capturing the event.

[0140] For situations where a target astronomical event has already begun, the camera cannot be rotated, or the user needs to adjust it themselves, personalized observation guidance is generated using a strategy of generating and executing guidance information. Guidance information refers to information used to remind and assist the user in observing the specific target astronomical event. Specifically, the system compares the event's predicted data with the user's current real-time observation direction. When it determines that the target astronomical event is not currently in the user's line of sight or is about to enter the field of view, the system generates guidance information. Guidance information can be represented as dynamic visual guidance markers superimposed on the augmented reality image, pointing to the event's current or upcoming location, such as arrows or highlighted areas. Simultaneously or alternately, voice prompts can be broadcast via audio components, containing not only directional guidance, such as "Please look towards the low northwest," but also potentially dynamic descriptions of the event's status, such as "A satellite is about to enter the field of view, please be aware."

[0141] As the target astronomical event progresses, the system continuously updates guidance information based on its prediction data, dynamically adjusts the position of visual markers, or updates voice prompts until the event ends or the user confirms completion of the observation.

[0142] Through the above response mechanism, this embodiment achieves a seamless transition from passive event forecasting to proactive observation assistance, ensuring that users can obtain clear and timely operational guidance even in the face of dynamically changing events, greatly improving the success rate of astronomical event observation and the integrity of the user experience.

[0143] In one embodiment, the vehicle seat is equipped with an attitude sensor, and the method further includes: The posture data of the seat is acquired by the posture sensor, and the user's real-time sitting posture is determined based on the posture data; The user's primary gaze direction is determined based on the mapping relationship between real-time sitting posture and gaze direction; The second target celestial body corresponding to the main line of sight is highlighted in the augmented reality image, and the information of the second celestial body corresponding to the second target celestial body is broadcast through the audio components on the vehicle.

[0144] Specifically, the system continuously acquires the seat's posture data through posture sensors installed within the seat. Posture sensors typically refer to sensing elements capable of measuring angles, positions, or pressure distributions, such as angle sensors, linear displacement sensors, or pressure sensor arrays. The posture data includes information such as the seat back tilt angle, seat cushion pitch angle, headrest position, and possible lateral support angles.

[0145] Based on this posture data, combined with pre-stored vehicle seat geometry models and standard human body dimensions, the system calculates the user's real-time seating posture using a preset kinematic model, estimating the approximate position and orientation of the user's head and torso within the vehicle space. Subsequently, based on preset statistical patterns of typical gaze directions under different sitting postures, the system determines a primary gaze direction vector originating from the user's eye position, representing their natural gaze direction. This primary gaze direction vector may point more towards the rear of the sunroof as the seat reclines, or towards the side window as the seat rotates.

[0146] After estimating the primary line-of-sight direction, it compares it with the coordinates of all visible celestial objects obtained through astronomical calculations. The system then selects one or more celestial objects as secondary target objects within the sky region near the primary line-of-sight direction—the area where the user is most likely to be viewing or has the most convenient view. Selection criteria can be based on the object's salience, such as the brightest planet; its cultural significance, such as a well-known constellation of the current season; or real-time events related to the object, such as a satellite passing in that direction.

[0147] In the displayed augmented reality footage, virtual markers corresponding to the second target celestial body, such as stars and constellation shapes, are highlighted. Specifically, this can be achieved by increasing the brightness of the shape, adding pulsed halos, enlarging the icon, changing the color, or drawing eye-catching circles around it, thereby intuitively attracting the user's attention to the area where their gaze naturally points in the complex starry sky.

[0148] Simultaneously, the system automatically broadcasts information about a second celestial body related to the target object via audio components. This information is structured popular science content, which may include the celestial body's name, basic type, current observational characteristics, related myths or brief scientific knowledge, etc. The triggering and depth of the broadcast can be adaptively adjusted based on the length of time the user gazes at the area.

[0149] This embodiment uses an attitude sensor to intelligently sense the user's posture and estimate their primary line of sight, thereby enabling personalized and proactive delivery of stargazing content. The system can automatically highlight prominent celestial objects in the area where the user's line of sight naturally falls, and simultaneously broadcast relevant scientific information, significantly reducing the user's exploration burden in the complex night sky and greatly enhancing the guidance and immersion of observation.

[0150] In another embodiment, the system can continuously monitor and integrate multiple real-time data streams during vehicle operation to determine whether safe and suitable conditions exist for in-vehicle stargazing. The criteria data primarily include light pollution environmental data, meteorological and environmental data, and road and vehicle status data.

[0151] Specifically, for light pollution environmental data, the system accesses the light pollution map database stored on the cloud platform connected to the system through the vehicle network, or uses the vehicle's optical sensors to analyze the ambient brightness, obtain the light pollution level data of the vehicle's real-time geographical location, and determine whether the current area's light pollution level is lower than the preset suitable threshold for stargazing. For example, the preset suitable threshold for stargazing can be level 2 or below in the Porter Dark Sky Classification.

[0152] For meteorological and environmental data, the system obtains local real-time weather information through the vehicle network, or integrates data monitored by vehicle sensors such as rain sensors, external temperature and humidity sensors, and infrared thermal imagers to comprehensively determine whether the current weather is sunny, cloud cover is sparse, and atmospheric transparency is high.

[0153] Regarding road and vehicle status data, the system reads real-time vehicle driving data, including vehicle speed, steering angle, acceleration, and road information from advanced driver assistance systems, such as lane curvature and road type. Simultaneously, it determines whether the vehicle is currently traveling in a specific area with straight and safe roads, such as high-altitude highways or desert roads, and whether the vehicle is in a stable cruising state without sudden acceleration, deceleration, or frequent steering, ensuring that the initiation of observation suggestions does not interfere with driving safety.

[0154] When multiple criteria, including light pollution level, weather conditions, and road safety, simultaneously meet preset conditions, the system determines that mobile stargazing mode can be activated. Subsequently, the system can proactively recommend activating stargazing mode to the user through the in-vehicle human-machine interface, such as a gentle prompt on the central control screen, illuminated instrument panel icons, or the voice system. Simultaneously with the recommendation or after the user confirms activation, the system will recommend the best observation target based on the vehicle's current location, orientation, time, and astronomical data. For example, the system might prompt: "Currently in a dark area; bright Jupiter is visible in the sky to your right. Click to view details." This embodiment seamlessly extends astronomical observation services from a user's conscious, planned, static activity to a dynamic, environment-aware, and triggered accompanying experience, greatly expanding the service's usage scenarios and frequency.

[0155] This embodiment also provides a vehicle-mounted star projection device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0156] This embodiment provides a starry sky projection device for a vehicle, such as... Figure 4 As shown, it includes: The data acquisition module 401 is used to acquire the real-time pose information of the vehicle and collect real-time environmental images around the vehicle.

[0157] The orientation determination module 402 is used to determine the first observation orientation based on the vehicle as a reference based on real-time pose information, and to determine the astronomical coordinate information of visible celestial bodies in the celestial coordinate system based on the first observation orientation.

[0158] The image overlay module 403 is used to determine the overlay position of visible celestial bodies in a real-time environmental image using astronomical coordinate information.

[0159] The image display module 404 is used to render the visual features of visible celestial bodies onto the overlay position to obtain an augmented reality image, and to display the augmented reality image through the vehicle's display components.

[0160] In one embodiment, the real-time pose information includes the vehicle's three-dimensional attitude angles. The orientation determination module 402 is specifically used to analyze multiple consecutive frames of real-time environmental images to extract and match star feature points from the real-time environmental images, perform visual synchronous positioning processing on the star feature points to obtain vehicle attitude auxiliary data, fuse the vehicle attitude auxiliary data with the three-dimensional attitude angles to generate corrected attitude information, and correct the first observation orientation based on the corrected attitude information.

[0161] In one embodiment, the image overlay module 403 is specifically used to establish a mapping relationship between the visual coordinate system of the vehicle camera and the celestial coordinate system based on real-time pose information and the calibration parameters of the vehicle's onboard camera; and to identify the overlay position of the visible celestial bodies in the real-time environmental image according to the astronomical coordinate information of the visible celestial bodies and the mapping relationship.

[0162] In one embodiment, after displaying the augmented reality image via the vehicle's display components, the device further includes: The offset adjustment module is used to respond to the user selecting a first target celestial body in the augmented reality scene, and to determine whether the first target celestial body is within the field of view of the vehicle's onboard camera; if not, it determines the offset azimuth of the first target celestial body relative to the user's real-time observation direction based on the astronomical coordinate information and real-time pose information of the first target celestial body, wherein the real-time observation direction is determined based on the user's seat orientation in the vehicle; and generates and executes a multimodal guidance strategy based on the offset azimuth to guide the user to adjust the real-time observation direction until the first target celestial body enters the field of view.

[0163] In one embodiment, the multimodal guidance strategy includes at least one of the following: Directional indicator graphics are overlaid on the edge areas of the augmented reality image, and the direction of the directional indicator graphics is updated in real time as the vehicle's three-dimensional attitude angle changes; and / or, The vehicle's audio system broadcasts voice guidance commands that include the direction of the deviation, and the vehicle's seats are adjusted to the opposite direction based on these voice guidance commands.

[0164] In one embodiment, the real-time pose information further includes the vehicle's geographical location and reference time, and the device further includes: The prediction module is used to acquire ephemeris data, which is used to characterize astronomical events of at least one celestial body in the future and the corresponding prediction data of the astronomical events; to filter and acquire at least one target astronomical event that is associated with a reference time and can be observed at a geographical location from the ephemeris data, and to determine the prediction data corresponding to the target astronomical event; and to integrate and display the prediction data corresponding to the target astronomical event on the augmented reality screen.

[0165] In one embodiment, after integrating and displaying the predicted data corresponding to the target astronomical event on the augmented reality screen, the device further includes: The guidance module is used to respond to the user's command to pay attention to any target astronomical event, switch the second observation position of the vehicle camera or generate guidance information to remind the user to observe the target astronomical event.

[0166] In one embodiment, the vehicle seat is equipped with an attitude sensor, and the device further includes: The augmented display module is used to acquire the posture data of the seat through the posture sensor, determine the user's real-time sitting posture based on the posture data, determine the user's main line of sight direction according to the mapping relationship between the real-time sitting posture and the line of sight direction, highlight the second target celestial body corresponding to the main line of sight direction in the augmented reality image, and broadcast the information of the second celestial body corresponding to the second target celestial body through the vehicle's audio components.

[0167] In this embodiment, the vehicle's star projection device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0168] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0169] This invention also provides a computer device having the above-described features. Figure 4 The vehicle shown has a star projection device.

[0170] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

[0171] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing a computer device according to embodiments of the present invention. The computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the computer device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0172] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows the computer device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Computer equipment with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0173] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the vehicle star projection method of the embodiments of the present invention.

[0174] Figure 5 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0175] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the vehicle star projection method shown in the above embodiments is implemented.

[0176] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0177] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

[0178] This invention provides a vehicle, which includes a controller, a memory, and a processor. The memory and the processor are communicatively connected. The memory stores computer instructions, and the processor executes the computer instructions to perform the star projection method of the vehicle according to the first aspect or any corresponding embodiment described above.

Claims

1. A method for projecting stars onto a vehicle, characterized in that, The method includes: Acquire the vehicle's real-time pose information and capture real-time environmental images around the vehicle; Based on the real-time pose information, a first observation orientation with the vehicle as the reference is determined, and the celestial coordinate information of the visible celestial body in the celestial coordinate system is determined according to the first observation orientation. The superposition position of the visible celestial body in the real-time environmental image is determined using the celestial coordinate information; The visual features of the visible celestial bodies are rendered onto the overlay position to obtain an augmented reality image, which is then displayed through the vehicle's display components.

2. The method according to claim 1, characterized in that, The real-time pose information includes the three-dimensional attitude angles of the vehicle, and the determination of the first observation orientation based on the vehicle using the real-time pose information includes: Analyze multiple consecutive frames of the real-time environment image to extract and match star feature points from the real-time environment image, perform visual synchronous localization processing on the star feature points, and obtain vehicle attitude assistance data; The vehicle attitude assistance data and the three-dimensional attitude angles are fused together to generate corrected attitude information; The first observation azimuth is corrected based on the corrected attitude information.

3. The method according to claim 1, characterized in that, Determining the superposition position of the visible celestial body in the real-time environmental image using the celestial coordinate information includes: Based on the real-time pose information and the calibration parameters of the vehicle's onboard camera, a mapping relationship is established between the visual coordinate system of the onboard camera and the celestial coordinate system. Based on the celestial coordinate information of the visible celestial body and the mapping relationship, the superimposed position of the visible celestial body in the real-time environmental image is identified.

4. The method according to claim 1, characterized in that, After displaying the augmented reality image through the vehicle's display component, the method further includes: In response to the user selecting a first target celestial body in the augmented reality scene, it is determined whether the first target celestial body is within the field of view of the vehicle's onboard camera; If not, then based on the celestial coordinate information of the first target celestial body and the real-time pose information, determine the offset of the first target celestial body relative to the user's real-time observation direction, wherein the real-time observation direction is determined based on the orientation of the user's seat in the vehicle; Based on the offset azimuth, a multimodal guidance strategy is generated and executed to guide the user to adjust the real-time observation direction until the first target celestial body enters the field of view.

5. The method according to claim 4, characterized in that, The multimodal guidance strategy includes at least one of the following: A directional indicator graphic is overlaid on the edge area of ​​the augmented reality image, and the direction of the directional indicator graphic is updated in real time according to the three-dimensional attitude angle of the vehicle; and / or, The vehicle's audio system broadcasts a voice guidance command containing the offset direction, and the vehicle's seats are adjusted to face the offset direction based on the voice guidance command.

6. The method according to claim 1, characterized in that, The real-time pose information also includes the vehicle's geographical location and reference time, and the method further includes: Acquire ephemeris data, which is used to characterize astronomical events of at least one celestial body in the future and the corresponding prediction data of the astronomical events; Filter and obtain at least one target astronomical event that is associated with the reference time and can be observed at the geographical location from the ephemeris data, and determine the prediction data corresponding to the target astronomical event; The predicted data corresponding to the target astronomical event is integrated and displayed on the augmented reality screen.

7. The method according to claim 6, characterized in that, After integrating and displaying the predicted data corresponding to the target astronomical event in the augmented reality screen, the method further includes: In response to a user's command to pay attention to any of the target astronomical events, the vehicle-mounted camera's second observation position is switched or guidance information is generated to remind the user to observe the target astronomical event.

8. The method according to claim 1, characterized in that, The vehicle's seat is equipped with a posture sensor, and the method further includes: The posture data of the seat is acquired through the posture sensor, and the user's real-time sitting posture is determined based on the posture data; The user's primary gaze direction is determined based on the mapping relationship between the real-time riding posture and the gaze direction. The second target celestial body corresponding to the main line of sight is highlighted in the augmented reality image, and the information of the second celestial body corresponding to the second target celestial body is broadcast through the audio component mounted on the vehicle.

9. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1 to 7.

Images