Starlight optical observation device, observation system, and image processing method

Abstract

The application provides a starry sky optical observation device, system and image processing method. The optical observation device can perform at least one of reflection, refraction and increase of virtual image distance on an electronic starry sky image emitted by a terminal device screen, and fold the obtained image to a user through a light path of a light splitting element. The light splitting element can simultaneously transmit an external real starry sky image to the user. In this way, the electronic starry sky image can accurately correspond to the position of a real celestial body directly observed by the naked eye of the user, and the propagation direction of the electronic starry sky image is approximately parallel to the external real starry sky image. The user can clearly observe both without adjusting the lens, thereby providing an unobstructed and immersive virtual-real fusion observation experience.

Description

Technical Field

[0001] This application relates to the field of augmented reality (AR), and more specifically, to a starry sky optical observation device, observation system, and image processing method. Background Technology

[0002] With the development of technology, people's interest in astronomical observation is increasing. Most existing AR stargazing products rely on mobile device cameras to capture images of the night sky, overlaying virtual star information and displaying it directly on the mobile device screen. However, this approach not only leads to obstructed observation and fails to achieve an immersive experience that blends the virtual and real worlds, but also results in distorted perspective, making it difficult to match celestial objects on the screen with real objects seen by the naked eye. Summary of the Invention

[0003] This application provides a starry sky optical observation device, system, and image processing method, which can superimpose distant virtual images with real scene light in starry sky observation scenarios, providing an unobstructed and immersive observation experience, allowing observers to clearly observe virtual markers without repeatedly adjusting their lenses.

[0004] In a first aspect, a starry sky optical observation device is provided, comprising: a mounting section for supporting a terminal device, such that the screen of the terminal device faces the device and emits a first electronic starry sky image; an optical path assembly for performing at least one of the following operations on the first electronic starry sky image to obtain a second electronic starry sky image: reflection, refraction, or increasing the virtual image distance; and a first beam-splitting element, comprising a first refractive surface and a first transmission surface; wherein the first refractive surface faces the optical path assembly, and the first refractive surface is used to reflect and / or refract the second electronic starry sky image to the user; and the first transmission surface is used to transmit a real starry sky image to the user.

[0005] In one example, the first electronic starry sky image can be obtained by processing a real starry sky image captured by a camera using a terminal device. For instance, the terminal device generates the first electronic starry sky image by processing information from the real starry sky image and combining it with at least one of the following: the attitude of the terminal device, the device's position, and the current observation time.

[0006] Regarding the installation unit carrying the terminal equipment, one possible implementation is that the installation unit can detachably or non-detachably fix the terminal device.

[0007] Based on the above technical solution, it is possible to superimpose distant virtual images with real scene light in stargazing scenarios, providing an unobstructed and immersive observation experience. Observers can clearly observe virtual markers without repeatedly adjusting their lenses. The stargazing optical observation device processes the electronic stargazing image emitted from the terminal device screen through at least one of the following methods: reflection, refraction, or increasing the virtual image distance. The resulting image is then refracted to the user via a beam splitter, which simultaneously transmits an external real stargazing image to the user. In this way, the electronic stargazing image accurately corresponds to the position of real celestial bodies directly observed by the user's naked eye, and the outward propagation direction of the electronic stargazing image is approximately parallel to the external real stargazing image. Users can clearly observe both without adjusting their lenses, providing an unobstructed and immersive virtual-real fusion observation experience.

[0008] In conjunction with the first aspect, in certain implementations of the first aspect, the first beam-splitting element satisfies at least one of the following: a reflectivity of less than or equal to 10% to reduce reflected light from the image source; or a transmittance of greater than or equal to 90% to enhance starlight transmittance; or a thickness of less than or equal to 1 mm to reduce ghosting. In one possible implementation, the first beam-splitting element can be a beam splitter, a beam-splitting prism, a holographic optical element, a polarizing beam splitter, etc., or a combination of two or more of these components.

[0009] Based on the above technical solutions, the starry sky optical observation device is suitable for nighttime observation or observation in low light conditions. A reflectivity of less than or equal to 10% can reduce reflected light from the image source, a transmittance of greater than or equal to 90% can enhance starlight transmission, and a thickness of less than or equal to 1 mm can reduce ghosting.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the optical path assembly includes a first optical path reversing element and a first virtual image retracting element; at least one side of the first optical path reversing element faces the screen, and the first optical path reversing element is used to reflect and / or refract a first electronic star image to obtain a third electronic star image; at least one side of the first virtual image retracting element faces the first optical path reversing element, and the first virtual image retracting element is used to increase the virtual image distance of the third electronic star image to obtain a second electronic star image.

[0011] Based on the above technical solution, the first virtual image teleportation element can effectively project the virtual image onto a distant point, matching its optical distance to the real starry sky. Therefore, users can clearly observe both the "electronic starry sky image" and the "real starry sky image" at the same focal length without frequently adjusting their eye lens. This design not only makes the fusion of virtual and real scenes more natural but also eliminates visual fatigue caused by constantly switching viewing distances, significantly improving the comfort of long-term observation.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first virtual image retracting element is a Fresnel lens or a lens group including at least one negative lens, used to correct aberrations. For example, the first virtual image retracting element is a cemented lens group. Compared to a lens group, a Fresnel lens is lighter, making it more convenient for users to carry and move, reducing the user's operational burden, and improving the ease of use in practice; compared to a Fresnel lens, a lens group can effectively reduce aberrations, such as chromatic aberration, and may be suitable for application scenarios requiring higher image quality.

[0013] One possible implementation is that the first optical path deflection element can be a mirror, prism, etc.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the first optical path deflection element is a front surface reflector. Since the reflective coating of the front surface reflector is directly disposed on the incident surface of the lens, this means that incident light is directly reflected upon contact with the surface of the front surface reflector without penetrating the glass substrate. Therefore, it effectively avoids the "parasitic optical path" (i.e., the "second reflecting surface" effect) caused by the superposition of "front surface refraction" and "rear surface reflection" when light penetrates the glass medium, thereby eliminating ghosting and beam distortion caused by the "second reflecting surface" effect and ensuring the fidelity and image clarity of the electronic starscape image during propagation.

[0015] In conjunction with the first aspect, in certain implementations of the first aspect, the optical path assembly includes a second beam splitter and a second optical path reversing element, the second optical path reversing element being disposed on the side of the second beam splitter away from the screen; the second beam splitter includes a second transmission surface and a second reversing surface; wherein, the second transmission surface faces the screen and is used to transmit a first electronic star image to obtain a fourth electronic star image; at least one side of the second optical path reversing element faces the second beam splitter, and the second optical path reversing element is used to reflect and / or refract the fourth electronic star image to obtain a fifth electronic star image; the second reversing surface faces the second optical path reversing element, and the second reversing surface is used to reflect and / or refract the fifth electronic star image to obtain a sixth electronic star image or the second electronic star image.

[0016] After the second folding surface reflects and / or refracts the fifth electron star image, a sixth electron star image or a second electron star image can be obtained.

[0017] In one possible implementation, the second refractive surface reflects and / or refracts the fifth electronic star image to obtain a second electronic star image. This second electronic star image is then propagated to the first beam splitter. Subsequently, the first refractive surface of the first beam splitter reflects and / or refracts the second electronic star image to the user. In another possible implementation, the second refractive surface reflects and / or refracts the fifth electronic star image to obtain a sixth electronic star image. In this case, the optical path assembly also includes a second virtual image extension element, which will be described in detail in the next paragraph.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the optical path component further includes a second virtual image retraction element, and at least one side of the second virtual image retraction element faces the second folding surface; the second folding surface is used to reflect and / or refract the fifth electronic star image to obtain a sixth electronic star image; the second virtual image retraction element is used to increase the virtual image distance of the sixth electronic star image to obtain a second electronic star image.

[0019] One possible implementation is that the second virtual image retracting element can be a Fresnel lens or a lens group containing at least one negative lens, used to correct aberrations. For example, the second virtual image retracting element is a cemented lens group. Compared to a lens group, a Fresnel lens is lighter, making it more convenient for users to carry and move, reducing the user's operational burden, and improving the ease of use in practice; compared to a Fresnel lens, a lens group can effectively reduce aberrations, such as chromatic aberration, and may be suitable for applications requiring higher image quality.

[0020] Based on the above technical solution, the second virtual image teleportation element can effectively project the virtual image onto a distant point, matching its optical distance to the real starry sky. Therefore, users can clearly observe both the "electronic starry sky image" and the "real starry sky image" at the same focal length without frequently adjusting their eye lens. This design not only makes the fusion of virtual and real scenes more natural but also eliminates visual fatigue caused by constantly switching viewing distances, significantly improving the comfort of long-term observation.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the second optical path reversing element includes a first concave surface facing the second beam splitter. Based on the above technical solution, the second optical path reversing element is provided with a first concave surface, and the concave surface structure itself possesses positive refractive power, which can increase the virtual image distance of the electronic star image. Based on this premise, other virtual image retraction elements only need to be set with lower diopter to achieve the same virtual image distance increase function. In other words, the first concave surface shares part of the task of increasing the virtual image distance, thereby reducing the optical power requirement for other virtual image retraction elements.

[0022] Regarding the second beam splitter, as an example, the second beam splitter can be a beam splitter mirror, a beam splitter prism, a holographic optical element, a polarizing beam splitter, etc., or it can be a combination of two or more of the following: beam splitter mirror, beam splitter prism, holographic optical element, polarizing beam splitter.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the second beam splitter and the first beam splitter constitute a first combination; the first combination satisfies at least one of the following: the reflectivity is less than or equal to 10% to reduce reflected light from the image source; or, the transmittance is greater than or equal to 90% to enhance starlight transmittance; or, the thickness of the second beam splitter in the first combination is less than or equal to 1 mm to reduce ghosting.

[0024] Based on the above technical solutions, the starry sky optical observation device is suitable for nighttime observation or observation in low light conditions. A reflectivity of less than or equal to 10% can reduce reflected light from the image source, a transmittance of greater than or equal to 90% can enhance starlight transmission, and a thickness of less than or equal to 1 mm can reduce ghosting.

[0025] One possible implementation is that the second optical path deflection element can be a mirror, prism, etc.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the second optical path deflection element is a front surface reflector. Since the reflective coating of the front surface reflector is directly disposed on the incident surface of the lens, this means that incident light is directly reflected upon contact with the surface of the front surface reflector without penetrating the glass substrate. Therefore, it effectively avoids the "parasitic optical path" (i.e., the "second reflecting surface" effect) caused by the superposition of "front surface refraction" and "rear surface reflection" when light penetrates the glass medium, thereby eliminating ghosting and beam distortion caused by the "second reflecting surface" effect and ensuring the fidelity and image clarity of the electronic starscape image during propagation.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the optical path assembly includes a third beam-splitting element and a third optical path reversing element; at least one side of the third optical path reversing element faces the screen, and the third optical path reversing element is used to reflect and / or refract a first electronic star image to obtain a seventh electronic star image; the third beam-splitting element includes a third reversing surface, the third reversing surface faces the third optical path reversing element, and the third reversing surface is used to reflect and / or refract the seventh electronic star image to obtain a second electronic star image; the first reversing surface faces the third reversing surface, and the first reversing surface is used to reflect and / or refract the second electronic star image to the user to obtain an eighth electronic star image; the third reversing surface is also used to transmit the eighth electronic star image and an external real star image to the user.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, the first folding surface is concave. Thus, the concave structure of the first folding surface itself possesses positive refractive power, which can increase the virtual image distance of the electronic starry sky image. Furthermore, it can effectively project the virtual image onto a distant point, matching its optical distance to the real starry sky. Therefore, users can clearly observe both the "electronic starry sky image" and the "real starry sky image" at the same focal length without frequently adjusting their eye lens. This design not only makes the fusion of virtual and real scenes more natural but also eliminates visual fatigue caused by switching viewing distances back and forth, significantly improving the comfort of long-term observation.

[0029] One possible implementation is that the third beam-splitting element can be a beam splitter, a beam splitting prism, a holographic optical element, a polarizing beam splitter, or a combination of two or more of these elements.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the third beam splitter and the first beam splitter constitute a second combination; the second combination satisfies at least one of the following: the reflectivity is less than or equal to 10% to reduce reflected light from the image source; or, the transmittance is greater than or equal to 90% to enhance starlight transmittance; or, the thickness of the third beam splitter in the second combination is less than or equal to 1 mm to reduce ghosting.

[0031] Based on the above technical solutions, the starry sky optical observation device is suitable for nighttime observation or observation in low light conditions. A reflectivity of less than or equal to 10% can reduce reflected light from the image source, a transmittance of greater than or equal to 90% can enhance starlight transmission, and a thickness of less than or equal to 1 mm can reduce ghosting.

[0032] One possible implementation is that the third optical path deflection element can be a mirror, prism, etc.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the third optical path deflection element is a front surface reflector. Since the reflective coating of the front surface reflector is directly disposed on the incident surface of the lens, this means that incident light is directly reflected upon contact with the surface of the front surface reflector without penetrating the glass substrate. Therefore, it effectively avoids the "parasitic optical path" (i.e., the "second reflecting surface" effect) caused by the superposition of "front surface refraction" and "rear surface reflection" when light penetrates the glass medium, thereby eliminating ghosting and beam distortion caused by the "second reflecting surface" effect and ensuring the fidelity and image clarity of the electronic starscape image during propagation.

[0034] In a second aspect, a starry sky optical observation system is provided, comprising: a terminal device and a starry sky optical observation apparatus as described in the first aspect and any possible implementation thereof; wherein, when the terminal device is installed in the mounting section, a first part of the terminal device protrudes outside the starry sky optical observation apparatus.

[0035] Regarding the installation of the terminal device in the mounting section, one possible implementation is that the terminal device is installed on the mounting section either detachably or non-detachably.

[0036] Based on the above technical solution, it is possible to superimpose distant virtual images with real scene light in stargazing scenarios, providing an unobstructed and immersive observation experience. Observers can clearly observe virtual markers without repeatedly adjusting their lenses. The stargazing optical observation device processes the electronic stargazing image emitted from the terminal device screen through at least one of the following methods: reflection, refraction, or increasing the virtual image distance. The resulting image is then refracted to the user via a beam splitter, which simultaneously transmits an external real stargazing image to the user. In this way, the electronic stargazing image accurately corresponds to the position of real celestial bodies directly observed by the user's naked eye, and the outward propagation direction of the electronic stargazing image is approximately parallel to the external real stargazing image. Users can clearly observe both without adjusting their lenses, providing an unobstructed and immersive virtual-real fusion observation experience.

[0037] Furthermore, users can interact with the terminal device through the first part without disassembling the terminal device, provided the terminal device is installed in the mounting unit.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the first part includes a first interactive area, which is a screen; the terminal device is configured to: adjust or generate a first electronic starry sky image in response to a first operation acting on the first interactive area; wherein adjusting the first electronic starry sky image includes adjusting at least one of position, size, and angle. In this way, the user can interact with the screen portion of the terminal device that is exposed outside the starry sky optical observation device, thereby adjusting or generating the first electronic starry sky image. This facilitates image correction without moving or disassembling the terminal device, making image correction more convenient and efficient.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the first operation includes at least one of a swipe operation, a long press operation, a double-tap operation, and a single-tap operation. For example, when a user swipes a portion of the screen, the position of the first electronic starry sky image on the screen adjusts accordingly, and in the direction of the swipe. Thus, different adjustment effects or generation scenarios can be achieved based on different user operations. Regarding adjustment effects, for example, a long press can enlarge the size of the first electronic starry sky image, a swipe can adjust the position of the first electronic starry sky image, a double-tap can increase the brightness of the first electronic starry sky image, and a single-tap can decrease the brightness of the first electronic starry sky image. This is just one example, and this application embodiment does not impose any special limitations on it. Regarding generation scenarios, such as the size of the generated first electronic starry sky image, the position of the generated first electronic starry sky image, and the brightness of the first electronic starry sky image, this application embodiment does not impose any special limitations on these aspects.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the first part includes a second interactive area, which is provided with at least one button; the terminal device is configured to adjust or generate a first electronic starry sky image in response to a second operation performed on the at least one button. For example, a physical button. In this way, the user can adjust or generate the first electronic starry sky image by operating the button on the outside of the starry sky optical observation device of the terminal device. This facilitates image correction without moving or disassembling the terminal device, making image correction more efficient and convenient.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the second operation includes at least one of a sliding operation, a long press operation, a double-click operation, and a single-click operation. For example, when a user clicks a button in the first part, the display size of the first electronic starry sky image on the screen changes. Thus, different adjustment effects or generation states can be achieved depending on the user's different operations. Regarding adjustment effects, for example, a long press can enlarge the size of the first electronic starry sky image, a sliding operation can adjust the position of the first electronic starry sky image, a double-click can increase the brightness of the first electronic starry sky image, and a single click can decrease the brightness of the first electronic starry sky image. This is just one example, and the embodiments of this application do not impose special limitations on it. Regarding generation states, such as the size of the generated first electronic starry sky image, the position of the generated first electronic starry sky image, and the brightness of the first electronic starry sky image, the embodiments of this application do not impose special limitations on them.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the terminal device includes a voice input module; the terminal device is configured such that the voice input module receives voice commands and, in response to the voice commands, adjusts or generates a first electronic starry sky image. The voice input module can be activated or deactivated through certain operations. In this way, the user can directly adjust or generate the first electronic starry sky image via voice control. Adjusting or generating the first electronic starry sky image does not require moving or disassembling the terminal device, and there are no significant restrictions on the size of the first part; the first part can be very small.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the first part includes a third interactive area, which is part of the screen; the terminal device is configured to activate or deactivate the voice input module in response to a third operation performed on the third interactive area. This allows the user to activate or deactivate the voice input module as needed, preventing the voice input module from misrecognizing speech when the user does not require voice control, thus avoiding incorrect adjustment or generation of the first electronic starry sky image, and also allowing the user to conveniently activate the voice input module as needed.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the third operation includes at least one of the following: swipe operation, long press operation, double-click operation, and single-click operation.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the terminal device includes a location acquisition module and / or an attitude acquisition module; wherein, the location acquisition module is used to acquire the geographical location information of the terminal device; and the attitude acquisition module is used to acquire the three-dimensional attitude and / or azimuth angle of the terminal device. In this way, the terminal device can determine its geographical location information with the help of the location acquisition module, and can also determine its three-dimensional attitude and / or azimuth angle with the help of the attitude acquisition module, which is beneficial for the generation and verification of the first electronic starscape image, and especially helpful for quickly generating virtual markers within it.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, the terminal device includes an image sensor; the first electronic starscape image includes a first calibration image, which includes at least one of the following: a positioning marker, which is set at a predetermined position on the first calibration image and includes at least one of the following: the center of the first calibration image, the edge of the first calibration image, and the principal imaging point of the first calibration image, used to calibrate position and orientation errors; or, image information representing the spatial layout of far-field targets acquired by the image sensor, used to calibrate position, orientation, and scaling errors. Thus, the positioning marker can provide users with visual reference points for aligning the starscape optical observation device and the terminal device, helping users calibrate and align the imaging angle and improve calibration efficiency; the image sensor can provide users with image propagation direction reference and image information of the spatial layout of far-field targets, which is beneficial for helping users calibrate and align the imaging angle and imaging distance, improving calibration efficiency.

[0047] Thirdly, an image processing method for astronomical optical observation is provided, applied to a terminal device in an astronomical optical observation system as described in any of the first to second aspects and any possible implementation thereof. The method includes: the terminal device's camera acquiring an original image; performing star detection on the original image to obtain a set of detected star points, the set of detected star points including multiple detected star points, each with corresponding coordinates; comparing the set of detected star points with a star catalog to obtain a final set of matching pairs, the final set of matching pairs including one or more matching pairs, each matching pair recording the coordinates of the detected star points and the coordinates of the matching star catalog star points, wherein the star catalog includes multiple star catalog star points, each with corresponding coordinates; calculating the pixel focal length based on the final set of matching pairs; obtaining a corrected rotation matrix based on the calculated pixel focal length, the final set of matching pairs, and the rotation matrix of the initial attitude of the terminal device; and rendering the terminal device's rendering module rendering a first electronic astronomical image based on the corrected rotation matrix.

[0048] It should be noted that after the first electronic starry sky image has been displayed, this method can ensure that the virtual identifiers or virtual images in the first electronic starry sky image are aligned with the real starry sky, thereby improving the user experience.

[0049] Based on the above technical solution, star detection of the original image can automatically identify celestial targets in the image without requiring manual selection of calibration points by the user; by calculating the pixel focal length, the camera's focal length deviation can be calibrated; and by calculating and correcting the rotation matrix, the attitude measurement error of the attitude acquisition module can be compensated, ensuring precise alignment of the star map with the real night sky, thereby improving imaging accuracy and image quality. Furthermore, the above calibration process is completed automatically by the system, requiring no astronomical observation knowledge or calibration experience from the user, lowering the barrier to entry, reducing the time spent on calibration operations, and enhancing the user experience.

[0050] In conjunction with the third aspect, in some implementations of the third aspect, the set of detected star points is compared with the star catalog to obtain the final set of matching pairs. This includes: selecting Q detected star points from multiple detected star points in descending order of brightness, where Q is a positive integer greater than 1; arbitrarily selecting two detected star points from the Q detected star points to form a reference pair combination, and the Q detected star points form a total of C(Q, 2) reference pairs, where... For each reference pair combination, the line connecting the positions of the two detected star points in the reference pair combination forms the first baseline, and the length of the first baseline is recorded as 1; the coordinates of the Q detected star points are used to calculate the normalized coordinates of the matching star points relative to the first baseline; two star points from multiple star catalogs are selected in descending order of brightness; the line connecting the positions of the two star points forms the second baseline, and the length of the second baseline is recorded as 1; the coordinates of the multiple star catalogs are used to calculate the normalized coordinates of the star catalogs relative to the second baseline; for each reference pair combination, the normalized coordinates of the matching star points and the normalized coordinates of the star catalogs are compared to obtain the matching pair set; the final matching pair set is selected from the C(Q,2) matching pair sets corresponding to the C(Q,2) reference pairs.

[0051] Based on the above technical solution, the detected star points can be compared and matched with real star points to obtain a set of successfully matched star points. The value of Q can be adjusted according to requirements; a larger Q value results in stronger anti-interference capability, while a smaller Q value leads to faster calculation speed.

[0052] In conjunction with the third aspect, in some implementations of the third aspect, the set of matching pairs satisfies the following two conditions: the difference between the coordinates of each matched detected star point and the first component coordinates of the star catalog star point is less than or equal to the third threshold; and the difference between the coordinates of each matched detected star point and the second component coordinates of the star catalog star point is less than or equal to the fourth threshold; wherein the values ​​of the first component coordinates and the second component coordinates are perpendicular to each other.

[0053] The third and fourth thresholds can be set and adjusted as needed, or users can set them themselves according to their requirements.

[0054] In conjunction with the third aspect, in some implementations of the third aspect, the final matching pair set is selected from the set of C(Q,2) matching pairs corresponding to the C(Q,2) reference pairs, including: selecting the set of matching pairs with the largest number of matching pairs that is greater than the reference number from the set of C(Q,2) matching pairs as the final matching pair set.

[0055] Based on the above technical solution, by matching multiple benchmark pairs separately and then selecting the optimal matching pair set from the multiple matching pair sets corresponding to the multiple benchmark pairs, the accuracy of matching can be improved.

[0056] In conjunction with the third aspect, in some implementations of the third aspect, the calculated pixel focal length is obtained based on the final set of matching pairs, including: the final set of matching pairs includes q pairs of matching stars; arbitrarily select 2 pairs of matching stars from the q pairs of matching stars to calculate the pixel focal length; the q pairs of matching stars constitute C(q,2) pixel focal length calculation combinations; solve for the pixel focal length estimate for each pixel focal length calculation combination, and obtain a total of C(q,2) pixel focal length estimates; determine the calculated pixel focal length from the C(q,2) pixel focal length estimates; and replace the camera calibration pixel focal length of the terminal device with the calculated pixel focal length.

[0057] Based on the above technical solution, the camera's calibrated focal length is continuously corrected by calculating the pixel focal length, resulting in a corrected focal length. This corrected focal length is then used to correct the rotation matrix, ensuring that the virtual image in the first electronic starry sky image is always aligned with the real starry sky, thus improving the user experience. This avoids situations where the virtual image in the first electronic starry sky image is misaligned with the real starry sky due to errors in the focal length calibrated by the terminal device's camera.

[0058] In conjunction with the third aspect, in some implementations of the third aspect, for each pixel focal length calculation combination, the corresponding estimated pixel focal length f is greater than 10 and satisfies the formula: f² = [cos²θ1(A+B)]. 2C ± ] / (2sin²θ1); The pixel focal length calculation combination includes the detection of star points I1 and I2, the direction vector of I1 in the body coordinate system d1=(x1, -y1, -f), and the direction vector of I2 in the body coordinate system d2=(x2, -y2, -f); where A satisfies the formula: A=x1²+y1², B satisfies the formula: B=x2²+y2², C satisfies the formula: C=x1x2+y1y2, θ1 satisfies the formula: cosθ1= (d1·d2) / (|d1|·|d2|), where · is the vector dot product operation, and a satisfies the formula: a = sin²θ1= 1 cos²θ1, b satisfies the formula: b = [ cos²θ1·(x1²+y1²+x2²+y2²) 2(x1x2+y1y2)], c satisfies the formula: c = (x1x2+y1y2)² cos²θ1·(x1²+y1²)(x2²+y2²).

[0059] In conjunction with the third aspect, in some implementations of the third aspect, determining the calculated pixel focal length from C(q,2) pixel focal length estimates includes: removing pixel focal length estimates that exceed the first range from the C(q,2) pixel focal length estimates to obtain one or more pixel focal length estimates; and statistically aggregating the one or more pixel focal length estimates to obtain the optimal estimate of the calculated pixel focal length.

[0060] One possible example is that the first range is related to the preset calibration pixel focal length of the terminal device camera, for example, greater than or equal to 0.8 times the preset calibration pixel focal length and less than or equal to 2 times the preset calibration pixel focal length. This application embodiment does not impose any special restrictions on this.

[0061] Based on the above technical solution, determining the optimal pixel focal length estimate from multiple pixel focal length estimates can improve the accuracy of pixel focal length and enhance the effect of pixel focal length correction.

[0062] In conjunction with the third aspect, in some implementations of the third aspect, a corrected rotation matrix is ​​obtained based on the rotation matrix of the final set of matching pairs and the initial attitude of the terminal device. This includes: calculating the direction vector b and direction vector r in the first coordinate system for each pair of matching stars in the final set of matching pairs, where direction vector b corresponds to the detected star point in each pair of matching stars and direction vector r corresponds to the star catalog point in each pair of matching stars; constructing a correlation matrix B based on direction vector b and direction vector r; performing singular value decomposition on the correlation matrix B to obtain the corrected rotation matrix R; and then, based on the corrected rotation matrix R and the initial attitude matrix of the terminal device... The corrected first effective attitude R is obtained. eff .

[0063] Based on the above technical solution, the initial attitude matrix of the terminal device is corrected to ensure that the virtual image in the first electronic starry sky image is always aligned with the real starry sky, thus improving the user experience. This avoids situations where the virtual image in the first electronic starry sky image is misaligned with the real starry sky due to errors in the initial attitude matrix of the terminal device itself.

[0064] In conjunction with the third aspect, in some implementations of the third aspect, a corrected rotation matrix is ​​obtained based on the rotation matrix of the final set of matching pairs and the initial pose of the terminal device. This includes: the final set of matching pairs includes q pairs of matching stars; for each pair of matching stars in the q pairs, a world system direction vector b and a direction vector r are calculated, where direction vector b corresponds to a detected star point in each pair of matching stars, and direction vector r corresponds to a star catalog point in each pair of matching stars; based on the correspondence between the direction vector b and direction vector r of each pair of matching stars, a correlation matrix B or its equivalent expression representing the overall rotation is constructed; the construction method of the correlation matrix B includes, for each pair of matching stars, calculating the outer product of the direction vector b and direction vector r, and summing the outer products calculated for all matching stars to obtain the correlation matrix B, where B satisfies: B = ; where r This represents the transpose of the direction vector r. This represents the summation calculation; for the correlation matrix B, the rotation estimation method is used to solve for the corrected rotation matrix R, so that the corrected rotation matrix satisfies the condition of minimizing the overall error; the rotation estimation method includes: performing singular value decomposition B=UΣV on the correlation matrix. Construct the corrected rotation matrix R = U·V When det(R) < 0, use U·diag(1,1, 1)·V Chirality correction; based on the corrected rotation matrix R and the initial attitude matrix of the terminal device. The corrected first effective attitude R is obtained. eff R eff satisfy: =R × (or equivalent expression), to complete the posture correction.

[0065] One possible implementation, whereby the modified rotation matrix satisfies the global error minimization condition, includes: ensuring that Σ‖b R·r‖² is minimized, meaning it represents the optimal rotation in the least squares sense.

[0066] In conjunction with the third aspect, in some implementations of the third aspect, the rotation matrix R and the initial attitude matrix of the terminal device are used as the basis. The corrected effective attitude is obtained. Previously, the method also included: calculating the rotation angle θ2 corresponding to the corrected rotation matrix R, where the rotation angle θ2 satisfies the formula: θ2 = arccos((trace 1) / 2); where arccos() is the inverse function of the cosine function cos(), and trace is the trace of the corrected rotation matrix R; when θ2 is less than or equal to a predetermined angle threshold, based on the corrected rotation matrix R and the initial attitude matrix of the terminal device... The corrected first effective attitude was obtained. .

[0067] In conjunction with the third aspect, in some implementations of the third aspect, the effective attitude truth value output by star map matching is used as the external observation anchor. Using multiple observations under different attitudes, the attitude error is decomposed into two independent parts: carrier fixed error and world fixed error. Among them, the carrier fixed error is fixed to the terminal device body and is constant in the carrier coordinate system, while the world fixed error is constant in the navigation coordinate system and does not change with the attitude of the terminal device. The carrier fixed error and the world fixed error are calibrated separately and applied to attitude correction respectively.

[0068] Based on the above technical solution, compared to only correcting one type of error in the attitude rotation matrix caused by the external environment or the carrier, this solution can simultaneously correct both the attitude rotation matrix error caused by the external environment and the attitude rotation matrix error caused by the mobile phone itself.

[0069] In conjunction with the third aspect, in some implementations of the third aspect, when the attitude deviation is small, a linearized fast initial value estimation is performed using rotation parameterization. The rotation parameterization method includes any of the following: rotation vector, quaternion, Lie algebra, or other parameterization methods that can achieve the same rotation representation. When using Lie algebra for linearized fast initial value estimation, the fixed error of the carrier in the carrier coordinate system is parameterized as a vector. And the fixed error parameterization into a vector in the world coordinate system Construct a linearized error model:

[0070] in, For the first Error vector of the second observation For the first The effective attitude of the second observation; will The combined observations form a system of linear equations:

[0071] in, It is a 3x3 identity matrix; to ensure the coefficient matrix is ​​of full rank, Each observation must include at least three different attitudes, and the carrier must exhibit attitude changes in at least two independent rotational degrees of freedom. Calculate the residual vector for each observation. And calculate the residual norm. If the residual norm is less than a preset threshold Then and The optimal estimate and The carrier fixed error rotation matrix is ​​obtained through exponential mapping. and world fixed error rotation matrix :

[0072] in, Let be the cross product matrix operator; where the optimal estimate is... and It refers to the estimated value obtained after calculating the residual norm. and Based on the carrier fixed error rotation matrix World Fixed Error Rotation Matrix and the current initial attitude matrix The corrected second effective attitude was obtained. Second effective posture Satisfying the formula:

[0073] The second effective posture Used for subsequent rendering projection. In one simplified implementation, when the world fixed error only includes rotation around the celestial axis, [the following will be used]. Degenerates into a single-parameter heading angle ,Right now In this case, the linearized model described above Degenerate into , , Let be the celestial unit vector in the world coordinate system; in this simplified case, the solution can be obtained using observation data from at least two different attitudes.

[0074] In conjunction with the third aspect, in some implementations of the third aspect, when the attitude deviation is large, a rotation parameterization method is used to iteratively solve the problem through exponential mapping or an equivalent rotation synthesis method. The rotation parameterization method includes any of the following: rotation vector, quaternion, Lie algebra, or other parameterization methods that can achieve the same rotation representation; wherein, when a Lie group is used... When performing exact solutions for nonlinear optimizations, the two types of error parameters are parameterized as Lie algebras and exponentially mapped: ; Construct a nonlinear least squares problem with the error matrix residuals as the cost function: ; Iterate the following two formulas separately until the residuals converge or the maximum number of iterations is reached to obtain the fixed error vector of the carrier. and world fixed error : ; ; in, Represents the cross product matrix operator. Represents the fixed error vector of the carrier The corresponding Lie algebra update amount, Represents a fixed deviation in the world coordinate system The corresponding Lie algebra update amount; and The calculation results are used to obtain the carrier fixed error rotation matrix through exponential mapping. and world fixed error rotation matrix :

[0075] in, and To obtain the solution estimated value , for Cross product matrix operator .

[0076] Based on the carrier fixed error rotation matrix World Fixed Error Rotation Matrix and the current initial attitude matrix The corrected second effective attitude was obtained. Second effective posture Satisfying the formula: ; the second effective posture Used for subsequent rendering projection; whereby, after solving for the carrier fixed error rotation matrix, it is directly applied to the error decomposition of subsequent frames; or, the carrier fixed error rotation matrix is ​​resolved based on the current observation attitude data during each star map observation.

[0077] Based on the above technical solution, the mathematical separation of carrier-fixed errors and world-fixed errors is achieved. A progressive strategy of fast estimation using first-round small-angle linear least squares and precise optimization using second-round complete exponential mapping nonlinearity achieves decoupled calibration of the two types of errors, balancing computational efficiency and robustness over large angles. Specifically, the first-round linear least squares approach converges quickly and is suitable for typical small-deviation scenarios; the second-round nonlinear precise optimization provides a fallback for large-deviation scenarios, balancing efficiency and robustness. Utilizing the real starry sky as the absolute attitude reference eliminates reliance on external reference equipment, improving convenience and reducing costs.

[0078] It should be noted that the carrier fixing error As an inherent property of the hardware, it changes little and remains effective after calibration; users in new locations can use previously known carrier-based fixed errors. Quickly re-estimate environmental bias by matching with star charts.

[0079] Generally, only 2-3 observations at different attitudes are needed to complete the initial separation calibration.

[0080] In conjunction with the third aspect, in some implementations of the third aspect, the terminal device includes an image sensor; the first electronic starry sky image includes a first calibration image, which includes at least one of the following: the center of the first calibration image, the edge of the first calibration image, and the device imaging principal point of the first calibration image, used to calibrate position and orientation errors; or, image information representing the spatial layout of far-field targets acquired by the image sensor, used to calibrate position errors, orientation errors, and scaling errors; the method further includes: classifying the projection error of the target virtual image relative to the observation line of sight into the following three categories: position errors, which are errors caused by the virtual image propagation direction being consistent with the observation direction but the image center not being aligned; orientation errors, which are errors caused by the virtual image propagation direction being inconsistent with the observation direction, specifically manifested as the virtual image plane not being perpendicular to the observation line of sight; scaling errors, which are errors caused by the object distance deviation causing a mismatch between the virtual image field of view distance and the magnification, specifically manifested as the overall image being scaled proportionally; and independently correcting the position errors, orientation errors, and scaling errors respectively, the correction including at least one of the following calibration methods: position error correction, by adjusting the terminal The projection position within the device's screen is aligned with the fixed position of the astronomical optical observation device to correct positional errors. For orientation errors, under the condition of small angles and without considering projection deformation, rotational errors around the horizontal and / or vertical axes are equivalent to an overall angular offset in the virtual image propagation direction. By adjusting the projection position within the terminal device's screen, the observation direction of the target virtual image is made consistent with the image acquisition direction of the image sensor, i.e., coincident with the projection propagation direction, thus correcting orientation errors. The correction of orientation errors is achieved through the translation of the projection center position, without introducing [missing information - likely related to image propagation direction]. Projection rotation parameters; scaling error correction: the image information representing the spatial layout of far-field targets acquired by the image sensor is scaled onto the virtual image plane to coincide with the observed entity, so that their perspectives are consistent, and the scaling matching coefficient R is obtained; the scaling matching coefficient is compared with the preset theoretical scaling factor to obtain the scaling deviation ΔR; based on the scaling deviation, the distance deviation Δu between the screen and the telescope element of the star observation device is calculated, and Δu = ΔR × F × f / d, where F is the equivalent focal length of the screen imaging, d is the exit pupil distance, and f is the focal length of the telescope element; at the same time, the equivalent focal length of the screen imaging is corrected to R. F enables precise alignment between the virtual image and the real observation target. It should be noted that the focal length of the zoom element here refers to the focal length of the zoom element in the astronomical optical observation system, or the equivalent focal length of the zoom element in the astronomical optical observation system. For example, when the astronomical optical observation system has only one zoom element, the focal length of the zoom element is the focal length of that single zoom element; when the astronomical optical observation system has two or more zoom elements, the focal length of the zoom element is the equivalent focal length of those two or more zoom elements, or the focal length of the zoom element as a combination of the two or more zoom elements.

[0081] Fourthly, an image processing method for astronomical optical observation is provided, applied to a terminal device in an astronomical optical observation system as described in any of the first to second aspects and any possible implementation thereof. The method includes: receiving a user's photo-taking request in a photo-taking mode; in response to the photo-taking request, the terminal device calculates the inter-frame rotation angle between the current attitude and the attitude of the previous frame in each subsequent sensor frame; triggering a photo-taking action when the inter-frame rotation angle is within a predetermined stationary angle window and / or when the time since receiving the photo-taking request exceeds a predetermined timeout period; extracting a complete full-frame image from a camera reference frame; and rendering a label overlay layer in a first electronic astronomical image at the size of the complete full-frame image based on the attitude of the terminal device and the focal length parameters of the terminal device, such that the label projection is geometrically matched with the photo; wherein the attitude is the attitude at the corresponding moment of the reference frame, and the focal length parameters are the focal length parameters at the corresponding moment of the reference frame.

[0082] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the reference frame satisfies at least one of the following: the reference frame is a frame triggered when taking a picture; the reference frame is the frame with the least jitter of the terminal device within a specified time after triggering the picture; the reference frame is the frame with the smallest rotation angle of the terminal device within a specified time after triggering the picture; at the time corresponding to the reference frame, the jitter of the terminal device is less than a sixth threshold; at the time corresponding to the reference frame, the rotation angle of the terminal device is less than a seventh threshold. This application does not impose special restrictions on the sixth and seventh thresholds. As an example, the seventh threshold is 0.01°. Attached Figure Description

[0083] Figure 1 This is a structural schematic diagram of a terminal device 100 applicable to this application; Figure 2 This is a schematic diagram of a starry sky optical observation device 200 proposed in an embodiment of this application; Figure 3 This is a schematic diagram of another starry sky optical observation device proposed in the embodiments of this application; Figure 4 This is a schematic diagram of another starry sky optical observation device 400 proposed in the embodiments of this application; Figure 5This is a schematic diagram of the optical path of a starry sky optical observation device proposed in an embodiment of this application; Figure 6 This is a schematic diagram of another starry sky optical observation device 600 proposed in the embodiments of this application; Figure 7 This is a schematic diagram of the optical path of another star-sky optical observation device proposed in the embodiments of this application; Figure 8 This is a schematic diagram of another starry sky optical observation device 800 proposed in the embodiments of this application; Figure 9 This is a schematic diagram of the optical path of another star-sky optical observation device proposed in the embodiments of this application; Figure 10 This application provides a starry sky optical observation system 1000. Figure 11 This is a schematic diagram of the calibration of a starry sky optical observation system proposed in an embodiment of this application; Figure 12 This is a schematic diagram of another starry sky optical observation system calibration proposed in the embodiments of this application; Figure 13 This is a calibration diagram of a starry sky optical observation proposed in an embodiment of this application; Figure 14 This application provides an image processing method for astronomical optical observation. Figure 15 This is a schematic diagram of the focal length for astronomical optical observation proposed in an embodiment of this application. Detailed Implementation

[0084] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0085] Before introducing the scheme of this application, the following points should be noted.

[0086] (1) In this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0087] (2) In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of A, B, and C can mean: A, or, B, or, C, or, A and B, or, A and C, or, B and C, or, A, B, and C. Where A, B, and C can be single or multiple.

[0088] (3) In this application, “first”, “second” and various numbers or letters (e.g., #1, #2, #A, #B, etc.) are used only to distinguish the objects being described for the purpose of explanation and do not constitute any limitation on the scope of the embodiments.

[0089] (4) In this application, the descriptions such as “when…”, “under the circumstances of…” and “if” all refer to the device making a corresponding action or being in a corresponding position under certain objective circumstances. They are not time limits, nor do they require the device to make a judgment action when it is implemented, nor do they mean that there are other limitations.

[0090] (5) In this application, “send” and “receive” indicate the direction of signal / light transmission. For example, “send an image to XX” can be understood as the destination of the image being XX. “receive an image from YY” can be understood as the source of the image being YY.

[0091] (6) In this application, “predefined” may mean that it is agreed or negotiated in advance within the device, or it may mean that it is pre-configured. The “pre-configuration” can be achieved by pre-saving the corresponding code, table or other means that can be used to indicate relevant information in the device. This application does not limit the implementation method.

[0092] (7) In this application, “message”, “information”, “signal” or “information element (IE)” can be used interchangeably. There are no restrictions on the name of the message or information, as long as it can achieve the corresponding function.

[0093] (8) In this application, the words “as an example,” “exemplarily,” “for example,” etc., are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as an “example” in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word “example” is intended to present the concept in a specific manner. In the embodiments of this application, “of,” “corresponding, relevant,” and “corresponding” may sometimes be used interchangeably, and it should be noted that their intended meanings are consistent when their distinctions are not emphasized.

[0094] (9) In this application, the sequence numbers of each process are used only for differentiation and do not indicate the order of execution. The actual execution order of each process should be determined based on its function and internal logic, and should not be construed as a limitation on the embodiments of this application. The terms 'including', 'comprising', 'having' and their variations all mean "including but not limited to", unless otherwise specified.

[0095] In this embodiment, the terminal device in the AR starry sky optical observation system can be a device for implementing the functions of the terminal device. It can be a terminal device itself, or a device capable of supporting the terminal device in implementing these functions, such as a chip system, chip, circuit, or communication module (i.e., a communication module that performs communication functions). This device can be installed in the terminal device. In this embodiment, the chip system can be composed of chips or can include chips and other discrete devices. Furthermore, the device can also be configured with program instructions for performing corresponding communication functions.

[0096] The terminal device can be an electronic device such as a mobile phone, tablet computer, wearable device, augmented reality device, laptop computer, ultra-mobile personal computer (UMPC), netbook, etc., that has AR technology capabilities (e.g., AR pose calculation, AR rendering, AR environment perception, etc.) and has AR collaborative applications installed. This application embodiment does not impose any restrictions on the specific type of terminal device.

[0097] The terminal device can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it can also be deployed on water; and it can also be deployed in the air on airplanes, balloons, and satellites. This application does not limit the scenario in which the terminal device is located.

[0098] The following combination Figure 1 This description applies to a terminal device applicable to this application.

[0099] The method provided in this application can be applied to terminal devices with image storage functions, such as mobile phones, tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application does not impose any restrictions on the specific type of terminal device.

[0100] For example, Figure 1 A schematic diagram of the terminal device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0101] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0102] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0103] The controller can serve as the central nervous system and command center of the terminal device 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0104] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0105] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0106] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal device 100, and can also be used for data transfer between terminal device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0107] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0108] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0109] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0110] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0111] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0112] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0113] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0114] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0115] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or K displays 194, where K is a positive integer greater than 1.

[0116] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0117] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0118] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 100 may include one or K cameras 193, where K is a positive integer greater than 1.

[0119] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0120] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0121] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0122] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0123] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of terminal device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0124] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0125] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0126] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 100 can receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0127] The software system of terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture system as an example to exemplify the software structure of terminal device 100.

[0128] To facilitate a better understanding of the technical solutions of this application, some terms or concepts involved in the technical solutions of this application are introduced. The terms or concepts introduced below are merely illustrative examples for ease of understanding and do not limit the scope of protection of the embodiments of this application.

[0129] 1. Augmented Reality (AR). AR refers to the clever integration of virtual information with the real world. For example, video or images of a real scene are captured in real time through the camera or other devices of an electronic device. Then, algorithms such as simultaneous localization and mapping (SLAM) are used to process the captured video or images and generate a map of the real scene environment. The real-time captured video or images of the real scene are then transmitted to the processing unit in the background of the electronic device. The processing unit analyzes and reconstructs the data, and combines it with the real scene environment map to analyze the relative positions of the virtual and real scenes, align the coordinate systems, and overlay the virtual and real scenes in real time. At the same time, the combined virtual and real scene can be displayed in front of the user in real time, and the user can interact with the combined virtual and real scene through interactive devices.

[0130] 2. Beam Splitter. A beam splitter is an optical element capable of partially reflecting and partially transmitting light, including but not limited to beam splitters, beam splitting prisms, holographic optical elements, and polarizing beam splitters. A beam splitter, in particular, is an optical element that splits a beam of light into two beams according to a specific reflection and transmission ratio. Beam splitters are common optical elements in typical laser optical systems, illumination optical systems, and spectrometer optical systems. Users can split incident light according to the percentage of intensity transmitted and reflected light, and the polarization state of the transmitted and reflected light, based on specific application requirements.

[0131] 3. Fresnel lens. A Fresnel lens, also known as a threaded lens, is generally a thin sheet made of polyolefin material by injection molding, but it can also be made of glass. One side of the lens surface is smooth, while the other side is engraved with concentric circles of varying sizes. Its texture is designed according to the interference and scattering of light, as well as the requirements for relative sensitivity and receiving angle.

[0132] The technical problem to be solved and the technical solution adopted in this application are described below.

[0133] With the development of technology, people's interest in astronomical observation is increasing. The application of AR technology in the field of astronomical observation is also becoming more and more widespread. At present, AR stargazing products are usually implemented in the following way: using the camera of a mobile device to capture images of the starry sky, and superimposing virtual star information (such as virtual star markers, constellation lines, etc.) on the captured images, and finally displaying the composite image on the screen of the mobile device for users to view.

[0134] However, the above solution has significant technical flaws. Firstly, it requires users to observe the night sky through the device's screen. The device's physical structure obstructs the user's field of view, limiting the virtual information to the screen's display area and preventing it from seamlessly integrating with the real night sky as seen by the naked eye. Secondly, due to the inherent difference between the device's camera's imaging viewpoint and the user's visual viewpoint, the virtual information superimposed on the screen image struggles to accurately correspond to the actual celestial positions observed by the user with the naked eye in terms of spatial perspective. This results in the user being unable to accurately identify target celestial objects in the actual night sky based on the enhanced information on the screen, leading to distorted perspective.

[0135] In view of this, this application proposes a starry sky optical observation device, system, and image processing method. This optical observation device can process electronic starry sky images emitted from a terminal device screen through at least one of the following methods: reflection, refraction, and increasing the virtual image distance. The resulting image is then refracted to the user via a beam splitter, which simultaneously transmits an external real starry sky image to the user. In this way, the electronic starry sky image accurately corresponds to the positions of real celestial objects directly observed by the user's naked eye, and the propagation direction of the electronic starry sky image is approximately parallel to the external real starry sky image. The user can clearly observe both without adjusting their lens, providing an unobstructed, immersive virtual-real fusion observation experience.

[0136] The apparatus and system provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. The embodiments provided in this application can be applied to the scenarios shown in the above figures and are not limited thereto. Furthermore, the terms used below are as explained above and will not be repeated hereafter. The term "terminal device" can be replaced by components of a terminal device (e.g., a chip, chip system, circuit, or communication module). Furthermore, the steps described below as being performed by a single execution entity can also be divided into steps performed by multiple execution entities, which can be logically and / or physically separated.

[0137] Figure 2 This is a schematic diagram of a starry sky optical observation device 200 according to an embodiment of this application. See also... Figure 2 The starry sky optical observation device 200 includes a mounting section 210, an optical path assembly 220, and a first beam-splitting element 230. The first beam-splitting element 230 includes a first transmission surface 231 and a first refractive surface 232. The mounting section 210 supports a terminal device (not shown in the figure), such that the screen of the terminal device faces the starry sky optical observation device, and the screen emits a first electronic starry sky image 241. It is understood that the first electronic starry sky image 241 is a virtual image. Figure 2 The first electronic starry sky image 241 emitted by the terminal device is transmitted to the optical path assembly 220. The optical path assembly 220 is used to perform at least one of the following operations on the first electronic starry sky image 241 to obtain a second electronic starry sky image 242: reflection, refraction, and increasing the virtual image distance. It is understood that the second electronic starry sky image 242 is a virtual image. Then, the second electronic starry sky image 242 is transmitted to the first beam splitter 230. The first refracting surface 232 faces the optical path assembly 220 and is used to reflect and / or refract the second electronic starry sky image 242 towards the user. The first transmission surface 231 is used to transmit a real starry sky image to the user.

[0138] It should be noted that, Figure 2The arrows in the diagram illustrate the image propagation path and are only used to indicate the order in which the image is transmitted between structures. They do not represent the actual exit angle, beam width, or physical propagation direction of the light rays. The relative positions between the components are also only schematic and do not constitute a limitation on the actual structure.

[0139] The first electronic starry sky image can be obtained by processing real starry sky images captured by a camera using a terminal device. For example, the first electronic starry sky image is generated by the terminal device based on information from real starry sky images and combined with at least one of the following: the attitude of the terminal device, the device position, and the current observation time.

[0140] Based on the above technical solution, the virtual starry sky image emitted by the terminal device, after being processed by the optical path components, reaches the first beam splitter. The first reversing surface of the first beam splitter transmits the virtual starry sky image to the user, while the first transmitting surface of the first beam splitter transmits an external real starry sky image to the user. In this way, the electronic starry sky image can accurately correspond to the positions of real celestial bodies directly observed by the user's naked eye, and the outward propagation direction of the electronic starry sky image is approximately parallel to the external real starry sky image, so that the two arrive at the user's field of vision simultaneously and are superimposed, thus providing the user with an unobstructed, immersive virtual-real fusion observation experience.

[0141] One possible implementation is to use a low-reflection, high-transmittance lens as the first beam splitter.

[0142] In one example, the first beam-splitting element satisfies at least one of the following: reflectivity less than or equal to 10%; or transmittance greater than or equal to 90%; or thickness less than or equal to 1 millimeter (mm). A reflectivity below 10% and a transmittance above 90% ensure better transmission of real starry sky images from the outside, reducing reflected light from virtual starry sky images and adapting to the needs of stargazing in low-light environments at night. A thickness of less than or equal to 1 mm in the first beam-splitting element effectively avoids the impact of ghosting on observation.

[0143] Regarding the mounting unit 210 carrying the terminal device, one possible implementation is that the mounting unit 210 can detachably or non-detachably fix the terminal device. For example, see... Figure 3 , Figure 3 This is a schematic diagram of another astronomical optical observation device proposed in an embodiment of this application, which shows a terminal device 311. The terminal device 311 is not part of the astronomical optical observation device; the two are used in conjunction. Figure 3 It can be seen that the mounting part 310 is used to carry the terminal device 311. This will be discussed later in conjunction with... Figures 3 to 9 This paper introduces several possible implementation methods for the 200-star optical observation device.

[0144] First, combine Figures 3 to 5This paper introduces an optical observation device for the starry sky. Among its components, Figure 4 This is a schematic diagram of another starry sky optical observation device 400 proposed in the embodiments of this application. Figure 5 This is a schematic diagram of the optical path of a starry sky optical observation device proposed in an embodiment of this application.

[0145] It should be noted that the propagation of the electronic starscape images mentioned in the embodiments of this application is essentially the propagation of light signals (or light beams). For the sake of simplicity and clarity in the accompanying drawings, Figure 5 , Figure 7 and Figure 9 Each propagation stage is shown with only one or a few arrowed lines to indicate the main propagation direction of the image. In actual physical scenarios, image propagation may be carried out by a surface light source or beam composed of more light rays. Furthermore, the arrowed lines in the figures are merely a virtual representation of the light path and do not represent corresponding physical entities within the device. Similarly, the dashed optical axis in the figures is merely a virtual representation of the light path propagation direction and does not represent corresponding physical entities within the device. Moreover, the optical axis angles, light incident points, incident / reflection / refraction angles, and relative proportions between optical elements shown in the figures are merely possible examples and technical illustrations drawn for ease of understanding; they do not constitute a strict limitation on specific optical parameters and physical dimensions in actual products.

[0146] See Figures 3 to 5 The starry sky optical observation device 400 includes a mounting section 310, a first beam splitter 330, and an optical path assembly. In one example, the optical path assembly includes a first optical path deflection element 321 and a first virtual image retraction element 322. The relevant components will be described in sequence according to the optical path direction.

[0147] At least one side of the first optical path reversing element 321 faces the screen of the terminal device 311. The first optical path reversing element 321 is used to reflect and / or refract the first electronic star image to obtain a third electronic star image. Figure 5 As shown, exemplarily, a first electronic starry sky image emitted from the screen of terminal device 311 propagates along the first optical axis 511 to the first optical path reversing element 321. The first optical path reversing element 321 reflects the first electronic starry sky image to obtain a third electronic starry sky image propagating along the second optical axis 512. Figure 5 The image also shows the normal 1 at the incident point of the first electronic star image at the first optical path deflection element 321.

[0148] The first optical path deflection element 321 can be a reflector, prism, etc., and this application embodiment does not make any special limitation on it. Figures 3 to 5The image only shows the case where the first optical path reversing element 321 is a reflector. One possible implementation is that the first optical path reversing element 321 is a front surface reflector. The reflective coating of the front surface reflector is directly disposed on the incident surface of the lens, meaning that incident light is directly reflected upon contact with the surface of the front surface reflector without penetrating the glass substrate. Therefore, it effectively avoids the "parasitic optical path" (i.e., the "second reflecting surface" effect) caused by the superposition of "front surface refraction" and "rear surface reflection" when light penetrates the glass medium, thereby eliminating ghosting and beam distortion caused by the "second reflecting surface" effect and ensuring the fidelity and image clarity of the electronic starry sky image during propagation.

[0149] Then, continue to see Figures 3 to 5 The third electronic starscape image, propagating along the second optical axis 512, reaches the first virtual image retractor 322. At least one side of the first virtual image retractor 322 faces the first optical path deflection element 321. The first virtual image retractor 322 is used to increase the virtual image distance of the third electronic starscape image, resulting in a second electronic starscape image. In other words, the virtual image distance of the second electronic starscape image is greater than the virtual image distance of the third electronic starscape image. One example is... Figure 5 As shown, the second electronic star image still propagates along the second optical axis 512. In another example (not shown in the figure), the second electronic star image no longer propagates along the second optical axis 512.

[0150] Based on the above technical solution, the first virtual image teleportation element 322 can effectively project the virtual image onto a distant point, matching its optical distance to the real starry sky. Therefore, users can clearly observe both the "electronic starry sky image" and the "real starry sky image" at the same focal length without frequently adjusting their eye lens. This design not only makes the fusion of virtual and real scenes more natural but also eliminates visual fatigue caused by switching viewing distances back and forth, significantly improving the comfort of long-term observation.

[0151] The first virtual image retracting element 322 can be a Fresnel lens or a cemented lens group, or it can be a single lens, a freeform mirror, a concave mirror, etc. The embodiments of this application do not impose any special limitations on this. Figures 3 to 5 Only the case where the first virtual image retractor 322 is a Fresnel lens is shown. Compared to cemented lens groups, Fresnel lenses are lighter, making them more convenient for users to carry and move, reducing the user's operational burden, and improving the ease of use in practice. One possible implementation is that the first virtual image retractor 322 is a cemented lens group, for example, a cemented lens group including a lens group and / or a mirror group, which can effectively reduce aberrations, such as chromatic aberration, and may be suitable for applications requiring higher image quality.

[0152] See also Figures 3 to 5The second electronic star image propagates along the second optical axis 512 to the first refractive surface 541 of the first beam splitter 330. The first refractive surface 541 reflects and / or refracts the second electronic star image toward the user 520. One example is... Figure 5 The image shows the reflection of the second electronic starry sky image by the first folding surface 541 towards the user 520, resulting in a target electronic starry sky image that propagates along the third optical axis 513 and eventually reaches the user 520. Figure 5 The diagram also shows the normal 2 of the second electronic star image at the incident point of the first beam splitter 330. Simultaneously, the first transmission surface 551 of the first beam splitter 330 transmits a real star image 530 to the user 520. As an example, the real star image 530 transmitted by the first beam splitter 330 also propagates along the third optical axis 513.

[0153] The first beam splitting element 330 can be a beam splitter, a beam splitting prism, a holographic optical element, a polarizing beam splitter, or a combination of two or more of these elements. This application does not impose any special limitations on this. Figures 3 to 5 Only the case where the first beam splitter 330 is a beam splitter is shown.

[0154] Based on the above technical solution, both the target electronic star image and the real star image 530 will arrive at the user 520 simultaneously, providing an unobstructed, immersive virtual-real fusion observation experience. Furthermore, the first virtual image telescoping element 322 effectively magnifies the virtual image distance of the electronic star image and matches it with the real star image 530. The user 520 can see the clear dual effects at the same focal length without frequent lens adjustments, effectively alleviating visual fatigue and improving user comfort.

[0155] Starry sky optical observation devices can also include other structures. Two examples are described below.

[0156] As an example, a starry sky optical observation device may also include a frame, such as Figures 3 to 5 The frame 340 is used to mount or fix optical elements, such as at least one of the following: a first beam splitter 330, a first optical path deflector 321, and a first virtual image retractor 322. The frame 340 can also be used to mount or fix the mounting portion 310. As an example, Figures 3 to 5 The first beam splitter 330, the first optical path deflector 321, the first virtual image retraction element 322, and the mounting portion 310 shown are all fixed to the frame 340. Furthermore, the frame 340, in addition to... Figures 3 to 5 Besides the tetrahedron shown, there can be many other types, such as triangular prisms, pentagonal prisms, hexagonal prisms, etc. The embodiments of this application do not impose any special restrictions on these.

[0157] Another example is that a star-gazing optical observation device may also include a display area, such as Figure 4 Display area 410 in the middle. As an example, Figure 4 The display area 410 shown is a planar lens, but it can also be other types, such as a magnifying glass. This embodiment does not impose any special limitations on this. As another example, when the device includes a frame 340, the display area 410 is an area on the frame 340. This area allows the first electronic starry sky image to pass through without any obstruction, ensuring that the image can be transmitted to the rear optical elements without loss. The material of the display area 410 can also be various, such as glass, acrylic, or a filter used to reduce the intensity of terminal light. This embodiment does not impose any special limitations on this.

[0158] Next, combine Figure 6 and Figure 7 Introducing another type of optical observation device for the starry sky.

[0159] Figure 6 This is a schematic diagram of another starry sky optical observation device 600 proposed in the embodiments of this application. Figure 7 This is a schematic diagram of the optical path of another star-sky optical observation device proposed in the embodiments of this application.

[0160] See Figure 6 and Figure 7 The starry sky optical observation device 600 includes a mounting section 610, a first beam splitter 630, and an optical path assembly. In one example, the optical path assembly includes a second beam splitter 621 and a second optical path deflector 622. The second optical path deflector 622 is disposed on the side of the second beam splitter 621 away from the terminal device screen (not shown in the figure). In other words, the second optical path deflector 622 is disposed on the side of the second beam splitter 621 away from the mounting section 610. For information regarding the mounting section 610 carrying the terminal device, please refer to [link to relevant documentation]. Figure 2 Related descriptions. When the mounting unit 610 carries the terminal device, the screen of the terminal device faces the second beam splitter 621, and the second optical path deflection element 622 is located on the side of the second beam splitter 621 away from the screen of the terminal device. The relevant components will be introduced in sequence according to the optical path.

[0161] See also Figure 6 and Figure 7 The second beam-splitting element 621 includes a second transmission surface 752. The second transmission surface 752 faces the terminal device, for example, the screen of the terminal device. The second transmission surface 752 is capable of transmitting the first electronic star image to obtain a fourth electronic star image. Figure 7As shown, exemplarily, a first electronic star image emitted from the screen of terminal device 711 propagates along the fourth optical axis 714 to the second beam splitter 621. The second transmission surface 752 of the second beam splitter 621 is used to transmit the first electronic star image to obtain a fourth electronic star image. In one example, the fourth electronic star image still propagates along the fourth optical axis 714. In another example (not shown in the figure), the fourth electronic star image no longer propagates along the fourth optical axis 714.

[0162] It should be noted that the fourth optical axis 714 can be connected with... Figure 5 The first optical axis 511 in the image can be parallel, or it can be non-parallel; this application does not impose special restrictions on this. For example, when... Figure 5 Terminal equipment 311 and Figure 7 When the terminal device 711 is placed at the same angle, the fourth optical axis 714 may be aligned with... Figure 5 The first optical axis 511 is parallel.

[0163] The second beam-splitting element 621 can be a beam splitter, a beam splitter prism, a holographic optical element, a polarizing beam splitter, or a combination of two or more of these elements. This application does not impose any special limitations on this aspect. Figure 6 and Figure 7 Only the case where the second beam splitter 621 is a beam splitter is shown.

[0164] One possible implementation is that the second beam splitter 621 is made of a low-reflection, high-transmittance lens.

[0165] In one example, the second beam-splitter 621 satisfies at least one of the following: a reflectivity of less than or equal to 10% to reduce reflected light from the image source; or a transmittance of greater than or equal to 90% to enhance starlight transmission; or a thickness of less than or equal to 1 millimeter (mm) to reduce ghosting. A reflectivity below 10% and a transmittance above 90% ensure better transmission of the real starry sky image from the outside, reducing reflected light from the virtual starry sky image and meeting the needs of stargazing in low-light environments at night. A thickness of less than or equal to 1 mm in the second beam-splitter 621 effectively avoids the impact of ghosting on observation.

[0166] In another example, the second beam splitter 621 and the first beam splitter 630 constitute a first combination; the first combination satisfies at least one of the following: the reflectivity is less than or equal to 10% to reduce reflected light from the image source; or, the transmittance is greater than or equal to 90% to enhance starlight transmittance; or, the thickness of the second beam splitter 621 in the first combination is less than or equal to 1 mm to reduce ghosting.

[0167] See also Figure 6 and Figure 7The fourth electron star image propagates along the fourth optical axis 714 to the second optical path reversing element 622. The second optical path reversing element 622 is used to reflect and / or refract the fourth electron star image to obtain a fifth electron star image propagating along the fifth optical axis 715.

[0168] The second optical path deflecting element 622 can be a mirror, prism, etc., and this application embodiment does not impose any special limitations on it. One possible implementation is that the second optical path deflecting element 622 includes a first concave surface 760, which faces the second beam splitting element 621. See also... Figure 7 In one example, the first concave surface 760 of the second optical path deflection element 622 reflects the fourth electronic star image to obtain the fifth electronic star image propagating along the fifth optical axis 715. Figure 7 The image also shows the normal 3 of the third electronic star image at the incident point of the second optical path reversing element 622. In this way, the first concave surface 760 of the second optical path reversing element 622 can increase the virtual image distance (virtual image pull-out) of the electronic star image while realizing optical path reversal. Figure 6 and Figure 7 The diagram illustrates a scenario where the second optical path deflection element 622 is a mirror including a first concave surface 760. One possible implementation is that the second optical path deflection element 622 is a front surface mirror. The reflective coating of the front surface mirror is directly disposed on the incident surface of the lens, meaning that incident light is directly reflected upon contact with the surface of the front surface mirror without penetrating the glass substrate. Therefore, the "parasitic optical path" (i.e., the "second reflecting surface" effect) caused by the superposition of "front surface refraction" and "rear surface reflection" when light penetrates the glass medium can be effectively avoided. This eliminates ghosting and beam distortion caused by the "second reflecting surface" effect, ensuring the fidelity and image clarity of the electronic starscape image during propagation.

[0169] See also Figure 6 and Figure 7 The second beam splitter 621 further includes a second refractive surface 742, which faces the second optical path refractive element 622. Understandably, at least one side of the second optical path refractive element 622 faces the second beam splitter 621. Subsequently, the fifth electron star image propagating along the fifth optical axis 715 propagates to the second refractive surface 742, which is used to reflect and / or refract the fifth electron star image. One example... Figure 7 The image shows the reflection of the fifth electron starscape by the second refracting surface 742. After the second refracting surface 742 reflects and / or refracts the fifth electron starscape, a sixth electron starscape or a second electron starscape can be obtained. These two possible implementations will be described below.

[0170] In a first possible implementation (not shown in the figure), the second refractive surface reflects and / or refracts the fifth electronic star image to obtain a second electronic star image, which is then propagated to the first beam splitter. Subsequently, the first refractive surface of the first beam splitter reflects and / or refracts the second electronic star image to the user.

[0171] The second possible implementation, the starry sky optical observation device, also includes a second virtual image extension element. Figure 6 and Figure 7 This possible implementation is illustrated, in which the optical path assembly further includes a second virtual image retractor 623, with at least one side of the second virtual image retractor 623 facing the second refractive surface 742. The second refractive surface 742 reflects and / or refracts the fifth electronic star image to obtain a sixth electronic star image. The sixth electronic star image propagates to the second virtual image retractor 623. The second virtual image retractor 623 then increases the virtual image distance of the sixth electronic star image to obtain a second electronic star image, which propagates to the first beam splitter 630. The first refractive surface 743 of the first beam splitter 630 then reflects and / or refracts the second electronic star image towards the user 720.

[0172] It is understandable that when the second optical path deflection element 622 is provided with the first concave surface 760, the concave surface structure itself possesses positive refractive power, which can increase the virtual image distance of the electronic starry sky image. Based on this premise, compared to Figures 3 to 5 The first virtual image zooming element 322 shown is... Figure 6 and Figure 7 The second virtual image extending element 623 only needs to be set with a lower diopter to achieve the same virtual image distance increasing function. In other words, the first concave surface 760 shares part of the task of increasing the virtual image distance, thereby reducing the optical power requirement of the second virtual image extending element 623.

[0173] Both of these possible implementations ultimately result in the first beam-splitting element 630 reflecting and / or refracting the second electronic starscape image to the user 720, allowing the user 720 to view this virtual electronic starscape image. Simultaneously, the first transmission surface 753 of the first beam-splitting element 630 can transmit an external real starscape image to the user 720. In this way, both the target electronic starscape image and the real starscape image 730 arrive at the user 720 simultaneously, providing an unobstructed, immersive virtual-real fusion observation experience. The following description will use the second possible implementation as an example.

[0174] See also Figure 6 and Figure 7The second possible implementation is described in detail below. The second folding surface 742 of the second beam splitter 621 reflects the fifth electron star image, obtaining a sixth electron star image propagating along the sixth optical axis 716. Then, the sixth electron star image propagates to the second virtual image retractor 623, which increases the virtual image distance of the sixth electron star image, resulting in a second electron star image. In other words, the virtual image distance of the second electron star image is greater than the virtual image distance of the sixth electron star image. One example is... Figure 7 As shown, the second electron star image still propagates along the sixth optical axis 716. In another example (not shown in the figure), the second electron star image no longer propagates along the sixth optical axis 716.

[0175] Based on the above technical solution, the second virtual image teleportation element 623 can effectively project the virtual image onto a distant point, matching its optical distance to the real starry sky. Therefore, the user 720 can clearly observe both the "electronic starry sky image" and the "real starry sky image" at the same focal length without frequently adjusting their eye lens. This design not only makes the fusion of virtual and real scenes more natural but also eliminates visual fatigue caused by constantly switching viewing distances, significantly improving the comfort of long-term observation.

[0176] The second virtual image retracting element 623 can be a Fresnel lens or a cemented lens group, or it can be a single lens, a freeform mirror, a concave mirror, etc. The embodiments of this application do not impose any special limitations on this. Figure 6 and Figure 7 Only the case where the second virtual image retractor 623 is a Fresnel lens is shown. Compared to cemented lens groups, Fresnel lenses are lighter, making them more convenient for users to carry and move, reducing the user's operational burden, and improving the ease of use in practice. One possible implementation is that the second virtual image retractor 623 is a cemented lens group, for example, a cemented lens group including a lens group and / or a mirror group, which can effectively reduce aberrations, such as chromatic aberration, and may be suitable for applications requiring higher image quality.

[0177] Subsequently, the second electronic star image propagates along the sixth optical axis 716 to the first refractive surface 743 of the first beam splitter 620. The first refractive surface 743 faces the optical path assembly and reflects and / or refracts the second electronic star image toward the user 720. One example is... Figure 7 The image shows the first folding surface 743 reflecting the second electronic star image to the user 720, resulting in a target electronic star image propagating along the seventh optical axis 717, which eventually reaches the user 720. Figure 7The diagram also shows the normal 5 of the second electronic star image at the incident point of the first beam splitter 630. Simultaneously, the first transmission surface 753 of the first beam splitter 630 transmits a real star image 730 to the user 720. In one example, the real star image 730 transmitted by the first beam splitter 630 also propagates along the seventh optical axis 717.

[0178] The first beam-splitting element 630 can be a beam splitter, a beam splitter prism, a holographic optical element, a polarizing beam splitter, or a combination of two or more of these elements. This application does not impose any special limitations on this aspect. Figure 6 and Figure 7 Only the case where the first beam splitter 630 is a beam splitter is shown.

[0179] Based on the above technical solution, both the target electronic star image and the real star image 730 will arrive at the user 720 simultaneously, providing an unobstructed, immersive virtual-real fusion observation experience. Furthermore, the first concave surface 760 of the second virtual image telescoping element 623 and / or the second optical path reversing element 622 effectively magnifies the virtual image distance of the electronic star image and matches it with the real star image 730. The user 720 can see a clear dual effect at the same focal length without frequent lens adjustments, effectively alleviating visual fatigue and improving user comfort.

[0180] Starry sky optical observation devices may also include other structures, such as Figure 6 and Figure 7 The frame 640 shown in the figure, for example, Figure 6 The display area 641 is shown in the image. See the preceding text for further details. Figures 3 to 5 The relevant descriptions in [the text] will not be elaborated upon here. Among them, Figure 6 and Figure 7 The frame 640 shown in the figure can be seen in the reference to Figures 3 to 5 The relevant description of frame 340 in the document; Figure 6 The display area 641 shown can be seen in the reference to Figure 4 The relevant description of display area 410 in the text.

[0181] Next, combine Figure 8 and Figure 9 Introducing another type of optical observation device for the starry sky.

[0182] Figure 8 This is a schematic diagram of another starry sky optical observation device 800 proposed in the embodiments of this application. Figure 9 This is a schematic diagram of the optical path of another star-sky optical observation device proposed in the embodiments of this application.

[0183] See Figure 8 and Figure 9The astronomical optical observation device 800 includes a mounting section 810, a first beam splitter 830, and an optical path assembly. As an example, the optical path assembly includes a third beam splitter 822 and a third optical path folding element 821. For details regarding the mounting section 810 carrying the terminal equipment, please refer to [link to relevant documentation]. Figure 2 Related descriptions. When the mounting unit 810 carries the terminal device, at least one side of the third optical path folding element 821 faces the screen of the terminal device. The relevant components will then be described in sequence according to the optical path.

[0184] See also Figure 8 and Figure 9 For example, the first electronic star image emitted by the screen of the terminal device 911 propagates along the eighth optical axis 918 to the third optical path reversing element 821. The third optical path reversing element 821 is used to reflect and / or refract the first electronic star image to obtain a seventh electronic star image propagating along the ninth optical axis 919. Figure 9 The image also shows the normal 6 of the first electronic star image at the incident point of the third optical path reversing element 821.

[0185] It should be noted that the eighth optical axis 918 can be connected with... Figure 5 The first optical axis 511 in the image can be parallel, or it can be non-parallel; this application does not impose special restrictions on this. For example, when... Figure 5 Terminal equipment 311 and Figure 9 When the terminal device 911 is placed at the same angle, the eighth optical axis 918 may be aligned with... Figure 5 The first optical axis 511 is parallel to it. Similarly, the eighth optical axis 918 can be parallel to it. Figure 7 The fourth optical axis 714 in the image can be parallel, or it can be non-parallel; this application does not impose special restrictions on this. For example, when... Figure 7 Terminal equipment 711 and Figure 9 When the terminal device 911 is placed at the same angle, the eighth optical axis 918 may be aligned with... Figure 7 The fourth optical axis 714 is parallel.

[0186] The third optical path deflection element 821 can be a reflector, prism, etc., and this application embodiment does not make any special limitation on it. Figure 8 and Figure 9The diagram only shows the case where the third optical path reversing element 821 is a reflector. One possible implementation is that the third optical path reversing element 821 is a front surface reflector. The reflective coating of the front surface reflector is directly disposed on the incident surface of the lens, meaning that incident light is directly reflected upon contact with the surface of the front surface reflector without penetrating the glass substrate. Therefore, it effectively avoids the "parasitic optical path" (i.e., the "second reflecting surface" effect) caused by the superposition of "front surface refraction" and "rear surface reflection" when light penetrates the glass medium, thereby eliminating ghosting and beam distortion caused by the "second reflecting surface" effect and ensuring the fidelity and image clarity of the electronic starry sky image during propagation.

[0187] See also Figure 8 and Figure 9 The third beam splitter 822 includes a third refractive surface 955, which faces the third optical path refractive element 821. The seventh electron star image propagating along the ninth optical axis 919 is propagated to the third refractive surface 955, which is used to reflect and / or refract the seventh electron star image to obtain a second electron star image propagating along the tenth optical axis 9110. Figure 9 The image also shows the normal 7 at the incident point of the third-order light element 822 in the seventh electron star image.

[0188] The third beam splitting element 822 can be a beam splitter, a beam splitting prism, a holographic optical element, a polarizing beam splitter, or a combination of two or more of these elements. This application does not impose any special limitations on this aspect. Figure 8 and Figure 9 Only the case where the third beam splitter 822 is a beam splitter is shown.

[0189] One possible implementation is to use a low-reflection, high-transmittance lens for the third beam splitter 822.

[0190] In one example, the third beam splitter 822 satisfies at least one of the following: a reflectivity of less than or equal to 10% to reduce reflected light from the image source; or a transmittance of greater than or equal to 90% to enhance starlight transmission; or a thickness of less than or equal to 1 millimeter (mm) to reduce ghosting. A reflectivity below 10% and a transmittance above 90% ensure better transmission of the real starry sky image from the outside, reducing reflected light from the virtual starry sky image and meeting the needs of stargazing in low-light environments at night. A thickness of less than or equal to 1 mm for the third beam splitter effectively avoids the impact of ghosting on observation.

[0191] In another example, the third beam splitter 822 and the first beam splitter 830 constitute a second combination; the second combination satisfies at least one of the following: the reflectivity is less than or equal to 10% to reduce reflected light from the image source; or, the transmittance is greater than or equal to 90% to enhance starlight transmittance; or, the thickness of the third beam splitter 822 in the second combination is less than or equal to 1 mm to reduce ghosting.

[0192] See also Figure 8 and Figure 9 The first refractive surface 944 of the first beam splitter 830 faces the third refractive surface 955 of the third beam splitter 822. The second electronic star image propagating along the tenth optical axis 9110 propagates to the first refractive surface 944, which is used to reflect and / or refract the second electronic star image to the user 920, thereby obtaining the eighth electronic star image propagating along the eleventh optical axis 9111.

[0193] In one example, the first refractive surface 944 of the first beam-splitting element 830 is concave. This concave structure of the first refractive surface 944 inherently possesses positive refractive power, which increases the virtual image distance of the electronic starry sky image. Consequently, it effectively projects the virtual image onto a distant point, matching the optical distance to the real starry sky. Therefore, the user 920 can clearly observe both the "electronic starry sky image" and the "real starry sky image" at the same focal length without frequently adjusting the lens of their eye. This design not only makes the fusion of virtual and real scenes more natural but also eliminates visual fatigue caused by switching viewing distances, significantly improving comfort during extended observation.

[0194] Subsequently, the eighth electron star image propagating along the eleventh optical axis 9111 is transmitted to the third refraction surface 955 of the third beam splitter 822. The third refraction surface 955 is also used to transmit the eighth electron star image to the user 920, obtaining the target electron star image, so that the user 920 can see the image after the eighth electron star image has been transmitted (the target electron star image). One example is... Figure 9 As shown, the target electron star image still propagates along the eleventh optical axis 9111. In another example (not shown in the figure), the target electron star image no longer propagates along the eleventh optical axis 9111.

[0195] See also Figure 8 and Figure 9 Meanwhile, the first transmission surface 954 of the first beam splitter 830 and the third refractive surface 955 of the third beam splitter 822 can both be used to transmit a real starry sky image 930 to the user 920.

[0196] The first beam-splitting element 830 can be a beam splitter, a beam splitter prism, a holographic optical element, a polarizing beam splitter, or a combination of two or more of these elements. This application does not impose any special limitations on this aspect. Figure 8 and Figure 9 Only the case where the first beam splitter 830 is a beam splitter is shown.

[0197] Based on the above technical solution, both the target electronic star image and the real star image 930 will arrive at the user 920 simultaneously, providing an unobstructed, immersive virtual-real fusion observation experience. Furthermore, the concave design of the first folding surface 944 effectively magnifies the virtual image distance of the electronic star image and matches it with the real star image 930. The user 920 can see the clear dual effects at the same focal length without frequent lens adjustments, effectively alleviating visual fatigue and improving user comfort.

[0198] Starry sky optical observation devices may also include other structures, such as Figure 8 and Figure 9 The frame 840 shown in the figure also includes, for example, Figure 8 The display area 841 is shown in the image. See the preceding text for further details. Figures 3 to 5 The relevant descriptions in [the text] will not be elaborated upon here. Among them, Figure 8 and Figure 9 The frame 840 shown in the figure can be seen in the reference to Figures 3 to 5 The relevant description of frame 340 in the document; Figure 8 The display area 841 shown can be seen in the reference to Figure 4 The relevant description of display area 410 in the text.

[0199] This application also proposes a starry sky optical observation system. This system includes the above-described... Figures 1 to 9 The terminal device described in the embodiments, and the above Figures 2 to 9 The astronomical optical observation device described in the embodiments.

[0200] See Figure 10 , Figure 10 This application provides a starry sky optical observation system 1000. When the terminal device 1010 is installed in the mounting section (not shown in the figure), the first part 1001 of the terminal device 1010 protrudes outside the starry sky optical observation device 1020.

[0201] Regarding the installation of the terminal device 1010 on the mounting section, one possible implementation is that the terminal device 1010 is installed on the mounting section either detachably or non-detachably.

[0202] Based on the above technical solution, users can interact with the terminal device 1010 through the first part 1001 without disassembling the terminal device 1010 when it is installed in the mounting section.

[0203] The following are some examples of adjusting or generating the first electronic starry sky image. For example, adjusting the display of the first electronic starry sky image, including its position, size, angle, etc. on the screen.

[0204] Example 1: A first electronic starry sky image can be adjusted or generated by interacting with a screen portion of the first part 1001. The first part 1001 includes a first interactive area belonging to the screen of the terminal device 1010. The terminal device 1010 is configured to adjust or generate the first electronic starry sky image in response to a first operation performed on the first interactive area.

[0205] Based on the above technical solution, users can interact with the screen portion of the terminal device 1010 that is exposed outside the starry sky optical observation device 1020, thereby adjusting or generating a first electronic starry sky image. This helps to correct the image without moving or disassembling the terminal device 1010, making image correction more convenient and efficient.

[0206] The first operation may include at least one of the following: a swipe operation, a long press operation, a double-tap operation, and a single-tap operation. For example, when a user swipes a portion of the screen of the first part 1001, the position of the first electronic starry sky image on the screen is adjusted according to the user's swipe and in the direction of the swipe.

[0207] Based on the above technical solution, different adjustment effects or generation results can be achieved according to different user operations. Regarding adjustment effects, for example, long-pressing can enlarge the size of the first electronic starry sky image, sliding can adjust the position of the first electronic starry sky image, double-tapping can increase the brightness of the first electronic starry sky image, and single-clicking can decrease the brightness of the first electronic starry sky image. This is just one example, and this application embodiment does not impose any special limitations on it. Regarding generation results, such as the size of the generated first electronic starry sky image, the position of the generated first electronic starry sky image, and the brightness of the first electronic starry sky image, this application embodiment does not impose any special limitations on these aspects.

[0208] Example 2: A first electronic starscape image can be adjusted or generated by interacting with the button portion of the first part 1001. The first part 1001 includes a second interactive area with at least one button. The terminal device 1010 is then configured to adjust or generate the first electronic starscape image in response to a second operation performed on the at least one button. The button can be a physical button. Based on the above technical solution, users can adjust or generate the first electronic starscape image by operating the buttons on the outside of the star-gazing optical observation device of the terminal device. This facilitates image correction without moving or disassembling the terminal device, making image correction more efficient and convenient.

[0209] The second operation includes at least one of the following: a swipe operation, a long press operation, a double-click operation, and a single-click operation. For example, when a user clicks a button on the first part 1001, the display size of the first electronic starry sky image on the screen changes.

[0210] Based on the above technical solution, different adjustment effects or generation results can be achieved according to different user operations. Regarding adjustment effects, for example, long-pressing can enlarge the size of the first electronic starry sky image, sliding can adjust the position of the first electronic starry sky image, double-tapping can increase the brightness of the first electronic starry sky image, and single-clicking can decrease the brightness of the first electronic starry sky image. This is just one example, and this application embodiment does not impose any special limitations on it. Regarding generation results, such as the size of the generated first electronic starry sky image, the position of the generated first electronic starry sky image, and the brightness of the first electronic starry sky image, this application embodiment does not impose any special limitations on these aspects.

[0211] Example 3: A first electronic starry sky image can be adjusted or generated via voice commands. Terminal device 1010 includes a voice input module, and is configured such that the voice input module receives a voice command and, in response to the voice command, adjusts or generates the first electronic starry sky image.

[0212] Based on the above technical solution, users can directly adjust or generate the first electronic starry sky image via voice control. When adjusting or generating the first electronic starry sky image, there is no need to move or disassemble the terminal device 1010, and there are no significant restrictions on the size of the first part; the first part can be very small. In one possible implementation, the first part 1001 also includes a third interactive area, which belongs to the screen of the terminal device 1010. In this case, the terminal device 1010 is configured to activate or deactivate the voice input module in response to a third operation performed on the third interactive area.

[0213] The voice input module can be activated or deactivated through certain operations. This allows users to turn the voice input module on or off as needed, preventing the module from misinterpreting voice when it is not needed, which could lead to incorrect adjustment or generation of the First Electronics Starry Sky Image. It also allows users to conveniently activate the voice input module as required.

[0214] The third operation includes at least one of the following: swipe operation, long press operation, double-tap operation, and single-tap operation. For example, when the user double-tap any or a specific part of the screen of the first part 1001, the voice input module is activated or deactivated.

[0215] The adjustments to the first electronic starscape image in the three examples above include at least one of the following: translation, scaling, and rotation.

[0216] The above embodiments illustrate three examples of generating or adjusting a first electronic starscape image.

[0217] One possible implementation is that the terminal device includes a location acquisition module and / or an attitude acquisition module; wherein the location acquisition module is used to acquire the geographical location information of the terminal device; and the attitude acquisition module is used to acquire the three-dimensional attitude and / or azimuth angle of the terminal device.

[0218] Based on the above technical solution, the terminal device can use the location acquisition module to determine the geographical location information of the terminal device, and can also use the attitude acquisition module to determine the three-dimensional attitude and / or azimuth angle of the terminal device, which is beneficial to the generation and correction of the first electronic star image, and especially helpful to generate the virtual identifier in it.

[0219] One possible implementation is that the terminal device includes an image sensor; the first electronic star image includes a first calibration image, the first calibration image including at least one of the following: a positioning marker, the positioning marker being set at a predetermined position on the first calibration image, including at least one of the following: the center of the first calibration image, the edge of the first calibration image, the device imaging principal point of the first calibration image, used to calibrate position and orientation errors; or, image information representing the spatial layout of far-field targets acquired by the image sensor, used to calibrate position, orientation and scaling errors.

[0220] Based on the above technical solutions, positioning markers can help users calibrate position and orientation errors, improving calibration efficiency; image sensors are beneficial for calibrating position, orientation, and scaling errors, improving calibration efficiency.

[0221] Next, we will describe in detail the methods for adjusting or generating the first electronic starry sky image and related embodiments. It should be noted that the interactive methods for receiving user input operations involved in the following methods can all be implemented in conjunction with the content of the foregoing embodiments. For example, when it involves user touch operation on the screen, the method of operating the first interactive area in Example 1 can be used; similarly, when it involves user operation on physical buttons, the method of operating the second interactive area in Example 2 can be used; when it involves user voice command control, the relevant implementation method in Example 3 can be used. For the sake of brevity, the specific interactive triggering forms of various commands will not be elaborated further when describing the calibration process later.

[0222] The terminal device processes attitude information collected by the attitude sensor, location information collected by the positioning module, and real starry sky images captured by the camera to obtain a first electronic starry sky image, which is then displayed on the screen. The specific process of processing this real starry sky image to obtain the first electronic starry sky image will be described in detail later and will not be elaborated upon here.

[0223] Considering that the terminal equipment and the astronomical optical observation device in the astronomical optical observation system are physically assembled structures, and that different terminal devices have different screen specifications, and that users' mobile phones may have external accessories (such as phone cases), errors inevitably occur between the actual display area of ​​the first electronic astronomical image and the observation device. This error will cause the final observed target electronic astronomical image's azimuth to be inconsistent with the preset. Therefore, users need to perform calibration during use. Several calibration methods are introduced below.

[0224] See Figure 11 , Figure 11 This is a schematic diagram of the calibration of a starry sky optical observation system proposed in an embodiment of this application. It should be noted that the structure of the starry sky optical observation device shown is only for illustrating the calibration process and is merely a schematic representation. The diagram illustrates the x-axis, y-axis, and z-axis, as well as the origin o of the spatial coordinate system. The origin o is defined as being located at the bottom of the surface of the terminal device facing away from its screen; and, in the x-axis direction, the position of the origin o is flush with the side edge of the starry sky optical observation device.

[0225] In one example, the aforementioned error sources are mainly divided into two categories: the first category is spatial translation error, which is the translational offset of the first electronic star image in at least one of the x-axis, y-axis, and z-axis directions. The second category is spatial rotation error, which is the attitude tilt around at least one of the x-axis, y-axis, and z-axis. Among these, the rotation error around the z-axis (rotation within the image plane) can usually be calculated by the terminal device's internal attitude sensor and automatically corrected for the first electronic star image.

[0226] Two types of error sources will produce three types of errors. The first type of spatial translation error produces positional errors in the x and y directions, and the component in the z direction, which affects the object distance u between the telescopic element and the screen, produces scaling errors. In the second type of spatial rotation error, the rotation around the z axis is corrected by the internal attitude sensor, and the rotation around the x and y axes produces directional errors.

[0227] Positional errors, i.e., the propagation direction of the first electronic star image is the same as the observation direction, but the image center is not aligned, are caused by translation errors of the x-axis and y-axis; Directional errors, namely, the propagation direction of the first electronic star image is inconsistent with the observation direction, specifically manifested as the virtual image plane not being perpendicular to the line of sight. These errors are caused by rotation errors around the x-axis and y-axis. Scaling errors, also known as object distance errors, result in a mismatch between the virtual image's field of view distance and the magnification. Specifically, the entire image is scaled proportionally. These errors are caused by translation errors along the z-axis.

[0228] The following section will introduce the relevant content on calibrating these three types of errors.

[0229] The first calibration method corrects positional errors. It involves aligning the position of a first electronic star image with a fixed position on the observation device. Predetermined positions on the first electronic star image, such as the image center and / or edges and / or principal points, are marked with correction marks. The user can align the correction marks at the center and / or edges and / or principal points with the fixed position on the star optical observation device by moving the position of the first electronic star image. For example, aligning it with the display area of ​​the star optical observation device, moving the center correction mark to the center of the display area and / or moving the edge correction mark to the edge of the display area. This calibration method can correct... Figure 11 Positional errors caused by translation errors along the x-axis and y-axis.

[0230] The second calibration method corrects for position and direction errors. By adjusting the position of the first electronic star image, the observation direction and propagation direction of the target electronic star image (the virtual image ultimately presented to the user by the star optical observation device) are aligned. Under conditions of small angular approximation and without considering projection distortion, the rotation error around the x / y axes is equivalent to the overall angular offset of the virtual image's propagation direction. Adjusting the center position of the first electronic star image allows the observation direction to coincide with the propagation direction. It can be understood that the propagation direction of the target electronic star image is perpendicular to the virtual image plane (i.e., along the normal direction of the virtual image plane) and parallel to the optical axis of the terminal device's camera. Therefore, when the camera is pointed at a distant reference object in the real starry environment, the propagation direction of the target electronic star image can be determined. The terminal device receives the user's translation command and executes the translation of the first electronic star image according to the command. This translation command can be obtained through user interaction with the terminal device's screen, physical buttons, or voice control. By translating the first electronic star image, the theoretical principal point mark on the target electronic star image is made to coincide with the intersection of the observation line of sight and the virtual image plane, thereby coinciding with the true principal point of the perspective projection of the actual observation line of sight, thus completing the direction correction; wherein, the theoretical principal point mark is the ideal position mark of the projection center of the observation perspective projection (with the human eye as the focus and the target electronic star image as the imaging plane) on the target electronic star image when no error is considered, such as using a cross mark or an image mark where the camera's optical axis is directly facing the object, etc., and the observation line of sight is the direction of the human eye pointing towards a distant reference object in the real environment. One example, Figure 11 The observed virtual image 1101 before correction is shown, which has translation and rotation errors; Figure 11 The corrected virtual image 1102 is also shown. This calibration method can correct... Figure 11 The translation errors of the x-axis and y-axis can also be corrected for the rotation errors around the x-axis and y-axis.

[0231] The third calibration method corrects errors related to position, orientation, and scaling. This method relies on the focal length of the terminal device's camera. The imaging formula is 1 / f = 1 / u + 1 / v, where f is the focal length, u is the object distance, and v is the image distance. Figure 11The z-axis translation error affects the object distance u, causing a change in the image distance v, which in turn affects the size of the virtual image (hereinafter referred to as the observation virtual image) of the target electronic starscape image (the virtual image ultimately presented to the user by the starscape optical observation device). By acquiring image information characterizing the spatial layout of far-field targets (the information includes locatable spatial features, such as far-field real-world images or celestial markers, celestial lines, etc.), the user can adjust the scaling and position of the image information of the spatial layout of far-field targets on the screen to completely superimpose its perspective projection onto the real object on the virtual image plane. Z-axis translation error correction is performed according to the scaling ratio, and x-axis and y-axis translation error correction is performed according to the movement position. Figure 12 This is a schematic diagram illustrating the calibration of another stellar optical observation system proposed in an embodiment of this application. One example... Figure 12 The observed virtual image 1201 before correction is shown, which contains errors; Figure 12 The corrected virtual image 1202 and the real object 1203 are also shown. The cross mark on the uncorrected virtual image 1201 is a camera optical axis marker, designed to facilitate user alignment. This calibration method can correct... Figure 11 The translation error along the z-axis. Furthermore, by moving the object's perspective projection, the observation method of the principal point of the observed virtual image projection is made consistent with the direction of the first electronic starscape image, thus correcting... Figure 11 Translation errors of the x-axis and y-axis, and corrections for rotation errors around the x-axis and y-axis.

[0232] Figure 13 This is a schematic diagram illustrating the calibration of a stellar optical observation method according to an embodiment of this application. See also... Figure 13 When user 1310 is observing, W_object: the actual size of the observed object.

[0233] D_cam: The distance from the observed object to the camera's focal point.

[0234] D_eye: The distance from the observed object to the human eye.

[0235] F: Screen imaging equivalent focal length, which is the focal length of the image captured by the camera and displayed on the screen.

[0236] u: Object distance from the mobile phone screen to the remote element (i.e., the distance between the plane where the screen is located and the remote element).

[0237] f: Focal length of the zoom element.

[0238] Δu: Object distance error between the mobile phone screen and the remote component.

[0239] d: Increase the distance between the device and the human eye (i.e., exit pupil distance).

[0240] W_screen: The size of the image of the object on the phone screen.

[0241] W_camInvscreen: The size of the object captured by the camera and displayed on the virtual image surface via the virtual image display chain.

[0242] W_vscreen: The size of the perspective projection of an object onto a virtual image plane as observed by the human eye.

[0243] R: The ratio of the perspective projection size of the object on the virtual image plane to the size of the image presented on the virtual image plane by the camera, i.e., the scaling matching factor.

[0244] ΔR: The amount of change in the scaling matching coefficient, i.e., ΔR = R R0, where R0 is the preset theoretical scaling factor.

[0245] It should be noted that, Figure 13 The focal point of a telephoto element in a astronomical optical observation system refers to the focal point of the telephoto element itself, or the equivalent focal point of the telephoto element in the astronomical optical observation system. For example, when there is only one telephoto element in the astronomical optical observation system, the focal point of the telephoto element is the focal point of that single telephoto element; when the astronomical optical observation system has two or more telephoto elements, the focal point of the telephoto element is the equivalent focal point of those two or more telephoto elements, or the focal point of the two or more telephoto elements as a combination.

[0246] Step 1: Based on the principle of perspective projection by the camera, the image size of the object on the phone screen is: W_screen = (F / D_cam) · W_object(1) Where F is the equivalent focal length for screen imaging. The above relationship can also be expressed in terms of angular dimensions as W_screen = F·α, where α is the angular dimension of the object relative to the camera; the two are equivalent.

[0247] Step 2: The image of the photographed object on the phone screen is projected onto a virtual image plane via a zoom element. Considering the screen distance error Δu, the size of the object on the virtual image plane is: W_camInvscreen = W_screen · f / (Δu + u f)(2) Where f / (Δu+u) f) is the magnification factor by which the screen is magnified into a virtual image by the remote element.

[0248] Step 3: When the human eye observes an object through a virtual image plane, the size of the object projected onto the virtual image plane by the eye is: W_vscreen = ( f ·u / (Δu+u f) + d ) / D_eye · W_object(3) where f ·u / (Δu+u f) is the distance from the virtual image to the telephoto element. Adding d gives the distance from the eye to the virtual image surface; the absolute value is taken when the virtual image is established.

[0249] Step 4: Substitute equations (1) and (2) into equation (3) to obtain the scaling matching coefficients: R = W_vscreen / W_camInvscreen = ( D_cam / D_eye ) · [ u / F + (d / F) · (Δu / f + u / f 1) ](4) When the observed object is a far-field target, D_cam ≈ D_eye, and the ratio of the two approaches 1. Equation (4) simplifies to: R = u / F + (d / F)·(Δu / f + u / f 1)(5) As can be seen from equation (5), the change in scaling matching coefficient caused by screen distance error Δu is: ΔR = R(Δu) R(0) = (d / F)·(Δu / f)(6) Δu=ΔR F f / d, that is, the accurate error can be solved by scaling the matching coefficient ΔR, the preset exit pupil distance d, the screen imaging equivalent focal length F, and the focal length f of the telephoto element.

[0250] At the same time, the equivalent focal length of the screen image is corrected to Fnew=R. F = u + (d)·(Δu / f + u / f) 1) At that time, W_camInvscreen=W_screen · f / (Δu + u f) = (Fnew / D_cam) · W_object f / (Δu + u f) = ((u + (d)·(Δu / f + u / f 1) ) / D_cam) · W_object f / (Δu + u f) ≈ (f·u / (Δu+u) f) + d) / D_eye · W_object =W_vscreen That is, the correction is complete.

[0251] It should be noted that the focal length of the telephoto element here refers to the focal length of the telephoto element in the astronomical optical observation system, or the equivalent focal length of the telephoto element in the astronomical optical observation system. For example, when there is only one telephoto element in the astronomical optical observation system, the focal length of the telephoto element is the focal length of that single telephoto element; when the astronomical optical observation system has two or more telephoto elements, the focal length of the telephoto element is the equivalent focal length of those two or more telephoto elements, or the focal length of the telephoto element as a combination of the two or more telephoto elements.

[0252] Regarding how the terminal device specifically captures real starry sky images through a camera, processes these real starry sky images, and adjusts the first electronic starry sky image, several possible implementation methods will be introduced below.

[0253] This application also proposes an image processing method for astronomical optical observation. The astronomical optical observation system includes the terminal equipment and astronomical optical observation device described above, wherein the terminal equipment is installed in the mounting section.

[0254] Figure 14 This application proposes an image processing method for astronomical optical observation. This method is executed by a terminal device in an astronomical optical observation system. It should be noted that after the first electronic star image has been displayed, this method can ensure that virtual markers or virtual images in the first electronic star image are aligned with the real starry sky, improving the user experience. Based on the above technical solution, by performing star point detection on the original image, celestial targets in the image can be automatically identified without the user manually selecting calibration points; by calculating the pixel focal length, the camera focal length deviation can be calibrated; by calculating the correction rotation matrix, the attitude measurement error of the attitude acquisition module can be compensated, ensuring precise alignment of the star map with the real starry sky, thereby improving imaging accuracy and image quality. Simultaneously, the above calibration process is completed automatically by the system, requiring no astronomical observation knowledge or calibration experience from the user, lowering the barrier to entry, reducing the time spent on calibration operations, and improving the user experience. This image processing method for astronomical optical observation includes the following steps: S1410, the terminal device's camera captures the raw image.

[0255] One possible implementation involves the exposure control module adjusting the camera's exposure parameters (sensitivity and exposure time) based on the grayscale feedback of the original image, ensuring clear and unsaturated star-like images and reliable detection. Specifically, exposure is increased when the grayscale is low and decreased when the grayscale is high.

[0256] S1420: Perform star detection on the original image to obtain a set of detected star points. The set of detected star points includes multiple detected star points, each with corresponding coordinates.

[0257] One possible example is that the star detection process includes the following steps: 1. Background estimation processing. The original image is divided into an M×M grid, and median / mean background estimation is performed using an m×m convolutional box. A smooth background is obtained through spline interpolation, and the global root mean square noise (RMS) is calculated. Here, M and m are positive integers. This application does not impose special restrictions on the values ​​of M and m; in one example, M is 64 and m is 3. This example will be used to continue the explanation.

[0258] 2. Background subtraction. The smoothed background is subtracted from the original image pixel by pixel to obtain a residual image containing only stars and noise.

[0259] 3. Star Point Extraction. Using h times the global root mean square noise as the relative detection threshold, a 3×3 matched filter kernel is used to convolve the residual image. Connected regions exceeding the threshold are marked as candidate star points. Overlapping regions are de-aliased multiple times using 32 layers of intensity thresholds. Here, h is a positive integer, and this application does not impose special restrictions on the value of h; one example is a value of 5.

[0260] 4. Morphological filtering. A point source criterion is applied to candidate star points—candidate star points with pixel areas smaller than a first threshold or greater than a second threshold are removed, as are candidate star points with an ellipticity (major axis / minor axis ratio) greater than a first ratio (these are often galaxies or star trails). This application does not impose special restrictions on the values ​​of the first threshold, second threshold, and first ratio. One example is a first threshold of 3, 4, or 5. Another example is a second threshold of 100, 150, or 200. A first ratio is 2, 3, or 7.

[0261] 5. Sort and select the best. Sort the candidate stars in descending order of their integral flux, and select the top few (e.g., 15) as the detection star set for subsequent matching.

[0262] S1430, the detected star point set and the star catalog are compared to obtain the final matching pair set. The star catalog can be received by the terminal device or stored internally. The final matching pair set includes one or more matching pairs. Each star catalog contains multiple star points, each star point corresponding to a coordinate. Each matching pair records the coordinates of the detected star point and the coordinates of the matched star point.

[0263] One possible implementation involves normalizing the coordinates corresponding to the detected star point set and the coordinates corresponding to the star catalog, respectively, to obtain their normalized coordinates. These normalized coordinates are then compared to obtain one or more pairs of matching stars. Based on this scheme, since the normalized coordinates remain unchanged during image rotation and scaling, even if device orientation errors cause some rotation and scale deviations between the projection of the star catalog coordinates onto the image and the actual star points, it does not affect the comparison results.

[0264] One possible example is the process of comparing the detected star point set with the star catalog to obtain P pairs of matching stars, which includes the following steps: 1. Selecting Reference Pairs. Select Q stars from the detected star point set, ranked from highest to lowest brightness, where Q is a positive integer greater than 1. This set of Q detected stars will be referred to as the matching star point set. Randomly select two stars from these Q detected stars to form reference pair combinations. A total of C(Q, 2) reference pair combinations can be formed from the Q detected stars. C represents the number of combinations. The baseline is formed by connecting the coordinates of the two detected stars in each reference pair, and the baseline pixel length is recorded as 1. Next, the following operation is performed on each reference pair. For ease of description, we take one reference pair as an example, assuming that the pair consists of star A and star B, forming baseline AB.

[0265] In this embodiment of the application, no special restrictions are placed on the value of Q. Q can be 2, 3, 6, 9, etc.

[0266] Q can be adjusted according to requirements. The larger the Q value, the stronger the anti-interference ability; the smaller the Q value, the faster the calculation speed.

[0267] 2. Normalized coordinate transformation of the matching star point set. For each detected star point P (coordinate position (px, py)) in the matching star point set, calculate its normalized coordinates (newX, newY) relative to the baseline AB according to the following formula, which will be referred to as the normalized coordinates of the matching star point set: newX = (px·dx + py·dy) / L² newY = (py·dx - px·dy) / L² Where L is the length of baseline AB, and (dx, dy) are the components of baseline vector BA. newX represents the projected length of P along the baseline direction divided by the square of the baseline length, and newY represents the component of P perpendicular to the baseline direction divided by the square of the baseline length. The "·" sign indicates multiplication. This transformation is invariant to image rotation, translation, and scaling.

[0268] It should be noted that since this step is introduced using a specific reference pair as an example, the normalized coordinates of the matching star set here are specific to that reference pair. The normalized coordinates of the matching star set may be different for different reference pairs. The normalized coordinates of the matching star set calculated for different reference pairs are recorded separately and independently, and will not be mixed together.

[0269] 3. Star Catalogue Normalized Coordinate Transformation. Similar to the previous step, select two star points from the star catalogue in descending order of brightness. Assume that the selected star C and star D form the baseline CD, with a pixel length of 1. For each star point R (coordinate position (rx, ry)) in the candidate sky region, calculate its normalized coordinates (newX_r, newY_r) relative to the baseline CD using the following formula. These will be referred to as the star catalogue normalized coordinates: newX_r=(rx·dx_r+ry·dy_r) / L_r² newY_r=(ry·dx_r-rx·dy_r) / L_r² Where L_r is the length of the baseline CD, and (dx_r, dy_r) are the components of the baseline vector DC. newX_r represents the projected length of R along the baseline direction divided by the square of the baseline length, and newY_r represents the component of R in the perpendicular direction to the baseline divided by the square of the baseline length. The "·" sign indicates multiplication. This transformation is invariant to image rotation, translation, and scaling.

[0270] One example is the bright star catalogue (BSC).

[0271] 4. Compare the matching star point set with the star catalog to determine the matching pair set. Compare the normalized coordinates of all stars in the matching star point set with the normalized coordinates of stars in the star catalog. Find one or more sets of stars whose first coordinate component (newX) differs from the first star catalog coordinate component (newX_r) by a factor less than or equal to the third threshold, and whose second coordinate component (newY) differs from the second star catalog coordinate component (newY_r) by a factor less than or equal to the fourth threshold. These sets of stars constitute the matching pair set. For example, for star point a1 in the matching star point set, the first coordinate component (newX) of the normalized coordinates is 1, and the second coordinate component (newY) is 1; for star point a2 in the star catalog, the first star catalog coordinate component (newX_r) is 1.1, and the second star catalog coordinate component (newY_r) is 0.99; the third threshold is 1, and the fourth threshold is 0.8. At this point, when star point a1 and star point a2 are compared, the difference between the first coordinate component (newX) and the second coordinate component (newY) is 0.1, which is less than the third threshold; the difference between the second coordinate component (newY) and the second star table coordinate component (newY_r) is 0.01, which is less than the fourth threshold. Therefore, star point a1 and star point a2 are successfully matched and are included in the matching pair set.

[0272] This application does not impose special restrictions on the values ​​of the third and fourth thresholds, and the third and fourth thresholds may be equal or unequal. Both the third and fourth thresholds can be set and adjusted as needed, or can be set by the user according to their requirements.

[0273] One possible implementation involves using sorting and linear scanning to reduce the power consumption and complexity of the matching algorithm, while accelerating the matching speed. The normalized coordinates of the matching star point set are sorted by the first coordinate component (newX), and the normalized coordinates of the star catalog are sorted by the first star catalog coordinate component (newX_r). A linear scan comparison of the two sorted lists is performed using two cursors. The sorting can be ascending or descending, and both sorts should be consistent, i.e., both should be descending or both should be ascending. When the difference between the first coordinate component of the normalized coordinates of the matching star point set and the first star catalog coordinate component of the normalized coordinates of the star catalog is less than or equal to a third threshold, the difference between the second coordinate component (newY) of the normalized coordinates of the matching star point set and the second star catalog coordinate component (newY_r) of the normalized coordinates of the star catalog is further determined. If it is less than or equal to a fourth threshold, the star point in the matching star point set and the star point in the star catalog are recorded as a match; otherwise, the corresponding cursor is advanced according to the coordinate size until a list is traversed. In this way, a set of matching pairs is obtained in approximately nearest neighbor time.

[0274] It should be noted that since this step is described using a specific baseline pair as an example, the set of matching pairs here refers to that baseline pair. The sets of matching pairs for different baseline pair combinations are also recorded separately and independently; they are not mixed together, and successfully matched points under different baseline pair conditions are not recorded in the same set. In other words, each baseline pair combination corresponds to its own set of matching pairs.

[0275] Furthermore, when there are no successfully matched star point pairs, the set of matching pairs can be an empty set, and this application embodiment does not impose any special restrictions on this.

[0276] 5. Matching Determination. This step can be skipped when Q is 2. We will now describe the case where Q is greater than 2. In step 4 above, each baseline pair combination yields a matching pair set, resulting in a total of C(Q, 2) matching pair sets. Comparing the number of stars recorded in each matching pair set, the set with the largest number of stars (greater than the reference match number) is selected as the final matching pair set. The baseline pair combination corresponding to this final matching pair set is the final baseline pair combination, and the corresponding baseline is the final baseline. Several special cases are described below.

[0277] Based on the above technical solution, by matching multiple benchmark pairs separately and then selecting the optimal matching pair set from the multiple matching pair sets corresponding to the multiple benchmark pairs, the accuracy of matching can be improved.

[0278] In one scenario, if there is no matching pair with the largest number of stars (e.g., all pairs have the same number of stars), then any matching pair set is selected as the final matching pair set, and the number of stars in this set must be greater than or equal to the reference number of matches. In other words, if there is no set with the largest number of stars, and the number of stars in any matching pair set is less than the reference number of matches, then the matching is considered a failure.

[0279] In another scenario, if the set of matching pairs with the largest number of stars has a number of stars that is less than or equal to the reference number of matches, the match is also considered to have failed.

[0280] In either of the above two cases, if the match is considered to have failed, the Q stars in the detection point set will be deleted, and steps 1 to 5 will be repeated.

[0281] The reference matching number can be 5, 6, 7, 11, etc., and this application embodiment does not impose any special limitation on it.

[0282] In steps 2 and 3, the normalized coordinate transformation can be replaced by schemes such as triangular angle ratio or four-star cross ratio, and this application embodiment does not impose special restrictions on this. In step 4, the dual-cursor linear scan (sorting and linear scan matching) can be replaced by approximate nearest neighbor retrieval methods such as k-dimensional tree (KD-tree) or grid hashing, and this application embodiment does not impose special restrictions on this.

[0283] Based on the above technical solution, the detected star points can be compared and matched with real star points to obtain a set of successfully matched star points.

[0284] In one example, to narrow down the search space for star catalog matching in astronomical optical observations, the following pre-screening steps can be performed before the above-mentioned star point set and star catalog comparison process (steps 1 to 5): 1. Offline pre-indexing. The celestial sphere is divided into several regions of approximately 45° × 45°. For each region, the set of star identifiers contained within it is pre-calculated, generating a celestial region index table stored locally on the mobile device. Each celestial region index corresponds to a set of star identifiers contained within that region.

[0285] 2. Pre-screening. Based on the current initial sensor attitude, the star points in each sky region index are projected onto the field of view corresponding to the starry sky image, and the depth of each star point is calculated (the component of the star point's direction vector in front of the camera, inverted to obtain the depth value; a depth greater than 0 indicates that it is visible in front of the camera). Only candidate sky regions in the sky region index where all star points are projected within a certain angle in front of the camera (e.g., depth ≥ 0.8) are retained as the matching search space.

[0286] 3. The S1430's detection star point set and star catalog comparison are only performed within the reserved candidate sky areas, avoiding full star catalog scanning and significantly reducing the computational load.

[0287] Another example is that the image processing method for astronomical optical observation is to reduce the search space for star catalog matching. Before the above-mentioned star point set and star catalog comparison process (steps 1 to 5), spatial partitioning methods such as dynamic grid partitioning and local sensitive hashing can also be used. This application embodiment does not make any special limitations on this.

[0288] Based on the above technical solutions, the efficiency and accuracy of matching can be improved, the time spent on matching can be reduced, and thus the user experience can be improved.

[0289] S1440, based on the final set of matching pairs, calculates the pixel focal length.

[0290] It should be noted that pixel focal length is the physical focal length (millimeters (mm)) / the physical size of a single pixel (mm / pixel). This is a commonly used value in camera calibration. This value measures the focal length in pixels and is unrelated to the actual size of the pixels.

[0291] Regarding how to record each pair of matching stars in the final set of matching pairs, this application embodiment does not impose any special restrictions on this. For ease of description, an example is given here: Matching Star (I q J q ). Where, represents the q-th matching star; I q It is the star point in the final matching star point set, I q It can be fully represented as (qewX, qewY); J q It is a star point in the star catalog, J q This can be fully represented as (qewX_r, qewY_r). Here, it's understandable that the star point I... q And star point J q Match successful. We will continue with this example.

[0292] One possible example is that calculating the corresponding pixel focal length for each pair of matching stars includes the following steps: 1. Randomly select 2 pairs of matching stars from the final set of matching pairs, that is, randomly select 2 pairs of matching stars from q pairs of matching stars. These q pairs of matching stars can form C(q, 2) pixel focal length calculation combinations. As you can understand, C represents the number of combinations. Each pixel focal length calculation combination undergoes steps 2 and 3 as described below, ultimately yielding the calculated pixel focal length from the calculated C(q,2) pixel focal length estimates, which will be explained in detail later. For ease of description, we will take one pixel focal length calculation combination as an example, assuming that the combination includes matching stars (i1, j1) and matching stars (i2, j2).

[0293] 2. Calculate the true angular distance. Take I1 and I2 and their right ascension and declination (RA1, DEC1) and (RA2, DEC2) of the corresponding star points J1 and J2 in the star catalog. Calculate the cosine of the true angular distance between the two stars using the spherical trigonometry formula: cosθ1 = sin(DEC1)·sin(DEC2) + cos(DEC1)·cos(DEC2)·cos(Δα).

[0294] Where DEC1 is the declination of star point J1, DEC2 is the declination of star point J2, Δα is the difference in right ascension between star points J1 and J2, i.e., (RA1 - RA2), and cos() is the cosine function. It should be noted that the difference in right ascension here is the absolute value of the difference.

[0295] 3. Using the pinhole imaging closure relationship, establish a quadratic equation about the square of the focal length f², and solve for the estimated focal length f (taking the effective root f>10 pixels). The specific formula derivation process is introduced below.

[0296] Construct the direction vector of the camera coordinate system for the star point in the image. Let the pixel position of the star point be (px, py), the principal point be (cx, cy), and the pixel focal length be f. Construct the direction vector of the camera coordinate system d = (x, -y, -f), where x = (px - cx) and y = (py - cy) are the pixel offsets of the image coordinates relative to the principal point. Since the image coordinate system y increases downward (screen pixels) and the camera coordinate system y increases upward, the two have opposite signs and are inverted. The z direction of the camera coordinate system is the direction towards the screen (towards the user), and the inversion corresponds to the forward direction of the camera.

[0297] For I1 and I2, the direction vectors of the two stars in the camera's coordinate system (camera facing...) Z): d1 = (x1, -y1, -f), d2= (x2, -y2, -f).

[0298] Closure relation: True angular distance = Angle between two direction vectors of the image, take the cosine: cosθ1= (d1·d2) / (|d1|·|d2|); Therefore, cosθ1 = (x1x2 + y1y2 + f²) / ( · Let A = x1² + y1², B = x2² + y2², and C = x1x2 + y1y2, then: cosθ1= (C + f²) / .

[0299] Squaring both sides of the above equation, expanding, rearranging and combining terms, f 4 Leave the term as is, and rearrange it into a quadratic expression according to u = f²: sin²θ1·u² [cos²θ1·(A+B) 2C]·u + [C² [cos²θ₁·AB] = 0. Final formula (quadratic equation + root finding + root selection): a = sin²θ₁ = 1 cos²θ1,b = [ cos²θ1·(x1²+y1²+x2²+y2²) 2(x1x2+y1y2)], c = (x1x2+y1y2)² cos²θ1·(x1²+y1²)(x2²+y2²), u = f² = [ cos²θ1(A+B) 2C ± ] / (2 sin²θ1). Let f = Valid roots of >10 pixels.

[0300] It should be noted that since this step is described using a single pixel focal length calculation combination as an example, the estimated pixel focal length value *f* here refers to that specific pixel focal length calculation combination. Different pixel focal length calculation combinations may yield the same or different estimated pixel focal length values, and these values ​​are recorded separately and independently, without being mixed together. In other words, each pixel focal length calculation combination corresponds to its own estimated focal length value.

[0301] 4. Obtain the calculated pixel focal length from the pixel focal length estimates obtained from all pixel focal length calculation combinations. Remove any pixel focal length estimates that exceed the first range. Take the median of the remaining values ​​or statistically aggregate the remaining one or more pixel focal length estimates to obtain the optimal estimated pixel focal length, which is used for subsequent attitude correction, camera projection, and assembly calibration.

[0302] Based on the above technical solution, determining the optimal pixel focal length estimate from multiple pixel focal length estimates can improve the accuracy of pixel focal length and enhance the effect of pixel focal length correction.

[0303] One possible example is that the first range is related to the preset calibration pixel focal length of the terminal device camera, for example, greater than or equal to 0.8 times the preset calibration pixel focal length and less than or equal to 2 times the preset calibration pixel focal length. This application embodiment does not impose any special restrictions on this.

[0304] Next, we will introduce a special case: if all the obtained pixel focal length estimates are excluded from the first range, then the pixel focal length will not be calculated, and the matching will be judged as a failure.

[0305] 5. Replace the camera's calibrated pixel focal length with the calculated pixel focal length. This step assumes that a calculated pixel focal length was obtained in the previous step (step 4). Replacing the camera's calibrated pixel focal length with the calculated pixel focal length means overwriting the calibrated pixel focal length with the calculated pixel focal length. When performing step S1440 again, the current calculated pixel focal length will again overwrite the current calibrated pixel focal length (which may have already been overwritten by the calculated pixel focal length). From now on, we will use the example where the calibrated pixel focal length has been overwritten / replaced by the calculated pixel focal length at least once, and will refer to the current calibrated pixel focal length or the latest calculated pixel focal length as "calculated pixel focal length".

[0306] Based on the above technical solution, the camera's calibrated focal length is continuously corrected by calculating the pixel focal length, resulting in a corrected focal length. This corrected focal length is then used to correct the rotation matrix, ensuring that the virtual image in the first electronic starry sky image is always aligned with the real starry sky, thus improving the user experience. This also overcomes the error between the calibrated focal length and the actual focal length of the terminal device's camera.

[0307] One possible implementation is that the image processing method for astronomical optical observations can replace S1440 with a global estimate optimized by nonlinear optimization.

[0308] S1450, based on the rotation matrix of the final matching set and the initial pose of the terminal device, the corrected rotation matrix is ​​obtained.

[0309] One possible example is that attitude error correction includes the following steps: 1. Construct the direction vector. For each matching pair (I) in the final set of matching pairs... q J q (For each of the q matching pairs), calculate the matching star I. q The corresponding fuselage coordinate system direction vector d = (x, -y, -f) is normalized to dPhone_normalized = dPhone / |dPhone|. The fuselage coordinate system unit direction vector dPhone_normalized is then rotated using the sensor's initial attitude to obtain the world frame (East-North-Sky) direction vector b. For example, the initial attitude includes a rotation matrix (fuselage coordinate system -> world coordinate system). Multiplying the rotation matrix on the left by dPhone_normalized yields the world system direction vector b of the star points in the image; then matching the star J... q The right ascension and declination of the (star catalog) are combined with time and latitude / longitude conversion to obtain the world coordinate system direction vector r.

[0310] 2. Accumulate the correlation matrix. For all matching pairs, sum the outer product of b and r to obtain the correlation matrix. Among them, r This indicates the transpose of r. This means summing the parentheses calculated for each of the q matching pairs, or performing a summation calculation.

[0311] 3. Singular Value Decomposition. Perform singular value decomposition on the correlation matrix B: B = UΣV Construct the corrected rotation matrix R = U·V When det(R) < 0 (called a left-handed system), use R = U·diag(1,1,-1)·V Perform chiral correction. B = UΣV In this matrix, U is a matrix variable (left singular vector matrix, orthogonal matrix); Σ is a matrix variable (diagonal matrix, diagonal values ​​are singular values, off-diagonal values ​​are 0); V is a matrix variable (right singular vector matrix); V This indicates that the inverse of matrix V is calculated, or the transpose of matrix V is calculated. The rotation matrix R = U·V In this code, R is a matrix variable (rotation matrix, the result to be calculated), det() is the determinant function, diag() is the diagonal matrix constructor that places the numbers in parentheses on the diagonal of the matrix, and "·" means matrix multiplication.

[0312] 4. Mismatch Rejection and Attitude Correction. Calculate the rotation angle corresponding to the corrected rotation matrix R, and find the trace of the corrected matrix (sum of the main diagonals): trace = 1 + 2cosθ², then solve for θ² = arccos((trace...)). 1) / 2); When the rotation angle θ2 is greater than a predetermined angle threshold (such as 20° or 15°, etc., this embodiment does not specifically limit this), it is determined to be a mismatch, and the application of the corrected rotation matrix is ​​rejected or the current rotation matrix is ​​not corrected to prevent incorrect correction from contaminating the attitude. When the rotation angle is less than or equal to the predetermined angle threshold, the attitude is corrected, and the corrected first effective attitude is obtained. =Corrected rotation matrix transpose R ×Initial attitude matrix The effect is to correct the rotation angle relative to the fixed world coordinate system.

[0313] One possible implementation is that the attitude error correction method can also calculate a correction rotation matrix between vectors in the fuselage coordinate system for each pair of matches, with the correction rotation angle fixed relative to the fuselage coordinate system.

[0314] It should be noted that both attitude error correction methods are accurate in correcting the attitude at the time of image capture, but they are not based on the separation error of the world coordinate system or the body coordinate system. This will cause the correction rotation matrix to change after the attitude or environment changes, and it will no longer be applicable to subsequent frames, requiring matching and correction again.

[0315] One possible implementation is that the image processing method for astronomical optical observations can replace S1450 with a quaternion method to solve the Wahba problem, such as the quaternion estimator (QUEST) algorithm.

[0316] Based on the above technical solution, the initial attitude error of the sensor is continuously corrected by recognizing the attitude, resulting in a corrected attitude for use by the rendering module. This ensures that the virtual image in the first electronic starry sky image is always aligned with the real starry sky, improving the user experience. Furthermore, this overcomes the error between the initial attitude matrix of the terminal device and the actual attitude matrix of the terminal device.

[0317] One possible implementation is that after processing S1450 in the image processing method for astronomical optical observation, further processing S1460 can be performed. Compared to S1450, which can only correct one type of error in the attitude rotation matrix caused by the external environment or the aircraft carrier, S1460 can simultaneously correct both the attitude rotation matrix errors caused by the external environment and those caused by the mobile phone itself. The processing procedure of S1460 will be described in detail below.

[0318] S1460, two types of attitude error correction.

[0319] Attitude rotation matrix error includes two types of errors: carrier-fixed error and world-fixed error. Carrier fixation error: including sensor installation angle deviation, equivalent distortion caused by soft magnetic materials, etc. These errors are fixed to the device body and are represented as a constant rotation matrix in the carrier coordinate system.

[0320] World fixed errors: These include environmental deviations such as local magnetic declination, geomagnetic anomalies, and local hard iron interference. These errors remain constant in the navigation coordinate system (world coordinate system) and do not change with the attitude of the equipment.

[0321] The two errors cannot be separated or corrected simultaneously using only S1450.

[0322] One possible example is to use sensors from the terminal device, such as an inertial measurement unit (IMU) containing a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, and combine them with a star map matching algorithm to achieve attitude calculation.

[0323] This embodiment provides a two-round progressive error separation and calibration method, which uses the errors obtained from star map matching under different attitudes as "external observation anchors" to decompose the errors.

[0324] 1. Establish the error model. Assuming the system simultaneously has both carrier-based fixed error and global fixed error, the IMU output satisfies:

[0325] in, The attitude rotation array provided by the system. The constant error in the carrier coordinate system (3 DOF); For constant environmental bias (3 DOF rotation) in the world coordinate system; This represents the true, error-free posture.

[0326] Define the error matrix. For the i-th observation, the attitude is corrected by star map matching. Original pose of IMU in the same frame Calculate the error matrix:

[0327] in This refers to the constant installation error in the carrier coordinate system; This represents a constant environmental deviation in the world coordinate system. This refers to the corrected effective attitude in S1450. . Similarity transformation By moving the error from the world frame to the current fuselage frame, the two types of errors fall into the same coordinate system, allowing for superposition and separation.

[0328] First round: Small-angle linear least squares fast estimation: For consumer-grade handheld devices, installation errors and magnetic declination are typically small under normal conditions. We employ Lie algebra linearization for fast initial value estimation. Let:

[0329] in Let be the fixed error vector of the carrier in the carrier coordinate system. Environmental deviation vector in the world coordinate system This is the cross product matrix operator.

[0330] The logarithm of E is mapped to the Lie algebra so(3):

[0331] in For the first Error vector of the second observation For the first Effective attitude for this observation. Data acquisition. Observation data under different attitudes were used to construct a system of linear equations. Unknown quantity :

[0332] in, It is a 3x3 identity matrix. For the first The effective attitude of the second observation.

[0333] Solve using the least squares method :

[0334] Observability conditions: Design matrix The rank must be 6. If and only if the stated... The matrix is ​​full rank when the observation includes at least three different attitudes and the carrier exhibits attitude changes in at least two independent rotational degrees of freedom. Specifically, the observed attitudes are required to cover different pitch and roll angles, avoiding all observations changing only around a single rotation axis.

[0335] Calculate the residual vector from the first round of estimation:

[0336] And calculate the residual norm. With respect to the preset sensor noise threshold (For example, comparing the product of the standard deviation of the heading noise and the number of data points in a star chart): like This indicates that the current environmental deviation is small, the small angle assumption holds, and the first round of solutions is accepted. and The calibration is now complete, serving as the final calibration result. like This indicates the presence of a large-angle environmental deviation (such as localized strong magnetic interference causing a heading deviation exceeding [a certain value]). The small-angle linear assumption fails, and the second round of precise optimization begins.

[0337] In one simplified implementation, when the world fixed error only includes rotation about the celestial direction, Degenerates into a single-parameter heading angle ,Right now In this case, the linearized model described above Degenerate into ,in Let be the celestial unit vector in the world coordinate system; in this simplified case, the solution can be obtained using observation data from at least two different attitudes.

[0338] Second round: Precise exponential mapping: When the residuals from the first round are too large, a Lie group is used based on the initial values ​​obtained from the first round of linear estimation. Exact solution for nonlinear optimization on the above. The two types of error are parameterized as Lie algebras and exponentially mapped:

[0339] Construct a nonlinear least squares problem with the error matrix residuals as the cost function:

[0340] Initial value selection: based on the value obtained in the first round. and Construct initial values:

[0341] Iterative update: at the first In the next iteration, the residual Jacobian matrix under the current estimate is calculated, and the small Lie algebra update is solved. With Δ (Meeting the small angle condition), and updating the parameters:

[0342]

[0343] Iterate until the residual converges or the maximum number of iterations is reached, finally obtaining the accurate result. and .

[0344] Transform Lie algebras (rotation vectors) back into rotation matrices using an exponential mapping (matrix exponentiation):

[0345] in, and To obtain the estimated value, This refers to the cross product matrix operator defined earlier.

[0346] The carrier fixed error rotation matrix is ​​obtained through two rounds of progressive calibration. Rotation matrix with world fixed error Then, these can be applied to the original IMU attitude to obtain the corrected second effective attitude. Precisely correct posture for projection in areas or gaps where recognition is not possible:

[0347] in Left multiplication eliminates the fixed error of the world system (magnetic declination, acting on the world system). Right-hand multiplication eliminates carrier fixation error (installation deviation, acting on the fuselage system). The rendering module uses... As a projection pose, it aligns the AR star chart with the real starry sky.

[0348] The following describes how to perform S1460 processing after S1450 processing. An example is provided, including the following steps: Step S1: During the initial identification, after processing in S1450, the system records the following information: the current location of the terminal device; the star map matching posture. IMU output attitude ; Calculate the error vector .

[0349] Step S2: Perform the following two rounds of progressive calibration.

[0350] First round: Within the position deviation threshold (to avoid changes in magnetic declination with geographical location), find multiple sets of observation frames covering different pitch and roll angles, substitute them into the above linear least squares equation, and quickly solve to obtain the initial value. and And calculate the residuals.

[0351] Residual check: If the residual is less than the preset threshold, accept the first round result directly; if the residual is greater than or equal to the threshold, execute the second round.

[0352] Second round: Using the results of the first round as initial values, construct a complete exponential mapping nonlinear optimization problem and iteratively solve for the exact solution. and .

[0353] Finally obtained and .in, The fixed error of the carrier is written into the non-volatile memory of the device as a constant; The current geographical location is a fixed global error, used as the initial magnetic declination compensation value.

[0354] Step S3: Validation.

[0355] After compensating using the calibration results, the system performs star map matching verification again. For the newly completed matches, [the system will...]. and the corrected effective attitude in S1450 Compare the results; if the residual heading error is less than the threshold (e.g.) If the calibration is successful, then the calibration is passed.

[0356] Step S4: Continuous recognition. In the recognition at different locations, It allows for persistence and reuse, further reducing the need for observation frames in the early stages of observation. It also introduces a forgetting factor, gradually reducing the weight of older observations and increasing the weight of newer observations, thus continuously optimizing the process. .

[0357] Based on the above technical solution, the S1460 achieves mathematical separation between carrier-fixed errors and world-fixed errors. Through a progressive strategy of fast estimation using first-round small-angle linear least squares and precise optimization using second-round full exponential mapping nonlinearity, the two types of errors are decoupled and calibrated, balancing computational efficiency and robustness over large angles. Specifically, the first-round linear least squares approach converges quickly, suitable for typical small-deviation scenarios; the second-round nonlinear precise optimization provides a fallback for large-deviation scenarios, balancing efficiency and robustness. Utilizing the real starry sky as the absolute attitude reference eliminates reliance on external reference equipment, improving convenience and reducing costs.

[0358] It should be noted that the carrier fixing error As an inherent property of the hardware, it changes little and remains effective after calibration; users in new locations can use previously known carrier-based fixed errors. Quickly re-estimate environmental bias by matching with star charts.

[0359] Generally, only 2-3 observations at different attitudes are needed to complete the initial separation calibration.

[0360] S1470 uses the corrected attitude matrix projection information.

[0361] The modified version obtained from the preceding steps (S1450 obtains the corrected first effective attitude matrix) Alternatively, S1460 yields the corrected second effective attitude matrix. ), rendering module As a projection attitude matrix, it projects celestial objects from the star catalog onto the image plane, overlaying AR star information to align the virtual image with the real starry sky. For ease of description, it will be referred to hereafter as the effective matrix. .

[0362] Specifically, the processing procedure of S1470 is as follows: (a) Constructing the world system direction vector of celestial bodies. For celestial bodies in the star catalog (stars, constellation nodes or navigation stars), the horizontal coordinates (azimuth angle az, altitude angle alt) are obtained by combining the right ascension and declination coordinates with time and latitude and longitude conversion. Construct the world system (east-north-sky) unit direction vector: enu = (cos(alt)·sin(az), cos(alt)·cos(az), sin(alt)).

[0363] (ii) Switch to the camera coordinate system. Use the corrected effective attitude matrix. Transform the world-system direction vector of the celestial body into the camera (body) coordinate system to obtain the celestial body's vector in the body coordinate system: s = · enu, · represents matrix-vector multiplication.

[0364] (iii) Perspective projection onto pixels. The camera along... Z direction, depth = sz (depth>0 indicates visible in front of the camera), pixel coordinates: x = cx + f·sx / depth, y = cy f·sy / depth. Where f is the projection focal length of the observation device, and cx and cy are the coordinates of the principal point of the image.

[0365] One possible implementation is to adjust the imaging display using different projection focal lengths (such as the projection focal length used by the combined optical device, the projection focal length for direct viewing on the terminal screen, and the projection focal length projected onto the captured image) and calibration parameters (focal length f, principal point) according to different display modes.

[0366] The following is the calculation of focal length for three types of projections: See Figure 15 , Figure 15 This is a schematic diagram of the focal length for astronomical optical observation proposed in an embodiment of this application. The projection focal length used by the combined optical device is as follows: When user 1510 observes, it can be considered that the observation point is the projection focus, and the final observed virtual image is used as the screen for perspective projection imaging. The projection focal length is the distance Dvscreen between the observation point and the virtual image. Let the size of the observed virtual image be Wvscreen, and the size of the first electronic star map be Wscreen. Through similarity relationships, the projection focal length of the first electronic star image is obtained as Fproject = (Wscreen / Wvscreen). Dvscreen, since the observed virtual image and the first electronic star map are the real and virtual images of the telescopic element, satisfy the perspective imaging relationship, Wscreen / Wvscreen is u object distance / v image distance, therefore the projection focal length Fproject = (u object distance / v image distance). Dvscreen, or intuitively speaking, the terminal projection focal length Fproject is the scaling of the observation device's projection focal length Dvscreen, and the scaling factor is the scaling factor of the virtual image of the first electronic starry sky.

[0367] When using the device, user 1510 can either view the image directly on the terminal device's screen or project the captured image for viewing. The projection focal length differs between the two scenarios, which will be briefly explained below.

[0368] When viewing directly on the terminal device's screen, the content displayed is captured by the camera and adaptively scaled (or cropped) to fit the screen size. Therefore, the projection focal length for direct viewing on the terminal device's screen is the camera's inherent pixel focal length. Screen pixel pitch The scaling ratio of the camera image displayed on the screen.

[0369] When projecting an image, the focal length projected onto the captured image is the camera's pixel focal length.

[0370] (iv) Rendering and overlay. The rendering module draws star points, constellation lines and annotation information based on the above pixel coordinates.

[0371] Users may need to take photos for recording purposes at any time during use. This application also proposes another image processing method for astronomical optical observation. One example of this method includes the following steps: 1. Photo capture triggered. In photo capture mode, in response to the user's tap on the screen, the terminal device marks the photo capture request and records the operation time as the start of the timeout window; at this time, the photo is not taken immediately, but enters a stable waiting state.

[0372] 2. Image Capture Gating. In each subsequent sensor frame, the terminal device calculates the inter-frame rotation angle between the current pose and the pose of the previous frame (obtained by inverse solving the Arccos formula of the rotation matrix trace). When the inter-frame rotation angle is within a predetermined stationary angle window (e.g., ≤0.15°, or <0.1°, etc., this embodiment does not impose special restrictions on this), it indicates that the terminal device has basically stopped and the image is stable, at which point image capture is triggered. Alternatively, to reduce the difficulty for users to remain still for extended periods, image capture is triggered regardless of whether the pose is stable when the time elapsed since the triggering time exceeds a predetermined timeout period (e.g., 5 seconds, or 10 seconds, etc., this embodiment does not impose special restrictions on this).

[0373] 3. Full-frame capture. Upon triggering, the camera extracts a complete full-frame image from the reference frame. It should be noted that sometimes the camera performs cropping or other operations on the extracted image before storing and / or displaying it. The complete full-frame image referred to here is the complete, uncropped image captured / extracted by the camera at the reference frame. It should also be noted that this complete full-frame image is the captured, actual external image.

[0374] 4. Overlay rendering. Based on the pose (corrected pose) and focal length parameters corresponding to the time of acquiring the full-frame image (i.e. the time of executing the previous step), the annotation overlay in the first electronic starry sky image is re-rendered at the size of the full-frame image, so that the annotation projection matches the geometry of the photograph, rather than using the geometry of the smaller head-up display (HUD) on the screen.

[0375] 5. Composite and Save: Overlay the overlay onto the full-frame image to composite an AR starry sky photo; at the same time, save the original full-frame and / or composite photo.

[0376] In the above steps, there are several possible examples of reference frames, three of which are described below.

[0377] In the first example, the reference frame is the frame when the photo is taken.

[0378] In the second example, the reference frame is the frame with the least jitter and / or the frame with the least rotation angle of the terminal device within a specified time after triggering the photo capture. Within this specified time, the terminal device's sensor can calculate the inter-frame rotation angle between the current pose and the previous frame's pose (e.g., by inversely solving the Arccos formula for the rotation matrix trace), and then compare the rotation angle magnitude within this time. Similarly, within this specified time, the terminal device's sensor can calculate the jitter between the current pose and the previous frame's pose, and then compare the jitter magnitude within this time. This allows for obtaining the highest quality photo within the specified time, further improving the shooting quality, and avoiding the impact of significant jitter or rotation on the photo quality within the frame at the time of triggering the photo capture.

[0379] In the third example, within a specified time after triggering the photo capture, the terminal device calculates the jitter and / or rotation angle in each frame. A frame is considered a reference frame if the jitter is less than a sixth threshold and / or the rotation angle is less than a seventh threshold. Possible implementations of the terminal device calculating the jitter and / or rotation angle in each frame can be found in the description of the second example. This approach allows for faster reference frame determination compared to the second example, improving the efficiency of image processing and storage.

[0380] This application does not impose special restrictions on the sixth and seventh thresholds. As an example, the seventh threshold is 0.01°.

[0381] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A starry sky optical observation device, characterized in that, include: The mounting section is used to support the terminal device, such that the screen of the terminal device faces the device and the screen emits a first electronic starry sky image. An optical path component is used to perform at least one of the following operations on the first electronic star image to obtain a second electronic star image: reflection, refraction, and increasing the virtual image distance; The first beam-splitting element includes a first folding surface and a first transmission surface; Wherein, the first folding surface faces the optical path component, and the first folding surface is used to reflect and / or refract the second electronic starry sky image to the user; The first transmissive surface is used to transmit a real starry sky image to the user.

2. The apparatus according to claim 1, characterized in that, The first beam splitter satisfies at least one of the following: A reflectivity of less than or equal to 10% is used to reduce reflected light from the image source; or, A transmittance greater than or equal to 90% is used to enhance starlight transmission; or, Thickness less than or equal to 1 mm to reduce ghosting.

3. The apparatus according to claim 1 or 2, characterized in that, The optical path assembly includes a first optical path deflection element and a first virtual image zooming element; At least one side of the first optical path deflection element faces the screen, and the first optical path deflection element is used to reflect and / or refract the first electronic star image to obtain a third electronic star image; At least one side of the first virtual image zooming element faces the first optical path deflection element. The first virtual image zooming element is used to increase the virtual image distance of the third electronic star image to obtain the second electronic star image.

4. The apparatus according to claim 3, characterized in that, The first virtual image retraction element is a Fresnel lens or a lens group containing at least one negative lens, used to correct aberrations.

5. The apparatus according to claim 3 or 4, characterized in that, The first optical path deflection element is a front surface reflector.

6. The apparatus according to claim 1 or 2, characterized in that, The optical path assembly includes a second beam splitter and a second optical path deflector, wherein the second optical path deflector is disposed on the side of the second beam splitter away from the screen; The second beam-splitting element includes a second transmission surface and a second deflection surface; Wherein, the second transmissive surface faces the screen, and the second transmissive surface is used to transmit the first electronic star image to obtain the fourth electronic star image; At least one side of the second optical path deflection element faces the second beam splitter element, and the second optical path deflection element is used to reflect and / or refract the fourth electronic star image to obtain the fifth electronic star image; The second folding surface faces the second optical path folding element, and the second folding surface is used to reflect and / or refract the fifth electronic star image to obtain the sixth electronic star image or the second electronic star image.

7. The apparatus according to claim 6, characterized in that, The optical path assembly further includes a second virtual image retraction element, and at least one side of the second virtual image retraction element faces the second folding surface; The second folding surface is used to reflect and / or refract the fifth electron star image to obtain the sixth electron star image; The second virtual image zooming element is used to increase the virtual image distance of the sixth electronic star image to obtain the second electronic star image.

8. The apparatus according to claim 6 or 7, characterized in that, The second optical path deflection element includes a first concave surface, which faces the second beam splitter.

9. The apparatus according to any one of claims 6 to 8, characterized in that, The second beam splitter and the first beam splitter form a first combination; The first combination satisfies at least one of the following: A reflectivity of less than or equal to 10% is used to reduce reflected light from the image source; or, A transmittance greater than or equal to 90% is used to enhance starlight transmission; or, The thickness of the second beam splitter in the first combination is less than or equal to 1 mm to reduce ghosting.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The second optical path deflection element is a front surface reflector.

11. The apparatus according to claim 1 or 2, characterized in that, The optical path assembly includes a third beam splitter and a third optical path deflector; At least one side of the third optical path deflection element faces the screen, and the third optical path deflection element is used to reflect and / or refract the first electronic star image to obtain the seventh electronic star image; The third beam splitter includes a third refractive surface, which faces the third optical path refractive element, and the third refractive surface is used to reflect and / or refract the seventh electron star image to obtain the second electron star image; The first folding surface faces the third folding surface, and the first folding surface is used to reflect and / or refract the second electronic star image to the user to obtain the eighth electronic star image; The third folding surface is also used to transmit the eighth electronic starry sky image and the external real starry sky image to the user.

12. The apparatus according to claim 11, characterized in that, The first folding surface is concave.

13. The apparatus according to claim 11 or 12, characterized in that, The third beam-splitting element and the first beam-splitting element constitute a second combination; The second combination satisfies at least one of the following: A reflectivity of less than or equal to 10% is used to reduce reflected light from the image source; or, A transmittance greater than or equal to 90% is used to enhance starlight transmission; or, The thickness of the third beam splitter in the second combination is less than or equal to 1 mm to reduce ghosting.

14. The apparatus according to any one of claims 11 to 13, characterized in that, The third optical path deflection element is a front surface reflector.

15. A starry sky optical observation system, characterized in that, include: Terminal equipment and starry sky optical observation device as described in any one of claims 1 to 14; When the terminal device is installed in the mounting section, the first part of the terminal device is exposed outside the starry sky optical observation device.

16. The system according to claim 15, characterized in that, The first part includes a first interactive area, which belongs to the screen; The terminal device is configured to adjust or generate the first electronic starry sky image in response to a first operation performed on the first interactive area. Adjusting the first electronic starry sky image includes adjusting at least one of the following: position, size, and angle.

17. The system according to claim 16, characterized in that, The first operation includes at least one of the following: swipe operation, long press operation, double-click operation, and single-click operation.

18. The system according to any one of claims 15 to 17, characterized in that, The first part includes a second interactive area, which is provided with at least one button; The terminal device is configured to adjust or generate the first electronic starry sky image in response to a second operation performed on the at least one button.

19. The system according to claim 18, characterized in that, The second operation includes at least one of the following: swipe operation, long press operation, double-click operation, and single-click operation.

20. The system according to any one of claims 15 to 19, characterized in that, The terminal device includes a voice input module; The terminal device is configured such that the voice input module receives a voice command and, in response to the voice command, adjusts or generates the first electronic starry sky image.

21. The system according to claim 20, characterized in that, The first part includes a third interactive area, which belongs to the screen; The terminal device is configured to activate or deactivate the voice input module in response to a third operation performed on the third interactive area.

22. The system according to claim 21, characterized in that, The third operation includes at least one of the following: swipe operation, long press operation, double-click operation, and single-click operation.

23. The system according to any one of claims 15 to 22, characterized in that, The terminal device includes a location acquisition module and / or an attitude acquisition module; The location acquisition module is used to obtain the geographical location information of the terminal device; The attitude acquisition module is used to acquire the three-dimensional attitude and / or azimuth angle of the terminal device.

24. The system according to any one of claims 15 to 23, characterized in that, The terminal device includes an image sensor; The first electronic star image includes a first calibration image, which includes at least one of the following: A positioning marker, wherein the positioning marker is disposed at a predetermined position on the first calibration image, includes at least one of the following: the center of the first calibration image, the edge of the first calibration image, or the device imaging principal point of the first calibration image, used for calibrating position and orientation errors; or, The image information acquired by the image sensor, representing the spatial layout of far-field targets, is used to calibrate position, orientation, and scaling errors.

25. An image processing method for astronomical optical observation, characterized in that, The method, applied to a terminal device in a stellar optical observation system as described in any one of claims 1 to 24, comprises: The camera of the terminal device captures raw images; Star detection is performed on the original image to obtain a set of detected star points. The set of detected star points includes multiple detected star points, and each detected star point has corresponding coordinates. The detected star point set is compared with the star catalog to obtain a final matching pair set. The final matching pair set includes one or more matching pairs. Each matching pair records the coordinates of the detected star point and the coordinates of the matching star catalog star point. The star catalog includes multiple star catalog star points, and each star catalog star point corresponds to a coordinate. Based on the final set of matching pairs, the pixel focal length is calculated. Based on the rotation matrix of the final set of matching pairs and the initial pose of the terminal device, a corrected rotation matrix is ​​obtained; The rendering module of the terminal device renders the first electronic starry sky image based on the calculated pixel focal length and the corrected rotation matrix.

26. The method according to claim 25, characterized in that, The step of comparing the detected star point set with the star catalog to obtain the final set of matching pairs includes: Q detection points are selected from the plurality of detection points in descending order of brightness, where Q is a positive integer greater than 1; Two detection points are randomly selected from the Q detection points to form a reference pair combination. A total of C(Q, 2) reference pairs are formed from the Q detection points. ; For each of the reference pair combinations, the line connecting the positions of the two detection star points in the reference pair combination forms a first baseline, and the length of the first baseline is recorded as 1. The coordinates of the Q detected star points are used to calculate the normalized coordinates of the matching star points relative to the first baseline; Select two star points from the multiple star catalogs, ranked from highest to lowest brightness. The line connecting the positions of the two star points forms the second baseline, and the length of the second baseline is denoted as 1. The coordinates of the star points in the multiple star catalogs are used to calculate the normalized coordinates of the star points relative to the second baseline; For each of the aforementioned reference pair combinations, the normalized coordinates of the matching star points are compared with the normalized coordinates of the star points in the star catalog to obtain a set of matching pairs; Select the final set of matching pairs from the set of C(Q,2) matching pairs corresponding to the C(Q,2) reference pairs.

27. The method according to claim 26, characterized in that, The set of matching pairs satisfies the following two conditions: The difference between the coordinates of each matched detected star point and the first component coordinates of the star catalog star point is less than or equal to the third threshold. Furthermore, the difference between the second component coordinates of each matched detected star point coordinate and the star table star point coordinates is less than or equal to the fourth threshold. The coordinates of the first component and the coordinates of the second component are perpendicular to each other.

28. The method according to claim 26 or 27, characterized in that, The step of selecting the final set of matching pairs from the set of C(Q,2) matching pairs corresponding to the C(Q,2) reference pairs includes: The final matching pair set is selected from the C(Q,2) matching pair sets that has the largest number of matching pairs and is greater than the reference number of matching pairs.

29. The method according to any one of claims 25 to 28, characterized in that, The step of calculating the pixel focal length based on the final set of matching pairs includes: The final set of matching pairs includes q pairs of matching stars. Two pairs of matching stars are randomly selected from the q pairs of matching stars to calculate the pixel focal length. The q pairs of matching stars constitute C(q,2) pixel focal length calculation combinations. For each pixel focal length, a combination of calculations is performed to obtain a pixel focal length estimate, resulting in a total of C(q,2) pixel focal length estimates. The calculated pixel focal length is determined from the C(q,2) pixel focal length estimates; The calculated pixel focal length replaces the camera calibration pixel focal length of the terminal device.

30. The method according to claim 29, characterized in that, For each of the pixel focal length calculation combinations, the corresponding estimated pixel focal length f is greater than 10 and satisfies the formula: f² = [ cos²θ1(A+B) 2C ± ] / (2 sin²θ1); The pixel focal length calculation combination includes detecting star points I1 and I2, where the direction vector of I1 in the body coordinate system is d1 = (x1, -y1, -f), and the direction vector of I2 in the body coordinate system is d2 = (x2, -y2, -f). Wherein, A satisfies the formula: A=x1²+y1², B satisfies the formula: B=x2²+y2², and C satisfies the formula: C=x1x2+y1y2. The θ1 satisfies the formula: cosθ1 = (d1·d2) / (|d1|·|d2|), where · is the vector dot product. The given a satisfies the formula: a = sin²θ₁ = 1 cos²θ1, where b satisfies the formula: b = [ cos²θ1·(x1²+y1²+x2²+y2²) 2(x1x2+y1y2)], where c satisfies the formula: c = (x1x2+y1y2)² cos²θ1·(x1²+y1²)(x2²+y2²).

31. The method according to claim 29 or 30, characterized in that, The step of determining the calculated pixel focal length from the C(q,2) pixel focal length estimates includes: One or more pixel focal length estimates are obtained by removing pixel focal length estimates that exceed the first range from the C(q,2) pixel focal length estimates. By statistically aggregating the estimated focal lengths of one or more pixels, the optimal estimate of the calculated pixel focal length is obtained.

32. The method according to any one of claims 25 to 31, characterized in that, The process of obtaining the corrected rotation matrix based on the final set of matching pairs and the initial pose of the terminal device includes: For each pair of matching stars in the final matching pair set, calculate the direction vector b and direction vector r in the first coordinate system. The direction vector b corresponds to the detection star point in each pair of matching stars, and the direction vector r corresponds to the star table star point in each pair of matching stars. Based on the direction vector b and the direction vector r, construct the correlation matrix B; Singular value decomposition is performed on the correlation matrix B to obtain the modified rotation matrix R; Based on the corrected rotation matrix R and the initial attitude matrix of the terminal device The corrected first effective attitude R is obtained. eff .

33. The method according to any one of claims 25 to 31, characterized in that, The process of obtaining the corrected rotation matrix based on the final set of matching pairs and the initial pose of the terminal device includes: The final set of matching pairs includes q pairs of matching stars. For each pair of matching stars in the q pairs of matching stars, a world system direction vector b and a direction vector r are calculated. The direction vector b corresponds to the detected star point in each pair of matching stars, and the direction vector r corresponds to the star table star point in each pair of matching stars. Based on the correspondence between the direction vector b and the direction vector r of each pair of matching stars, a correlation matrix B or its equivalent expression representing the overall rotation is constructed. The correlation matrix B is constructed by calculating the outer product of the direction vector b and the direction vector r for each pair of matching stars, and summing the outer products calculated for all matching stars to obtain the correlation matrix B, wherein B satisfies: B= ； Wherein, the r This represents the transpose of the direction vector r, where... This indicates a summation calculation; For the correlation matrix B, the rotation estimation method is used to solve for the corrected rotation matrix R, so that the corrected rotation matrix satisfies the condition of minimizing the overall error; The rotation estimation method includes: performing singular value decomposition B=UΣV on the correlation matrix. Construct the corrected rotation matrix R = U·V When det(R) < 0, use U·diag(1,1, 1)·V Chirality correction; Based on the corrected rotation matrix R and the initial attitude matrix of the terminal device The corrected first effective attitude R is obtained. eff The R eff satisfy: = R × (or equivalent expression), to complete the posture correction.

34. The method according to claim 32 or 33, characterized in that, The initial attitude matrix based on the corrected rotation matrix R and the terminal device The corrected effective attitude is obtained. Previously, the method also included: Calculate the rotation angle θ2 corresponding to the corrected rotation matrix R, where the rotation angle θ2 satisfies the formula: θ2=arccos((trace 1) / 2); Wherein, arccos() is the inverse function of the cosine function cos(), and trace is the trace of the modified rotation matrix R; When θ2 is less than or equal to a predetermined angle threshold, the rotation is adjusted based on the corrected rotation matrix R and the initial attitude matrix of the terminal device. The corrected first effective attitude was obtained. .

35. The method according to any one of claims 32 to 34, characterized in that, Using the effective attitude true value output by star map matching as the external observation anchor, the attitude error is decomposed into two independent parts using multiple observations under different attitudes: carrier fixed error and world fixed error. The carrier fixed error is fixed to the terminal device body and is constant in the carrier coordinate system, while the world fixed error is constant in the navigation coordinate system and does not change with the attitude of the terminal device. The carrier fixation error and the world fixation error are calibrated respectively, and applied to attitude correction respectively.

36. The method according to claim 35, characterized in that, When the attitude deviation is small, a linearized fast initial value estimation is performed using a rotation parameterization method. The rotation parameterization method includes any of the following: rotation vector, quaternion, Lie algebra, or other parameterization methods that can achieve the same rotation representation. Specifically, when using Lie algebras for linearized fast initial value estimation, the fixed error parameter of the carrier is parameterized as a Lie algebra. The world's fixed error parameter is parameterized as a Lie algebra. Construct a linearized error model: Where e(i) is the error vector of the i-th observation, the The effective pose of the i-th observation The N observations are combined into a system of linear equations: in, It is a 3x3 identity matrix; to ensure the coefficient matrix is ​​full rank, the N observations must include at least three different attitudes, and the different attitudes must ensure that the projection of the world sky direction in the carrier coordinate system covers at least two linearly independent directions, so as to ensure the coefficient matrix is ​​full rank. Calculate the residual vector for each observation. And calculate the residual norm. If the residual norm is less than a preset threshold Then and The optimal estimate and The carrier fixed error rotation matrix is ​​obtained through exponential mapping. and world fixed error rotation matrix : ； ； in, For cross product matrix operators; based on carrier fixed error rotation matrix World Fixed Error Rotation Matrix and the current initial attitude matrix The corrected second effective attitude was obtained. ; Second effective posture Satisfying the formula: ； The second effective posture Used for subsequent rendering projection.

37. The method according to claim 35, characterized in that, When the attitude deviation is large, a rotation parameterization method is used to iteratively solve the problem through exponential mapping or equivalent rotation synthesis. The rotation parameterization method includes any of the following: rotation vector, quaternion, Lie algebra or other parameterization methods that can achieve the same rotation representation. Among them, when using Lie groups When solving nonlinear optimizations precisely, The two types of errors are parameterized as Lie algebras and mapped exponentially: ； Construct a nonlinear least squares problem with the error matrix residuals as the cost function: ； Iterate the following two formulas separately until the residuals converge or the maximum number of iterations is reached to obtain the fixed error vector of the carrier. and the world fixed error vector : ； ； Among them, the The cross product matrix operator is represented as follows: Represents the fixed error vector of the carrier The corresponding Lie algebra update amount, the Represents the world fixed error in the world coordinate system. The corresponding Lie algebra update amount; Will and The calculation results are used to obtain the carrier fixed error rotation matrix through exponential mapping. and world fixed error rotation matrix : ； ； Based on the carrier fixed error rotation matrix World Fixed Error Rotation Matrix and the current initial attitude matrix The corrected second effective attitude was obtained. ; Second effective posture Satisfying the formula: ； The second effective posture Used for subsequent rendering projection; Alternatively, after obtaining the carrier fixed error rotation matrix, it can be directly applied to the star map projection of subsequent frames; or, the carrier fixed error rotation matrix can be re-solved based on the current observation attitude data during each star map observation.

38. The method according to any one of claims 25 to 37, characterized in that, The method further includes: The projection error of the virtual target image relative to the observation line of sight is divided into the following three categories: Positional errors are errors caused when the virtual image propagation direction is consistent with the observation direction, but the image center is not aligned. Directional errors are errors caused by the inconsistency between the propagation direction of the virtual image and the observation direction, specifically manifested as the virtual image plane not being perpendicular to the observation line of sight. Scaling errors are errors caused by the mismatch between the virtual image field of view distance and the magnification due to object distance deviation, specifically manifested as errors caused by proportional scaling of the entire image. The position error, orientation error, and scaling error are each independently corrected, and the correction includes at least one of the following calibration methods: Position error correction is achieved by adjusting the projection position within the screen of the terminal device to align the projected positioning mark with the fixed position of the starry sky optical observation device, thereby correcting the position error. Directional error correction, under the condition of small angles and without considering projection deformation, equates the rotational error around the horizontal axis and / or vertical axis to the overall angular offset of the virtual image propagation direction; by adjusting the projection position within the screen of the terminal device, the observation direction of the target virtual image is made consistent with the image acquisition direction of the image sensor, that is, coincides with the projection propagation direction, so as to achieve the correction of the directional error; wherein, the correction of the directional error is achieved by the translation of the projection center position, without introducing projection rotation parameters; Scaling error correction involves scaling the image information representing the spatial layout of far-field targets acquired by the image sensor onto the virtual image plane until it coincides with the observed entity, ensuring consistent viewing angles, thus obtaining a scaling matching coefficient R. This scaling matching coefficient is then compared with a preset theoretical scaling factor to obtain a scaling deviation ΔR. Based on this scaling deviation, the distance deviation Δu between the screen and the telescope element of the stargazing device is calculated, where Δu = ΔR × F × f / d, where F is the equivalent focal length of the screen imaging, d is the exit pupil distance, and f is the focal length of the telescope element. Simultaneously, the equivalent focal length of the screen imaging is corrected to R. F allows for precise alignment between the virtual image and the real observation target.

39. An image processing method for star observation, characterized in that, The method, applied to a terminal device in a stellar optical observation system as described in any one of claims 15 to 24, comprises: In photo mode, receive user photo requests; In response to the photo-taking request, the terminal device calculates the inter-frame rotation angle between the current posture and the posture of the previous frame in each subsequent sensor frame; When the inter-frame rotation angle is within a predetermined stationary angle window, and / or the time since receiving the photo-taking request exceeds a predetermined timeout period, a photo is taken. Extract the complete full-frame image from the camera reference frame; Based on the attitude and focal length parameters of the terminal device, the annotation overlay layer in the first electronic starry sky image is rendered at the size of the complete full-frame image, so that the annotation projection is geometrically matched with the photo. Wherein, the attitude is the attitude at the corresponding moment of the reference frame, and the focal length parameter is the focal length parameter at the corresponding moment of the reference frame.

40. The method according to claim 39, characterized in that, The reference frame satisfies at least one of the following: The reference frame is a frame at the time the photo is triggered; The reference frame is the frame with the least jitter of the terminal device within a specified time after the triggering of the photo capture; The reference frame is the frame in which the terminal device rotates the smallest angle within a specified time after the triggering of the photo capture. At the time corresponding to the reference frame, the jitter of the terminal device is less than the sixth threshold. At the time corresponding to the reference frame, the rotation angle of the terminal device is less than the seventh threshold.

Images