Brightness extraction method, electronic equipment, storage medium and program product

Abstract

The invention discloses a brightness extraction method, electronic equipment, a storage medium and a program product, and relates to the technical field of near-to-eye display, and the method comprises the steps: when an instruction of carrying out the brightness extraction of a display screen of near-to-eye display equipment through a target camera is detected; the mapping relation between camera pixels of a target camera and screen pixels of the display screen is determined, and the target camera is located at an exit pupil frame of the near-eye display device; displaying the calibration brightness image on a display screen, and shooting through a target camera to obtain a calibration brightness camera image corresponding to the calibration brightness image; and based on the mapping relationship, extracting brightness information corresponding to each screen pixel on the display screen from the calibrated brightness camera image. According to the invention, under the condition that the camera image is seriously distorted due to distortion of the optical system, the accurate brightness information completely matched with the original pixel distribution of the screen can be reliably and efficiently obtained.

Description

Technical Field

[0001] This application relates to the field of near-eye display technology, and more particularly to a brightness extraction method, electronic device, storage medium, and program product. Background Technology

[0002] With the widespread adoption of near-eye display devices such as virtual reality and augmented reality, the brightness uniformity and color fidelity of their display screens have become core indicators determining the quality of user experience. To ensure users obtain an immersive, comfortable, and consistent visual experience, it is essential to perform high-precision calibration on the brightness of each pixel on the display screen to eliminate brightness unevenness caused by manufacturing tolerances, optical system attenuation, or assembly deviations. This is of paramount importance for improving product display quality and user satisfaction.

[0003] Currently, obtaining pixel-level brightness data for screen correction typically relies on analyzing images captured by a camera located at the exit pupil. However, due to the inherent and complex distortions of the optical system in near-eye display devices, the images captured by the camera exhibit significant and non-linear distortions in both geometry and brightness distribution. This distortion introduced by the optical path creates a systematic deviation between the brightness information represented in the camera image and the original brightness information actually emitted by the display screen and expected to be observed. Traditional methods attempt to correct this distortion using complex image processing algorithms, but these correction processes are not only computationally intensive and inefficient, but more importantly, the accuracy and universality of their correction models are difficult to guarantee, easily introducing new errors and reducing the reliability of the final extracted brightness data, thus failing to meet the requirements for high-precision brightness correction.

[0004] Therefore, how to reliably and efficiently obtain accurate brightness information that perfectly matches the original pixel distribution of the screen even when the camera image is severely distorted due to optical system distortion has become the core technical bottleneck in achieving high-quality brightness correction in the field of near-eye displays. Summary of the Invention

[0005] The main objective of this application is to provide a brightness extraction method, electronic device, storage medium, and program product, which aims to solve the technical problem of how to reliably and efficiently obtain accurate brightness information that perfectly matches the original pixel distribution of the screen when the camera image is severely distorted due to optical system distortion.

[0006] To achieve the above objectives, this application proposes a brightness extraction method, the method comprising: When an instruction to extract brightness from the display screen of a near-eye display device via a target camera is detected, the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen is determined, wherein the target camera is located at the exit pupil frame of the near-eye display device; A calibration brightness image is displayed on the display screen, and a calibration brightness camera image corresponding to the calibration brightness image is captured by the target camera. Based on the mapping relationship, the brightness information corresponding to each screen pixel on the display screen is extracted from the calibrated brightness camera image.

[0007] In addition, to achieve the above objectives, this application also provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the brightness extraction method as described above.

[0008] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the brightness extraction method described above.

[0009] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the brightness extraction method described above.

[0010] This application provides a brightness extraction method, electronic device, storage medium, and program product, relating to the field of near-eye display technology. The brightness extraction method includes: upon detecting an instruction to extract brightness from the display screen of a near-eye display device via a target camera, determining the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen, wherein the target camera is located at the exit pupil frame of the near-eye display device; displaying a calibration brightness image on the display screen, and capturing a calibration brightness camera image corresponding to the calibration brightness image via the target camera; and extracting brightness information corresponding to each screen pixel on the display screen from the calibration brightness camera image based on the mapping relationship.

[0011] This application's embodiments effectively solve the technical problem of how to reliably and efficiently obtain accurate brightness information that perfectly matches the original pixel distribution of the screen when the camera image is severely distorted due to near-eye optical system distortion. This is achieved through an innovative technical approach combining mapping and direct extraction. First, upon detecting a brightness extraction command, a mapping relationship is pre-established between the camera pixels of the target camera (located at the exit pupil) and the screen pixels of the display screen. This crucial step bypasses the indirect process of complex geometric correction of distorted images in traditional methods, instead directly constructing a corresponding bridge from the distorted camera image space to the original screen pixel space. Subsequently, a known calibrated brightness image is displayed on the display screen, and a corresponding calibrated brightness camera image is obtained by capturing it using the target camera. Finally, based on the established mapping relationship, brightness information precisely corresponding to each screen pixel is directly extracted from the calibrated brightness camera image. The core of this application's embodiments lies in the fact that, when extracting brightness from the display screen of a near-eye display device using a target camera, the distorted pixels are directly used to locate the camera image, and the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen is calibrated on the spot. This on-the-spot calibration of the mapping relationship allows the subsequent brightness extraction process to be performed directly on the original camera image without complex distortion correction processing. This not only significantly reduces computational complexity and processing latency but also fundamentally avoids the risk of introducing additional errors due to inaccuracies in traditional correction models. Thus, even with the presence of optical distortion, efficient, high-precision, and highly reliable acquisition of screen pixel-level brightness information is still achieved, providing an accurate data foundation for subsequent brightness correction. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic flowchart of an embodiment of the brightness extraction method of this application; Figure 2 This is a flowchart illustrating one embodiment of the brightness extraction method of this application; Figure 3 This is a flowchart illustrating another embodiment of the brightness extraction method of this application. Figure 4 This is a schematic diagram of pixel positioning provided as an example of the brightness extraction method of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the brightness extraction method in this application embodiment.

[0015] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0017] Currently, obtaining pixel-level brightness data for screen correction typically relies on analyzing images captured by a camera located at the exit pupil. However, due to the inherent and complex distortions of the optical system in near-eye display devices, the images captured by the camera exhibit significant and non-linear distortions in both geometry and brightness distribution. This distortion introduced by the optical path creates a systematic deviation between the brightness information represented in the camera image and the original brightness information actually emitted by the display screen and expected to be observed. Traditional methods attempt to correct this distortion using complex image processing algorithms, but these correction processes are not only computationally intensive and inefficient, but more importantly, the accuracy and universality of their correction models are difficult to guarantee, easily introducing new errors and reducing the reliability of the final extracted brightness data, thus failing to meet the requirements for high-precision brightness correction.

[0018] Therefore, how to reliably and efficiently obtain accurate brightness information that perfectly matches the original pixel distribution of the screen even when the camera image is severely distorted due to optical system distortion has become the core technical bottleneck in achieving high-quality brightness correction in the field of near-eye displays.

[0019] In contrast, the solution of this application embodiment is a brightness extraction method, comprising: upon detecting an instruction to extract brightness from the display screen of a near-eye display device via a target camera, determining a mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen, wherein the target camera is located at the exit pupil frame of the near-eye display device; displaying a calibration brightness image on the display screen, and capturing a calibration brightness camera image corresponding to the calibration brightness image via the target camera; and extracting brightness information corresponding to each screen pixel on the display screen from the calibration brightness camera image based on the mapping relationship.

[0020] This application's embodiments effectively solve the technical problem of how to reliably and efficiently obtain accurate brightness information that perfectly matches the original pixel distribution of the screen when the camera image is severely distorted due to near-eye optical system distortion. This is achieved through an innovative technical approach combining mapping and direct extraction. First, upon detecting a brightness extraction command, a mapping relationship is pre-established between the camera pixels of the target camera (located at the exit pupil) and the screen pixels of the display screen. This crucial step bypasses the indirect process of complex geometric correction of distorted images in traditional methods, instead directly constructing a corresponding bridge from the distorted camera image space to the original screen pixel space. Subsequently, a known calibrated brightness image is displayed on the display screen, and a corresponding calibrated brightness camera image is obtained by capturing it using the target camera. Finally, based on the established mapping relationship, brightness information precisely corresponding to each screen pixel is directly extracted from the calibrated brightness camera image. The core of this application's embodiments lies in the fact that, when extracting brightness from the display screen of a near-eye display device using a target camera, the distorted pixels are directly used to locate the camera image, and the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen is calibrated on the spot. This on-the-spot calibration of the mapping relationship allows the subsequent brightness extraction process to be performed directly on the original camera image without complex distortion correction processing. This not only significantly reduces computational complexity and processing latency but also fundamentally avoids the risk of introducing additional errors due to inaccuracies in traditional correction models. Thus, even with the presence of optical distortion, efficient, high-precision, and highly reliable acquisition of screen pixel-level brightness information is still achieved, providing an accurate data foundation for subsequent brightness correction.

[0021] The near-eye display devices in the embodiments of this application may include, but are not limited to, near-eye display devices such as Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] In one embodiment, such as Figure 1 As shown, the brightness extraction method may include steps S100~S300: Step S100: When an instruction to extract brightness from the display screen of the near-eye display device via the target camera is detected, the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen is determined, wherein the target camera is located at the exit pupil frame of the near-eye display device; It should be noted that, in this embodiment, a near-eye display device refers to an electronic device worn on the user's head with the display screen close to the user's eyes, such as a virtual reality (VR) headset or augmented reality (AR) glasses. Its display screen is the core optical component used to generate and display image content. The exit pupil is a key concept in a near-eye display optical system, referring to the area within the beam of light emitted from the last optical element (such as the eyepiece) that allows the observer's pupil to move within a plane perpendicular to the optical axis without image vignetting (i.e., image loss). Simply put, it is the optimal viewing position area outside the optical system, within the space where the observer's eyes reside, where the entire display image can be seen.

[0024] The target camera is a miniature image sensor that is specially configured and fixedly installed at the exit pupil frame to replace the human eye in imaging the display screen. Its position is precisely designed to ensure that its optical center coincides with the ideal human eye observation position, so that the image it captures is geometrically closest to the image actually seen by the human eye.

[0025] A mapping relationship refers to a mathematical or lookup table-like correspondence rule that establishes a one-to-one, one-to-many, or many-to-one association between each photosensitive unit (i.e., camera pixel) on the target camera's image sensor and each light-emitting or controlled unit (i.e., screen pixel) on the display screen. This relationship describes "which (or which) camera pixels receive light signals primarily from which screen pixel." Due to the existence of optical system distortion, this mapping is usually non-linear and non-uniform.

[0026] It's easy to understand that each camera pixel in a camera image corresponds to a unique pixel coordinate. Similarly, each screen pixel in a display screen also corresponds to a unique pixel coordinate. Therefore, this mapping relationship can also be presented in the form of pixel coordinate pairs. By querying this mapping relationship, the pixel position of any screen pixel on the display screen in the camera image captured by the target camera can be determined.

[0027] In this embodiment, after the system issues a brightness extraction command, the mapping relationship is first determined. This is typically accomplished through a one-time, high-precision calibration process. For example, a series of known, high-contrast geometric patterns (such as dot matrix, checkerboard, or single-pixel illumination sequences) can be displayed sequentially on the screen, while corresponding images are captured by a target camera. Image processing algorithms (such as feature point extraction and template matching) identify the position of each feature point (corresponding to one or a group of screen pixels) in the camera image and match it with known screen pixel coordinates, thereby establishing a mapping function or mapping table from camera pixel coordinates to screen pixel coordinates. This process fully characterizes all geometric distortions and optical path bending effects introduced by the optical system (including lenses, mirrors, etc.).

[0028] This embodiment overturns the traditional serial processing mode of "correcting the image first, then extracting brightness". Traditional methods attempt to perform global geometric correction on distorted camera images to "restore" the screen shape. This process is computationally intensive, and the correction model itself may introduce errors. This embodiment, however, takes a different approach, directly establishing a mapping "dictionary" from the starting point to the points. Once this mapping relationship is established, the system no longer needs to care whether the overall camera image is "distorted", but only needs to know which (or which) specific pixels in the camera image the light information of each screen pixel is "hidden" in. This is equivalent to bypassing the difficult problem of distortion correction at the source of data extraction, transforming the problem from "how to correct a distorted image" to "how to find the target based on the map", laying the most critical foundation for subsequent direct and accurate brightness extraction.

[0029] Step S200: Display the calibration brightness image on the display screen, and capture the calibration brightness camera image corresponding to the calibration brightness image using the target camera; It should be noted that, in this embodiment, the calibration brightness image refers to a known image specifically designed and displayed on the screen for brightness extraction. It is typically a series of image patterns with specific, known absolute or relative brightness values. For example, it can be a uniform single grayscale image across the entire screen (e.g., displaying 50% of the maximum brightness), or a test image composed of multiple different brightness blocks (e.g., a grayscale map from black to white). The calibration brightness camera image refers to the digital image simultaneously captured by a target camera located at the exit pupil frame when the calibration brightness image is displayed on the screen.

[0030] In this embodiment, the system controls the display screen to show a predefined calibrated brightness image. Simultaneously, a target camera located at the exit pupil frame captures the display image with appropriate exposure parameters (ensuring no overexposure or underexposure), thereby capturing one or more frames of calibrated brightness camera images. Because the target camera is located at the exit pupil frame, and the optical path (including all optical elements) between it and the screen is fixed, the image captured by the camera, although geometrically distorted due to optical distortion, maintains a stable response relationship between the grayscale value (or RGB channel value) of each pixel and the light intensity (i.e., brightness) received from the corresponding screen pixel. This image fully records the intensity distribution of the screen's emitted signal on the camera sensor plane under the influence of distortion.

[0031] The technical advantage of this step lies in acquiring first-hand "encrypted" data containing the original brightness information. Here, "encryption" refers to the non-linear perturbation of the spatial position of the original screen image by optical distortion. The calibrated brightness image provides a known portion of the brightness "key" (i.e., the light intensity emitted from a certain point on the screen is known), while the captured calibrated brightness camera image is the "encrypted" result. The key to this step is that it involves targeted data acquisition under the premise that the mapping relationship is known. Therefore, the camera image does not need to be geometrically correct; it only needs to faithfully record the spatial distribution of light intensity. This relaxes the requirements for camera image quality (in a geometric sense), allowing the system to directly use the raw, uncorrected camera image for subsequent processing, avoiding the complexity and uncertainty brought about by the high-precision geometric correction required in traditional processes.

[0032] Step S300: Based on the mapping relationship, extract the brightness information corresponding to each screen pixel on the display screen from the calibrated brightness camera image.

[0033] It should be noted that, in this embodiment, brightness information refers to quantified data characterizing the light intensity emitted by each screen pixel on the display screen. It can typically be a relative value (such as a 0-1 range value based on normalized camera pixel grayscale values), or it can be converted to physical units (such as nits) through pre-defined absolute brightness calibration. The extraction process refers to locating and reading the camera pixel data associated with each screen pixel in the calibrated brightness camera image obtained in step S200, based on the mapping relationship established in step S100, and calculating the brightness value of that screen pixel.

[0034] This step is the decoding and restoration stage of brightness information, and it is also the final step in realizing the value of the brightness extraction method provided in this embodiment. The specific operation is as follows: For each screen pixel P_screen on the display screen, the system queries the mapping relationship predetermined in step S100 to find a set of one or more camera pixels (denoted as {P_camera}) corresponding to it in the calibration brightness camera image. This set may contain multiple pixels due to the point spread function, aberrations, etc., of the optical system. Then, the grayscale values ​​of these corresponding camera pixels are read from the calibration brightness camera image (e.g., taking the green channel value or converting it to the brightness value Y). Next, the grayscale values ​​of these camera pixels are fused (e.g., weighted average) to calculate the original brightness representation value of the screen pixel P_screen. If the absolute brightness of the calibration brightness image itself is known, this representation value can also be converted to an absolute brightness value through simple proportional calculation or lookup table. By traversing all screen pixels and repeating this process, the brightness information matrix of each screen pixel on the entire display screen can be obtained.

[0035] This embodiment utilizes a pre-established, precise mapping relationship as a "decoder" to directly perform pixel-level data mining on raw camera images that have not undergone any geometric distortion correction. Its core advantage lies in: High precision: Extraction is performed directly based on the mapping relationship obtained by physical calibration, avoiding the quadratic error that may be introduced by traditional image geometric correction algorithms (such as polynomial fitting and resampling interpolation). The extracted brightness information is strictly aligned with the screen pixels in spatial position.

[0036] High efficiency: The entire extraction process is essentially a lookup and simple calculation based on the mapping table, with extremely low computational complexity. It completely avoids computationally intensive image transformation operations (such as affine transformation, perspective transformation, and remapping), and the processing speed is extremely fast, which can meet the needs of high-speed production line inspection or real-time calibration.

[0037] High reliability: The accuracy of the brightness extraction method provided in this embodiment depends on the quality of the mapping relationship calibrated in the early stage. This calibration can be completed in a controlled environment using high-precision equipment, and its accuracy can be fully verified and guaranteed. Once the calibration is completed, the subsequent brightness extraction process is stable and reliable, and is not affected by changes in image content.

[0038] This embodiment effectively solves the technical problem of how to reliably and efficiently obtain accurate brightness information that perfectly matches the original pixel distribution of the screen when the camera image is severely distorted due to near-eye optical system distortion. First, upon detecting a brightness extraction command, a mapping relationship is pre-established between the camera pixels of the target camera (located at the exit pupil) and the screen pixels of the display screen. This crucial step bypasses the indirect process of complex geometric correction of distorted images in traditional methods, instead directly constructing a corresponding bridge from the distorted camera image space to the original screen pixel space. Then, a known calibrated brightness image is displayed on the display screen, and a corresponding calibrated brightness camera image is obtained by capturing it using the target camera. Finally, based on the established mapping relationship, brightness information precisely corresponding to each screen pixel is directly extracted from the calibrated brightness camera image. The core of this embodiment lies in directly utilizing the distorted pixels in the camera image to locate the camera image when extracting brightness from the near-eye display device's screen using the target camera. The mapping relationship between the camera pixels and the screen pixels of the display screen is then calibrated on the spot. This on-the-spot calibration allows the subsequent brightness extraction process to be performed directly on the original camera image without complex distortion correction. This significantly reduces computational complexity and processing latency, and fundamentally avoids the risk of introducing additional errors due to inaccuracies in traditional correction models. Thus, even with optical distortion, efficient, high-precision, and highly reliable acquisition of screen pixel-level brightness information is achieved, providing an accurate data foundation for subsequent brightness correction.

[0039] In one feasible implementation, such as Figure 2 As shown, the step S100 above, which determines the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen, may include steps S110 to S130: Step S110: Display a pixel positioning image on the display screen, and capture a pixel positioning camera image corresponding to the pixel positioning image using the target camera. The pixel positioning image includes a positioning point array composed of multiple positioning points. It should be noted that, in this embodiment, the pixel positioning image is a specific pattern displayed on the display screen, specifically designed to establish mapping relationships. Its core feature is that it includes an array of positioning points arranged in a regular pattern. This array should uniformly cover most or all of the effective display area of ​​the screen to ensure that the subsequently established mapping relationship has sufficient representativeness and spatial coverage. Positioning points can be highlighted (e.g., white) dots, patterns of specific shapes (e.g., circles, crosses), or any features that are easily and accurately identified in the camera image. Common positioning point array forms include, but are not limited to: equidistant two-dimensional dot arrays, checkerboard corner dot arrays, etc. The pixel positioning camera image refers to the image simultaneously captured by the target camera when the pixel positioning image is displayed on the display screen.

[0040] In this embodiment, the system controls the display screen to display a pre-generated pixel positioning image. At this time, each positioning point in the positioning point array corresponds to one or a set of known coordinates of a screen pixel on the display screen (for example, the center of a positioning point may correspond to a specific screen pixel coordinate). Simultaneously, a target camera located at the exit pupil frame captures an image of the displayed scene. Due to optical system distortion, the originally regular positioning point array in the captured image undergoes geometric distortion, and the image point position (i.e., the corresponding camera pixel coordinate) of each positioning point in the camera image is shifted. This pixel positioning camera image faithfully records this distorted positional relationship, providing direct visual data for subsequent extraction of accurate pixel correspondences.

[0041] Step S120: Determine the screen pixel corresponding to each positioning point in the positioning point array, and locate the camera image through the pixel to determine the camera pixel corresponding to each positioning point in the positioning point array. It should be noted that this step involves parsing the data obtained in step S110 to extract the core correspondence pairs. Specifically, this includes two parallel parsing processes: 1. Determine the screen pixel corresponding to the positioning point: Since the pixel positioning image is generated and displayed by the system, the theoretical position of each positioning point in the display screen coordinate system (i.e., its corresponding screen pixel coordinate) is known in advance or can be accurately calculated according to the image generation rules.

[0042] 2. Determine the camera pixels corresponding to the positioning points: The pixel positioning camera image obtained in step S110 can be analyzed by image processing algorithms (such as binarization, contour detection, centroid calculation, corner detection algorithms, etc.) to identify the image points of all positioning points in the image and accurately calculate the position of each image point in the camera image coordinate system, that is, its corresponding camera pixel coordinates.

[0043] At this point, the system has obtained a set of correspondence pairs: each correspondence pair contains a known screen pixel coordinate (Screen_X, Screen_Y) and a camera pixel coordinate (Camera_X, Camera_Y) obtained through image analysis. This set of discrete correspondence pairs represents a direct connection between the screen pixel space and the camera pixel space at multiple positioning points.

[0044] Step S130: Based on the screen pixels and camera pixels corresponding to each positioning point in the positioning point array, establish a mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen.

[0045] It should be noted that this step uses the set of discrete correspondence pairs obtained in step S120 to construct a continuous and complete mapping relationship through mathematical modeling or interpolation methods, so that it can cover all screen pixels on the display screen (not just the screen pixels at the positioning point).

[0046] Specifically, the system can use the obtained N sets of correspondence pairs (Screen_i, Camera_i) as known data points and establish a mapping function from one space to another through a fitting algorithm. This can be a mapping function F: (Camera_X, Camera_Y) -> (Screen_X, Screen_Y) from camera pixel coordinates to screen pixel coordinates, or it can be a reverse mapping. Commonly used fitting models include, but are not limited to: polynomial models (such as second-order and third-order polynomials), perspective transformation models, or more complex non-uniform rational B-spline models, etc. The choice of model depends on the complexity of the optical system distortion and the accuracy requirements. The fitting process aims to find a function such that the error between the screen pixel coordinates calculated by this function at the known camera pixel coordinates and the known screen pixel coordinates is minimized (e.g., using the least squares method).

[0047] The established mapping relationship can be stored in the system as a lookup table. This lookup table stores the corresponding camera pixel coordinates for each screen pixel. During actual brightness extraction, the correspondence between any screen pixel and the camera image area can be quickly determined by querying this lookup table.

[0048] This embodiment provides a specific, operable, and high-precision method for establishing mapping relationships through steps S110 to S130. Its core value lies in decomposing the abstract process of "determining mapping relationships" into a clear pipeline of "displaying a positioning pattern → capturing a distorted image → extracting discrete corresponding points → fitting a continuous mapping model." This method fully utilizes the controllability of the display system and the maturity of image processing technology, enabling high-precision calibration to be completed in a controlled environment (such as factory calibration). The established mapping relationship directly and accurately encodes all geometric distortion information of the optical path, providing reliable and easily implementable technical support for the innovative scheme described in the above embodiment that bypasses image correction to directly extract brightness. Through this discrete sampling and continuous modeling approach, this embodiment ensures mapping accuracy while also considering storage and computational efficiency, making the entire brightness extraction scheme innovative, practical, and engineering feasible.

[0049] In one feasible implementation, step S120 above, which involves determining the camera pixel corresponding to each positioning point in the positioning point array by locating the camera image using the pixel, may include steps S121-S122: Step S121: Determine the camera pixels corresponding to the first positioning point and the second positioning point from the pixel positioning camera image, wherein the first positioning point is the center positioning point of the positioning point array, and the second positioning point is the positioning point in the positioning point array that is adjacent to the first positioning point. It should be noted that in this embodiment, the center positioning point is the positioning point that is logically (or geometrically) at the center of the specified point array. For example, for an M-row, N-column matrix, where M and N are odd numbers, its center positioning point can be the intersection of the (M+1) / 2th row and the (N+1) / 2th column; or the geometric center of the array can be used as the center positioning point. Due to its positional symmetry, the center positioning point usually exhibits less distortion in the camera image and is easily and reliably identified using search algorithms (such as finding brightness peak regions or centers of symmetry). Adjacent positioning points refer to those directly adjacent to the center positioning point in the original array arrangement (e.g., in one of the four directions: up, down, left, or right). These adjacent positioning points are used to provide reference directions and distance scales.

[0050] In one example, the system first performs a global analysis of the pixel-localized camera image, utilizing the characteristics of the localization point array (such as high brightness and specific shape) for target detection. This initially identifies the image point regions most likely corresponding to the first localization point (center localization point) and the second localization point (adjacent localization points of the center localization point). Then, sub-pixel precision algorithms (such as gray-level centroid method and Gaussian surface fitting) are used to accurately calculate the camera pixel coordinates of the centers of these image point regions, denoted as (Cx_center, Cy_center) and (Cx_neighbor, Cy_neighbor), respectively. Successfully identifying these two key localization points provides initial "anchor points" and "beacons" for the subsequent identification of all other localization points.

[0051] It is easy to understand that when designing a positioning point array, the first and second positioning points can be designed to have easily identifiable features, such as shape, size, brightness, or color that distinguish them from other positioning points.

[0052] In addition, contour detection can be performed on the distorted image point pattern sent by the positioning point array in the pixel positioning camera image, thereby locking the image point region corresponding to the central positioning point based on the geometric center of the contour. Then, the camera pixel coordinates corresponding to the central positioning point are determined by the gray-scale centroid method. Then, taking advantage of the fact that the optical system has a small degree of distortion at the center, which can be almost ignored, the image points corresponding to the eight second positioning points adjacent to the central point are searched within a certain radius around the camera pixel coordinates, thereby determining the camera pixel coordinates corresponding to the second positioning points.

[0053] Step S122: Based on the camera pixels corresponding to the first positioning point and the second positioning point, determine the camera pixel corresponding to the third positioning point from the pixel positioning camera image, wherein the third positioning point is a positioning point other than the first positioning point and the second positioning point in the positioning point array.

[0054] It should be noted that this step is crucial for achieving efficient and robust identification of all localization points. The third localization point refers to all other localization points in the array besides the first and second localization points. The system employs a model-guided progressive search strategy.

[0055] The principle is as follows: The logical positions (row and column indices) of the first and second positioning points in the original screen array and their precise pixel coordinates in the camera image are known. Based on the physical characteristics of optical system distortion, it can be assumed that the distortion is relatively smooth and continuous within a small local area of ​​the camera image. Therefore, the approximate position of the next positioning point to be identified in the camera image can be predicted using two or more identified positioning points.

[0056] The process is as follows: Establish a local transformation model: Based on the screen coordinates (known) and camera coordinates (measured) of the first and second positioning points, a simple local transformation relationship can be calculated (e.g., an affine transformation approximation considering translation, scaling, and rotation). This model describes a coarse mapping from screen coordinates to camera coordinates in the region near these two points.

[0057] Prediction and Search: For any third localization point, its coordinates in the screen array are known. Using the local transformation model established above, the approximate coordinate region of this point in the camera image can be predicted.

[0058] Precise matching: The system performs precise positioning in the vicinity of the predicted coordinate area (within a small search window) using the same feature detection and sub-pixel localization algorithm as in step S121, thereby determining the precise camera pixel coordinates of the third positioning point.

[0059] Iteration and Expansion: Whenever a new localization point is successfully identified, it can be used together with the already identified points to update or optimize the local transformation model, and then the prediction and identification of the next localization point can continue. This process can start from the central region and spread outward like a "wave" until all localization points in the array are identified.

[0060] This implementation method provides a highly efficient and interference-resistant location point identification method by introducing a progressive identification strategy of "from the center outwards, and from the known to the unknown." Its technical advantages are: High robustness: Traditional global recognition algorithms may fail due to excessive distortion at image edges or noise interference. This method starts from the central region with relatively small distortion and gradually moves outward. Each step searches within a small prediction window, greatly eliminating interference from irrelevant regions and improving the recognition success rate.

[0061] High efficiency: It avoids the huge computational overhead of performing a global intensive search across the entire high-resolution camera image. The search range is strictly limited to a local window based on model predictions, significantly reducing computational cost and enabling faster processing.

[0062] Adaptability: The local transformation model can be dynamically updated based on the identified points, adapting to distortion changes in different regions and improving the overall accuracy of identification.

[0063] Steps S121 and S122 of this embodiment together constitute an intelligent positioning point recognition solution. This not only solves the problem of "how to find points," but also, through ingenious algorithm design, addresses the challenge of "how to quickly and accurately find all points" in environments with severe distortion and noise. This ensures the reliability and efficiency of acquiring the original data (corresponding point pairs) during the mapping relationship establishment process, and is a crucial step in ensuring the practical application of the entire brightness extraction method.

[0064] In one feasible implementation, such as Figure 3 As shown, step S122 above may include steps S400~S700: Step S400: Determine the first positioning point and the second positioning point as mapped positioning points, and determine the third positioning point as an unmapped positioning point; It should be noted that, in this embodiment, mapped positioning points refer to those positioning points whose precise correspondence between their corresponding screen pixel coordinates and camera pixel coordinates has been successfully determined. The first and second positioning points successfully identified in the first stage (step S121) constitute the initial set of mapped positioning points. Unmapped positioning points are all other positioning points in the specified point array whose corresponding camera pixel coordinates have not yet been determined; their initial set is all the third positioning points.

[0065] Through this step, the system clearly classifies and marks the status of all positioning points, establishing a clear starting state and a queue of tasks to be processed for the subsequent step-by-step and orderly identification process.

[0066] Step S500: Determine the target unmapped location point from the unmapped location points, and determine the first target mapped location point and the second target mapped location point corresponding to the target unmapped location point from the mapped location points; Wherein, the unmapped target location point is a location point adjacent to the mapped location point among the unmapped location points, the first mapped target location point is a location point adjacent to the unmapped target location point among the mapped location points, and the second mapped target location point is a location point adjacent to the first mapped target location point among the mapped location points, and is on the same straight line as the first mapped target location point and the unmapped target location point; It should be noted that, in this embodiment, the target unmapped location point is the specific unmapped location point to be processed in the current iteration, and its selection criterion is "adjacent to mapped location points". This means that the point is adjacent to the boundary of the region with established mapping relationship, which is convenient for making predictions using existing information. The adjacency relationship can be defined according to the topology of the original location point array. For example, in a rectangular point array, it usually refers to direct adjacency in the row direction or column direction.

[0067] Furthermore, to accurately predict the geometric constraints of the unmapped target location points, the system needs to select two specific reference points from the mapped location points: the first mapped target location point is a reference point directly adjacent to the unmapped target location point, providing the most direct positional association; the second mapped target location point is adjacent to the first mapped target location point, and all three (the first mapped target location point, the second mapped target location point, and the unmapped target location point) lie on the same straight line (e.g., in the same row or column) in the location point array. This condition is crucial, as it ensures that these three points form a straight line segment in the screen coordinate system. In the camera image, although this straight line may become a curve due to distortion, within a small local area, it can be approximated that they still satisfy some predictable linear or low-order nonlinear relationship. This provides a strong geometric constraint model (e.g., linear extrapolation or simple curve fitting) for predicting the camera pixel coordinates of the third point (the unmapped target location point) from the camera pixel coordinates of the two known points (the first and second mapped target location points).

[0068] Step S600: Based on the camera pixels corresponding to the first target mapped positioning point and the second target mapped positioning point, determine the camera pixels corresponding to the unmapped positioning point of the target from the pixel positioning camera image; In this embodiment, the system knows the precise coordinates of the first and second target mapped positioning points in the camera image. Based on the geometric relationship that these three points lie on the same straight line in screen space, and assuming that the optical distortion is smooth locally, a prediction model can be constructed. For example, it can be assumed that the camera pixel coordinates of the unmapped target positioning points lie on the extension line of the local vector direction determined by the two known points. Based on this model, the system can calculate a predicted search region in the pixel-localized camera image. Then, within this predicted region (usually a window with a small radius centered on the predicted point), the same sub-pixel precision feature detection algorithm (such as gray-scale centroid method, template matching, etc.) as in step S121 is used for searching, thereby accurately locating the camera pixel coordinates corresponding to the actual imaging of the unmapped target positioning points. This method transforms the global search problem into a series of highly constrained local searches, greatly improving efficiency and noise resistance.

[0069] Step S700: Determine the unmapped target positioning point as a mapped positioning point, and return to execute the step of determining the unmapped target positioning point from the unmapped positioning points until there are no unmapped positioning points, and obtain the camera pixels corresponding to all third positioning points.

[0070] In this embodiment, once the camera pixel coordinates of an unmapped target location point are successfully identified, its state is updated from "unmapped" to "mapped," and it is added to the set of mapped locations. This newly added point expands the boundary of the known area. Subsequently, the system returns to step S500, selects the next "target unmapped location point" from the updated set of "unmapped locations" based on the criterion of "adjacent to a mapped location point," and repeats steps S500 to S600. This cycle continues, and the mapped area expands outward from the initial center location point as if "growing," until it covers the entire location point array. All "unmapped locations" are successfully identified and converted into "mapped locations," thereby obtaining the camera pixel coordinates corresponding to all third location points.

[0071] This implementation constructs a highly efficient, robust, and geometrically self-consistent fully automatic localization point recognition algorithm through steps S400 to S700. Its core technological advantages are reflected in the following aspects: First, by managing the "mapped / unmapped" state and employing an iterative mechanism of "inside-out, layer-by-layer diffusion," it decomposes the complex global recognition problem into a series of simple local recognition tasks, significantly reducing computational complexity and ensuring the orderly nature of the processing. Second, by strictly limiting that "unmapped target localization points" must be adjacent to "mapped" areas and using two "mapped" points on the same straight line as prediction benchmarks, the algorithm constrains the search range to a very small prediction window. This not only greatly improves search efficiency but, more importantly, effectively avoids mismatches caused by image distortion, noise, or false features (such as lens smudges or reflections) in areas far from the correct location, thereby significantly improving the robustness and accuracy of recognition. Finally, the entire recognition process is fully automated, requiring no manual intervention. It can autonomously complete the recognition of the entire array starting from the initial two points, demonstrating strong practicality and engineering application value. This algorithm cleverly transforms the image processing problem into a graph growth problem based on topological and geometric constraints, providing high-quality and highly reliable original corresponding point data for establishing the core mapping relationship.

[0072] In one feasible implementation, step S600 above may include steps S610 to S630: Step S610: Based on the camera pixels corresponding to the first target mapped positioning point and the second target mapped positioning point, determine the first camera pixel distance and the first camera pixel direction between the first target mapped positioning point and the second target mapped positioning point; It should be noted that, in this embodiment, the first camera pixel distance is a scalar value, representing the Euclidean distance between the camera pixel coordinates corresponding to the first mapped target location point and the second mapped target location point in the camera image coordinate system. This distance reflects the physical interval between these two identified location point image points on the camera sensor plane, and its value already includes the influence of local distortion of the optical system. The first camera pixel direction is a vector pointing from the second mapped target location point to the first mapped target location point, representing the direction of the straight line from the second mapped target location point to the first mapped target location point in the camera image. It can be represented by a direction angle (such as an angle relative to the horizontal axis of the image) or by a normalized direction vector. The first camera pixel distance and the first camera pixel direction together constitute the local geometric reference connecting these two known points in the camera image, which is a key parameter for subsequent linear prediction.

[0073] Step S620: Determine the camera pixels to be corrected corresponding to the unmapped positioning points of the target based on the camera pixels corresponding to the first target mapped positioning points, the distance of the first camera pixels, and the orientation of the first camera pixels; It should be noted that this step is a position prediction based on the assumption of local linearity. The camera pixel to be corrected is a preliminary, unverified prediction of the camera pixel corresponding to the unmapped target location point. Since the unmapped target location point is on the same straight line as the first and second mapped target location points in the original screen location point array, and both the unmapped and mapped target location points are adjacent to the first mapped target location point, this means that in screen space, the distance between the unmapped target location point and the first mapped target location point is equal to the distance between the second mapped target location point and the first mapped target location point. Furthermore, the direction from the second mapped target location point to the first mapped target location point is the same as the direction from the first mapped target location point to the unmapped target location point. Therefore, although the straight line on the screen may become a curve in the camera image due to distortion, in a small local area, it can be assumed that the distortion is approximately uniform. That is, equal distances in screen space are approximately equal in the camera image, and the same directions in screen space are approximately the same in the camera image.

[0074] Based on this, the pixel coordinates of the camera pixel to be corrected can be determined as follows: taking the camera pixel coordinates corresponding to the first target mapped positioning point as the starting point, extend the first camera pixel distance along the first camera pixel direction, thereby calculating the camera pixel coordinates of the camera pixel to be corrected corresponding to the target unmapped positioning point.

[0075] Furthermore, the above method is based on the premise that adjacent positioning points in the same row or column of the positioning point array are at the same distance. However, even if the distances between adjacent positioning points in the same row or column are not the same, the target camera pixel distance between the camera pixel to be corrected and the camera pixel corresponding to the first target mapped positioning point can be calculated based on the first camera pixel distance, combined with the first screen pixel distance between the first target mapped positioning point and the second target mapped positioning point, and the target screen pixel distance between the first target mapped positioning point and the target unmapped positioning point. Thus, when determining the camera pixel coordinates of the camera pixel to be corrected, the target camera pixel distance is extended along the direction of the first camera pixel, starting from the camera pixel coordinates corresponding to the first target mapped positioning point.

[0076] Step S630: Based on the camera pixels to be corrected, determine the camera pixels corresponding to the unmapped positioning point of the target from the pixel positioning camera image.

[0077] This implementation defines a finite-sized search window (e.g., a circular region with a radius of R camera pixels or a rectangular region with length and width of R pixels) centered on the pixel coordinates of the camera to be corrected. Since the prediction is based on the assumption of local linearity, even with nonlinear distortion, the true image point should fall near the predicted point. Subsequently, within this small window, the system performs a detailed analysis of the pixel-localized camera image, using sub-pixel localization algorithms (e.g., calculating the gray-level centroid of the image patch within the region, performing Gaussian surface fitting, or performing correlation matching with a known localization point template) to accurately find the actual center position of the brightness feature (i.e., the localization point image point). The coordinates of this finally determined center position are the precise coordinates of the camera pixel corresponding to the unmapped localization point of the target. This process effectively transforms the global, time-consuming feature search into a series of fast, localized, and precise alignment operations.

[0078] It is worth mentioning that the size of the search window can be determined by the pixel distance of the first camera or the pixel distance of the target camera. Specifically, it can be based on half of the pixel distance of the camera as the radius or side length. Preferably, the radius or side length of the search window should be less than or equal to half of the pixel distance of the camera and greater than or equal to one-third of the pixel distance of the camera.

[0079] This implementation, through steps S610 to S630, concretizes the core idea of ​​"predicting unknown points based on known points" in localization point recognition into a clear and computable "distance-direction" prediction and local refinement algorithm. Its technical advantage lies in achieving an optimal balance between efficiency and accuracy: First, steps S610 and S620 utilize established local geometric relationships (distance and direction) for rapid linear extrapolation, generating a high-quality initial predicted coordinate (camera pixels to be corrected). This fundamentally avoids the enormous computational overhead of blindly searching the entire high-resolution image. Second, step S630 strictly limits the search range to a very small window centered on the predicted point. This not only minimizes computation but also significantly reduces the risk of mismatches in other areas of the image due to noise, artifacts, or complex background interference, thus ensuring the robustness of the recognition process. Finally, the entire prediction-refinement process is fully automated, and its accuracy relies on sub-pixel localization technology, achieving localization accuracy higher than that of a single pixel, providing reliable data points for establishing high-precision pixel-level mapping relationships. This two-stage strategy of "coarse prediction" plus "fine search" is the key to the ability of this implementation method to efficiently, accurately, and automatically complete the identification of large-scale positioning point arrays in complex optical distortion environments.

[0080] For example, such as Figure 4 As shown, point M is the camera pixel coordinate corresponding to the mapped positioning point of the second target, point N is the camera pixel coordinate corresponding to the mapped positioning point of the first target, point B is the camera pixel coordinate corresponding to the unmapped positioning point of the target, point A is the camera pixel coordinate of the camera pixel to be corrected, the direction from M to N is the direction of the first camera pixel, and the distance MN between M and N is the distance between the first camera pixels.

[0081] When determining point A, in the pixel positioning camera image, starting from point N, extend the distance of the first camera pixel along the direction from M to N to obtain point A. At this time, the distance AN between A and N is equal to MN. Then, with point A as the center, a search window is generated with a radius of less than half of AN and greater than one-third of AN as the camera pixel distance. Based on the brightness information of each camera pixel in the search window, point B is obtained by weighted calculation using the centroid method.

[0082] In one feasible implementation, step S630 above may include steps S631 to S632: Step S631: Based on the camera pixels to be corrected, determine the image area corresponding to the unmapped positioning point of the target from the pixel positioning camera image, and extract the brightness information corresponding to each camera pixel in the image area from the pixel positioning camera image. It should be noted that, in this embodiment, the image point imaging region is a local image block defined on the pixel-localization camera image, centered on the pixel coordinates of the camera to be corrected (i.e., the predicted coarse position), which is the search window in the above embodiment. The size of this image point imaging region is usually slightly larger than the actual imaging size of the localization point (such as a bright spot) in the camera image (i.e., the influence range of the point spread function), for example, defining a 5x5 or 7x7 pixel square area. This region aims to include as completely as possible the actual brightness distribution (i.e., "image point") formed on the sensor after the unmapped localization point of the target is imaged by the optical system, while avoiding the introduction of brightness interference from other localization points as much as possible.

[0083] In this embodiment, the system first determines the search range (i.e., the image area of ​​the image point) based on the camera pixels to be calibrated, and then extracts the brightness values ​​of all camera pixels within that area from the original pixel-localized camera image. This process obtains raw data describing the actual energy spatial distribution of the image point at that localization point, providing a direct physical quantity input for subsequent weighted centroid calculation.

[0084] Step S632: Based on the brightness information corresponding to each camera pixel in the image point imaging region, determine the brightness weight corresponding to each camera pixel in the image point imaging region, and based on the brightness weight, determine the camera pixel corresponding to the unmapped target positioning point from the image point imaging region using the centroid method.

[0085] In this embodiment, brightness weight refers to a weight value assigned to each camera pixel within the image area of ​​the image point. This weight is directly related to its brightness information (brightness value), and the higher the brightness, the greater the weight. An intuitive and commonly used method is to directly use the brightness value of the pixel (or the value after subtracting a reference dark level) as its weight. That is, the brighter the pixel, the greater its contribution to the positioning calculation, because the pixel receives a stronger light signal from the target positioning point.

[0086] After determining the brightness weight of each camera pixel in the image region, the system uses the centroid method (or gray-scale centroid method) to calculate the precise center position of the image point, which is used as the camera pixel coordinates corresponding to the unmapped positioning point of the target.

[0087] This implementation, through steps S631 and S632, concretizes the operation of "precisely locating image points near the predicted position" into "weighted centroid calculation based on local brightness distribution." Its technical advantages are as follows: First, by using a brightness-weighted approach, it makes the positioning process sensitive to the true energy distribution of the image points, rather than relying solely on pixel-level edge or peak detection. This makes it more robust to image point shape changes (such as ellipticization due to defocus or astigmatism) and brightness non-uniformity. Second, the centroid method has a simple calculation principle and low computational cost, enabling highly efficient sub-pixel-level positioning while meeting real-time processing requirements. Finally, this positioning method based on physical brightness information theoretically has accuracy limited only by the image's signal-to-noise ratio and the sampling theorem. By using a high-quality image sensor and appropriate exposure control, extremely stable and repeatable sub-pixel positioning results can be obtained. This provides extremely high precision for individual corresponding point pairs in the entire mapping relationship establishment process, serving as the cornerstone for ensuring the overall accuracy of the final pixel-level mapping relationship.

[0088] In one feasible implementation, step S500, which involves determining the first mapped target location point and the second mapped target location point corresponding to the unmapped target location point from the mapped location points, may include steps S510-S520: Step S510: Detect whether there is a third target mapped positioning point corresponding to the target unmapped positioning point among the mapped positioning points, wherein the third target mapped positioning point is a positioning point among the mapped positioning points that is adjacent to the second target mapped positioning point and is on the same straight line as the second target mapped positioning point, the first target mapped positioning point and the target unmapped positioning point; It should be noted that, in this embodiment, the mapped location of the third target is the key to determining whether a higher-precision prediction model can be used.

[0089] This step involves the adaptive selection of logical branch points by the prediction model. When attempting to select a reference point for the current unmapped target location point C, the system not only searches for its adjacent mapped point (i.e., the first target mapped location point A) and the mapped point on the other side of that adjacent point (i.e., the second target mapped location point B), but also further checks whether there exists a third target mapped location point D on the other side of the second target mapped location point B (i.e., in the direction away from C) that is mapped and collinear with B, A, and C. If such a point D exists, it means that we have more complete local distortion information about this line in the camera image (given the three points A, B, and D), thus enabling the construction of a more robust prediction model.

[0090] Step S520: If there is no third target mapped location point corresponding to the unmapped target location point among the mapped location points, determine the first target mapped location point and the second target mapped location point corresponding to the unmapped target location point from the mapped location points. This step defines a fallback to the basic prediction mode when the higher-order condition of "three points are collinear and known" in step S510 is not met. In this case, the system uses only two known points (A and B) as references and determines the camera pixels to be corrected for the unmapped target location point C using the "two-point linear extrapolation" method described in the aforementioned implementation (steps S610-S630). This typically occurs in the early stages of the recognition process or near the line endpoints.

[0091] Following step S510 above, the method further includes steps S530 to S580: Step S530: If there is a third target mapped location point corresponding to the target unmapped location point among the mapped location points, determine the first target mapped location point, the second target mapped location point, and the third target mapped location point corresponding to the target unmapped location point from the mapped location points. In this embodiment, when step S510 detects the existence of a third target mapped location point D, the system confirms entry into the higher-order prediction mode. At this time, the set of known reference points used to predict point C expands from {A, B} to {A, B, D}. These four points are strictly collinear in screen space, in the order D — B — A — C. Among them, D, B, and A are known points (mapped), and C is the point to be predicted (the target is not mapped).

[0092] Step S540: Based on the camera pixels corresponding to the first target mapped positioning point and the second target mapped positioning point, determine the first camera pixel distance and the first camera pixel direction between the first target mapped positioning point and the second target mapped positioning point; Step S550: Based on the camera pixels corresponding to the second target mapped positioning point and the third target mapped positioning point, determine the second camera pixel distance and the second camera pixel direction between the second target mapped positioning point and the third target mapped positioning point; It should be noted that in this embodiment, steps S540 and S550 are executed in parallel, aiming to quantify the local distortion characteristics of two segments of a known line segment in the camera image. The first camera pixel distance and the first camera pixel direction (based on A and B) describe the local distortion characteristics from B to A. The second camera pixel distance and the second camera pixel direction (based on B and D) describe the local distortion characteristics from D to B. By comparing these two local distortion characteristics, the local distortion characteristics from A to C can be inferred, namely the third camera pixel distance and the third camera pixel direction.

[0093] Step S560: Calculate the third camera pixel distance based on the first camera pixel distance and the second camera pixel distance; This step predicts the distortion distance from point A to the desired point C (i.e., the third camera pixel distance) based on two known distortion distances (i.e., the first camera pixel distance and the second camera pixel distance). A typical calculation method is interpolation or extrapolation. For example, assuming the screen pixel spacing is uniform (AB distance equals BC distance) and the distortion changes approximately linearly locally, the third camera pixel distance (the predicted distance from A to C) can be taken as the first camera pixel distance (the distance from A to B). A more refined approach, considering the possibility of nonlinearity, can be based on a weighted average or curve fitting of the trends reflected by the first and second camera pixel distances (the distance from D to B) to make the prediction. For example, if the first camera pixel distance is greater than the second camera pixel distance, indicating that the distortion distance from A to C is increasing, the predicted value of the third camera pixel distance can be appropriately increased, for example, so that the third camera pixel distance, the first camera pixel distance, and the second camera pixel distance form a geometric or arithmetic sequence.

[0094] Step S570: Calculate the third camera pixel direction based on the first camera pixel direction and the second camera pixel direction; This step is similar to step S560 above, and will not be repeated here.

[0095] Step S580: Based on the camera pixel corresponding to the first target mapped positioning point, the distance of the third camera pixel, and the direction of the third camera pixel, determine the camera pixel to be corrected corresponding to the target unmapped positioning point, and execute the step of determining the camera pixel corresponding to the target unmapped positioning point from the pixel positioning camera image based on the camera pixel to be corrected.

[0096] This implementation starts with the camera coordinates of the mapped positioning point A of the first target, and extends along the calculated direction of the third camera pixel by the distance of the third camera pixel to calculate the camera pixel coordinates to be corrected for the unmapped positioning point C of the target. This prediction model, by utilizing the local distortion "history" and "trend" revealed by three known points (D, B, A), is theoretically more accurate and robust than simple extrapolation using only two points (A, B), especially in regions with strong distortion nonlinearity. Subsequently, the system uses this camera pixel to be corrected as the center to define a small search window in the pixel-localized camera image, and uses sub-pixel localization techniques such as the centroid method to accurately determine the final camera pixel coordinates corresponding to point C.

[0097] It should be noted that the above embodiments and implementation methods are only used to assist in understanding this application and do not constitute a limitation on the brightness extraction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0098] In addition, please refer to Figure 5 , Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the brightness extraction method in this application embodiment.

[0099] This application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the brightness extraction method in the above embodiments.

[0100] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application.

[0101] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays, speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tape, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0102] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0103] The electronic device provided in this application, employing the brightness extraction method described in the above embodiments, solves the technical problem of reliably and efficiently acquiring accurate brightness information that perfectly matches the original pixel distribution of the screen when the camera image suffers severe distortion due to optical system distortion. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the brightness extraction method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0105] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the above claims.

[0106] In addition, this application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the steps of the brightness extraction method in the above embodiments.

[0107] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0108] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0109] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: upon detecting an instruction to extract brightness from the display screen of a near-eye display device via a target camera, determine a mapping relationship between camera pixels of the target camera and screen pixels of the display screen, wherein the target camera is located at the exit pupil frame of the near-eye display device; display a calibration brightness image on the display screen and capture a calibration brightness camera image corresponding to the calibration brightness image using the target camera; and extract brightness information corresponding to each screen pixel on the display screen from the calibration brightness camera image based on the mapping relationship.

[0110] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0113] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for performing the steps of the above-described brightness extraction method. This solves the technical problem of reliably and efficiently obtaining accurate brightness information that perfectly matches the original pixel distribution of the screen when the camera image suffers severe distortion due to optical system distortion. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the brightness extraction method provided in the above embodiments, and will not be repeated here.

[0114] Furthermore, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the brightness extraction method described in the above embodiments.

[0115] The computer program product provided in this application solves the technical problem of reliably and efficiently acquiring accurate brightness information that perfectly matches the original pixel distribution of the screen when camera images suffer severe distortion due to optical system distortion. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the brightness extraction method provided in the above embodiments, and will not be repeated here.

[0116] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A brightness extraction method, characterized in that, The method includes: When an instruction to extract brightness from the display screen of a near-eye display device via a target camera is detected, the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen is determined, wherein the target camera is located at the exit pupil frame of the near-eye display device; A calibration brightness image is displayed on the display screen, and a calibration brightness camera image corresponding to the calibration brightness image is captured by the target camera. Based on the mapping relationship, the brightness information corresponding to each screen pixel on the display screen is extracted from the calibrated brightness camera image.

2. The method as described in claim 1, characterized in that, The step of determining the mapping relationship between the camera pixels of the target camera and the screen pixels of the display screen includes: A pixel positioning image is displayed on the display screen, and a pixel positioning camera image corresponding to the pixel positioning image is captured by the target camera. The pixel positioning image includes a positioning point array composed of multiple positioning points. Determine the screen pixel corresponding to each positioning point in the positioning point array, and locate the camera image through the pixel to determine the camera pixel corresponding to each positioning point in the positioning point array. Based on the screen pixels and camera pixels corresponding to each positioning point in the positioning point array, a mapping relationship is established between the camera pixels of the target camera and the screen pixels of the display screen.

3. The method as described in claim 2, characterized in that, The step of determining the camera pixel corresponding to each positioning point in the positioning point array by locating the camera image through the pixels includes: From the pixel positioning camera image, the camera pixels corresponding to the first positioning point and the second positioning point are determined, wherein the first positioning point is the center positioning point of the positioning point array, and the second positioning point is the positioning point in the positioning point array that is adjacent to the first positioning point. Based on the camera pixels corresponding to the first and second positioning points, the camera pixels corresponding to the third positioning point are determined from the pixel-positioned camera image, wherein the third positioning point is a positioning point other than the first and second positioning points in the positioning point array.

4. The method as described in claim 3, characterized in that, The step of determining the camera pixel corresponding to the third positioning point from the pixel-localized camera image based on the camera pixels corresponding to the first positioning point and the second positioning point includes: The first and second positioning points are determined as mapped positioning points, and the third positioning point is determined as an unmapped positioning point; The target unmapped location point is determined from the unmapped location points, and the first target mapped location point and the second target mapped location point corresponding to the target unmapped location point are determined from the mapped location points. The target unmapped location point is the location point adjacent to the mapped location point among the unmapped location points, the first target mapped location point is the location point adjacent to the target unmapped location point among the mapped location points, and the second target mapped location point is the location point adjacent to the first target mapped location point among the mapped location points and on the same straight line as the first target mapped location point and the target unmapped location point. Based on the camera pixels corresponding to the first target mapped positioning point and the second target mapped positioning point, determine the camera pixels corresponding to the unmapped positioning point of the target from the pixel positioning camera image; The unmapped target positioning point is determined as a mapped positioning point, and the process returns to the step of determining the unmapped target positioning point from the unmapped positioning points, until there are no more unmapped positioning points, thus obtaining the camera pixels corresponding to all third positioning points.

5. The method as described in claim 4, characterized in that, The step of determining the camera pixel corresponding to the unmapped target positioning point from the pixel-localized camera image based on the camera pixels corresponding to the first mapped target positioning point and the second mapped target positioning point includes: Based on the camera pixels corresponding to the first target mapped positioning point and the second target mapped positioning point, determine the first camera pixel distance and the first camera pixel direction between the first target mapped positioning point and the second target mapped positioning point; Based on the camera pixels corresponding to the first target mapped positioning point, the distance of the first camera pixels, and the orientation of the first camera pixels, determine the camera pixels to be corrected corresponding to the unmapped positioning point of the target; Based on the camera pixels to be corrected, the camera pixels corresponding to the unmapped positioning points of the target are determined from the pixel-localized camera image.

6. The method as described in claim 5, characterized in that, The step of determining the camera pixel corresponding to the unmapped target positioning point from the pixel-localized camera image based on the camera pixel to be corrected includes: Based on the camera pixels to be corrected, the image region corresponding to the unmapped positioning point of the target is determined from the pixel positioning camera image, and the brightness information corresponding to each camera pixel in the image region is extracted from the pixel positioning camera image. Based on the brightness information corresponding to each camera pixel in the image point imaging region, the brightness weight corresponding to each camera pixel in the image point imaging region is determined, and based on the brightness weight, the camera pixel corresponding to the unmapped positioning point of the target is determined from the image point imaging region by the centroid method.

7. The method as described in claim 6, characterized in that, The step of determining the first and second mapped target positioning points corresponding to the unmapped target positioning points from the mapped positioning points includes: Detect whether there is a third target mapped positioning point corresponding to the target unmapped positioning point among the mapped positioning points, wherein the third target mapped positioning point is a positioning point among the mapped positioning points that is adjacent to the second target mapped positioning point and is on the same straight line as the second target mapped positioning point, the first target mapped positioning point and the target unmapped positioning point; If there is no third target mapped location point corresponding to the unmapped target location point among the mapped location points, determine the first target mapped location point and the second target mapped location point corresponding to the unmapped target location point from the mapped location points; After the step of detecting whether there is a third target mapped location point corresponding to the unmapped target location point among the mapped location points, the method further includes: If a third target mapped location point corresponding to the unmapped target location point exists among the mapped location points, the first target mapped location point, the second target mapped location point, and the third target mapped location point corresponding to the unmapped target location point are determined from the mapped location points. Based on the camera pixels corresponding to the first target mapped positioning point and the second target mapped positioning point, determine the first camera pixel distance and the first camera pixel direction between the first target mapped positioning point and the second target mapped positioning point; Based on the camera pixels corresponding to the second target mapped positioning point and the third target mapped positioning point, determine the second camera pixel distance and the second camera pixel direction between the second target mapped positioning point and the third target mapped positioning point; The third camera pixel distance is calculated based on the pixel distance of the first camera and the pixel distance of the second camera; The third camera pixel direction is calculated based on the first camera pixel direction and the second camera pixel direction; Based on the camera pixels corresponding to the first target mapped positioning point, the distance to the third camera pixels, and the direction of the third camera pixels, determine the camera pixels to be corrected corresponding to the unmapped positioning point of the target, and execute the step of determining the camera pixels corresponding to the unmapped positioning point of the target from the pixel positioning camera image based on the camera pixels to be corrected.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the brightness extraction method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the brightness extraction method as described in any one of claims 1 to 7.

10. A program product, characterized in that, The program product is a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the brightness extraction method as described in any one of claims 1 to 7.

Images