Display system and display control method and apparatus thereof, medium, and vehicle

Abstract

The present disclosure provides a display system and a display control method and apparatus thereof, a medium, and a vehicle. The display control method comprises: on the basis of depth information of an image to be displayed, dividing said image into image regions corresponding to projection surfaces formed by a display portion, wherein the projection surfaces formed by the display portion are successively arranged from a proximal end close to the position of an eye box to a distal end far away from the position of the eye box; and controlling each image region to be displayed on a corresponding projection surface.

Description

Display system, display control method, device, medium and vehicle thereof CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to a Chinese patent application filed with the Patent Office of China on January 22, 2024, with application number 2024100920360 and invention name “Display system and display control method, device, medium and vehicle thereof”. The contents of the Chinese patent application are hereby incorporated into this application by reference. Technical Field

[0002] The present disclosure relates to the field of image display technology, and more specifically to a display system and a display control method, device, medium, and vehicle thereof. Background Art

[0003] A head-up display (HUD) device projects light from an image source onto an imaging window (e.g., an imaging board, windshield, etc.) through, for example, a reflective optical design, to display vehicle status information such as speed and fuel level, as well as navigation, hazard warnings, and other indication information at an appropriate location in front of the driver. This allows the driver to obtain relevant information such as speed and fuel level without shifting their line of sight from the road ahead, thereby improving driving safety and the driving experience. Technical content

[0004] The embodiments of the present disclosure provide a display system and a display control method, device, medium, and vehicle thereof; these can increase the vividness and immersion of the driver's visual effects and improve the user experience.

[0005] The technical solution of the embodiment of the present disclosure is achieved as follows:

[0006] In a first aspect, the present disclosure provides a display control method, the method comprising:

[0007] Based on depth information of the image to be displayed, the image to be displayed is divided into image areas corresponding to the projection surface formed by the display unit; wherein the projection surface formed by the display unit is arranged in sequence from a proximal end on a side close to the eye box position to a distal end on a side away from the eye box position;

[0008] Each of the image areas is controlled to be displayed on a corresponding projection surface.

[0009] In a second aspect, the present disclosure provides a display control device, comprising: a dividing part and a control part; wherein,

[0010] The dividing portion is configured to divide the image to be displayed into image areas corresponding to the projection surface formed by the display unit based on depth information of the image to be displayed; wherein the projection surface formed by the display unit is arranged in sequence from a proximal end on a side close to the eye box position to a distal end on a side away from the eye box position;

[0011] The control part is configured to control each of the image areas to be displayed on the corresponding projection surface.

[0012] In a third aspect, the present disclosure provides a display control device, comprising: a processor and a memory; the processor is configured to execute instructions stored in the memory to implement the display control method as described in the first aspect.

[0013] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction is used to be executed by a processor to implement the display control method as described in the first aspect.

[0014] In a fifth aspect, the present disclosure provides a display system, comprising a display control unit and a display unit; wherein,

[0015] The display control unit is configured to divide the image to be displayed into image areas corresponding to the projection surface formed by the display unit based on the depth information of the image to be displayed;

[0016] and controlling each of the image areas to be displayed on a corresponding projection surface;

[0017] The display unit is configured to form projection surfaces arranged in sequence from a proximal end on a side close to the eye box position to a distal end on a side away from the eye box position; and each of the image areas is displayed on the corresponding projection surface based on the control of the display control unit.

[0018] In a sixth aspect, the present disclosure provides a vehicle comprising the display system described in the fifth aspect.

[0019] The present disclosure provides a display system and a display control method, device, medium and vehicle thereof; based on the number of projection surfaces formed by a display unit and distributed from near to far distances from the viewer, image data to be displayed is divided into image areas corresponding to the projection surfaces according to depth information, and then displayed using the projection surfaces, thereby increasing the vividness and immersion of the viewer's visual effects when viewing the image data and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. The drawings described below are only exemplary embodiments of the present disclosure.

[0021] FIG1 is a schematic diagram of the architecture of a display system provided by the present disclosure.

[0022] FIG. 2 is an exemplary top view of a vehicle provided by the present disclosure.

[0023] FIG. 3 is an exemplary perspective view from a driver's seat of a vehicle provided by the present disclosure.

[0024] FIG4 is a flow chart of a display control method provided by the present disclosure.

[0025] FIG5(A) is a schematic diagram of an image for marking buildings in a street scene provided by the present disclosure.

[0026] FIG5(B) is a schematic diagram of a two-dimensional logo image provided by the present disclosure.

[0027] FIG6 (A) is a side view schematic diagram of an image area provided by the present disclosure displayed on a projection surface.

[0028] FIG6(B) is a schematic diagram of an image region provided by the present disclosure.

[0029] FIG7 is a schematic oblique view of an image area provided by the present disclosure displayed on a projection surface.

[0030] FIG8 is a schematic diagram of a process of dividing an image area corresponding to a projection surface provided by the present disclosure.

[0031] FIG9(A) is a schematic diagram of an image to be displayed provided by the present disclosure.

[0032] FIG9(B) is a schematic diagram of a target object provided by the present disclosure.

[0033] FIG10 is a schematic diagram of a process for obtaining an estimated average depth information value of each target area provided by the present disclosure.

[0034] FIG11 is a schematic diagram of another image region provided by the present disclosure.

[0035] FIG12(A) is a side view schematic diagram of another image area provided by the present disclosure displayed on a projection surface.

[0036] FIG12(B) is a schematic oblique view of another image region provided by the present disclosure displayed on a projection surface.

[0037] FIG13 is a schematic diagram of another process of dividing the image area corresponding to the projection surface provided by the present disclosure.

[0038] FIG14(A) is a schematic diagram of another image to be displayed provided by the present disclosure.

[0039] FIG14(B) is another schematic diagram of dividing an image to be displayed into image regions provided by the present disclosure.

[0040] FIG15 is a schematic diagram of another image region provided by the present disclosure.

[0041] FIG16(A) is a side view schematic diagram of another image area provided by the present disclosure displayed on a projection surface.

[0042] FIG16(B) is a schematic oblique view of another image region provided by the present disclosure displayed on a projection surface.

[0043] FIG17(A) is a schematic diagram of a virtual three-dimensional camera capture image provided by the present disclosure.

[0044] FIG17(B) is a schematic diagram of an image area seen by a viewer provided by the present disclosure.

[0045] FIG18(A) is a schematic diagram of the image content before the gaze point moves provided by the present disclosure.

[0046] FIG18(B) is a schematic diagram of the image content after the gaze point moves according to the present disclosure.

[0047] FIG19 is a schematic diagram showing the composition of a display control device provided by the present disclosure.

[0048] FIG20 is a schematic structural diagram of a display control device provided by the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the following will describe in detail exemplary embodiments of the present disclosure with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0050] Referring to FIG. 1 , an exemplary implementation architecture of a display system 10 provided by the present disclosure is shown. The display system 10 includes a display control unit 110, a display unit 120, and a presentation component 130. The display control unit 110 can process received data to generate display information to be displayed, and project the display information to the presentation component 130 via the display unit 120 for display. Specifically, after receiving data 320, the display control unit 110 generates a signal 312. The display unit 120 may include a light source 121 and an optical path component 122. Based on the signal 312 from the display control unit 150, the light source 121 outputs light (e.g., a virtual image) for display on the presentation component 130. For example, the light source 121 may include one or more lasers and output red, green, and blue light.

[0051] The optical path assembly 122 can reflect the output of the light source 121 onto the display assembly 130, allowing viewers to view the display information 212 in the display area projected onto the display assembly 130. In some examples, the optical path assembly 122 can include one or more reflectors (plane mirrors) and concave mirrors (magnifying glasses). In some examples, the optical path assembly 122 can reflect the output of the light source 121 onto the display assembly 130 in the form of two or more (e.g., N) optical paths (such as optical paths L-1, ..., LN in FIG1 ), forming a virtual image 30 in front of the viewer. Based on the control of the display control unit 110, the virtual image 30 can be projected onto one of the N projection surfaces 31-1, ..., 31-N corresponding to the N optical paths (optical paths L-1, ..., LN) for display. In some examples, the optical path assembly 122 can polarize the output of the light source 121 to form two or more (e.g., N) optical paths. Depending on the application scenario, the image display assembly 13 will also vary. For example, when the application scenario is a movie projection in a theater, the image display component 13 is a projection screen or a display screen. When the application scenario is to display information on the windshield of a vehicle, the image display component 13 is the windshield of the vehicle.

[0052] In some examples, these N projection surfaces (also referred to as focal planes) are distributed from the viewer's viewpoint (also referred to as the eye point or eye box) in a range of 5 to 17 meters from the viewer's viewpoint (also referred to as the eye point or eye box position) in order of distance from the viewer. In some examples, the display control unit 110 can control the display unit 120 to display the virtual image 30 on the projection surface farther or closer to the viewer's viewpoint, so that the viewer can experience the visual effect of an AR display when observing.

[0053] In this disclosure, the display system 10 shown in FIG. 1 is used to display information on the windshield of a vehicle. See FIG. 2 and FIG. 3 , which respectively show an exemplary top view of the vehicle and an exemplary perspective view from the driver's seat of the vehicle. In this disclosure, the vehicle is equipped with a display system 10 (see FIG. 1 ) that is applicable to the technical solution of this disclosure. In some examples, the vehicle may be an internal combustion engine vehicle powered by an engine, a hybrid vehicle powered by an engine and an electric motor, an electric vehicle powered by an electric motor, or other types of vehicles. In the remainder of this specification, the vehicle equipped with the display system 10 will be referred to as the present vehicle. In FIG. 2 , the present vehicle includes a windshield 204 located at the front of the vehicle. The driver and passengers in the passenger cabin 208 of the present vehicle can see the front of the present vehicle through the windshield 204.

[0054] 2 , the windshield 204 is visually positioned above the vehicle dashboard 206. The driver can turn the steering wheel 210 within the passenger compartment 208 to steer the vehicle, such as to change lanes, merge, and park the vehicle. In some embodiments, the steering wheel 210 can be retracted or omitted.

[0055] In FIG2 , the display system 10 projects display information 212 (e.g., a virtual image) onto a portion of the windshield 204 through one or more apertures (e.g., aperture 216 ) in the instrument panel 206 . Specifically, the windshield 204 serves as the image display component 13 of the display system 10 . Although FIG3 illustrates an example size of the display information 212 , the display information 212 can be presented over a larger or smaller area. Examples of display information 212 include various vehicle information and streaming information such as images and videos provided by an in-vehicle entertainment system (not shown). The display system 10 provides this information to the vehicle driver, an example viewer, without the driver having to look away from objects in front of the vehicle. In some examples, when the display system 10 shown in FIG1 is used to display information on a vehicle's windshield, the display system 10 may also be referred to as a heads-up display (HUD) system. Taking the display system 10 as an example, which is a HUD system, after activating the vehicle's main control switch (power on), as long as the parking P gear or the brake has not been released for the first time, that is, after power on, the brake has not been released for the first time, the HUD system may be allowed to display information not related to driving, such as the aforementioned streaming information such as images and videos.

[0056] In the aforementioned display control scheme, although the virtual image 30 is displayed on projection surfaces at different distances from the driver's viewpoint to produce an AR display visual effect, the virtual image 30 is still only displayed on a single projection surface, resulting in a less vivid and immersive visual effect. Therefore, the present disclosure contemplates dividing the image data to be displayed into image areas corresponding to the projection surfaces based on depth information based on the number of projection surfaces formed by the display unit 120, distributed from near to far distance from the viewer, and then displaying them on the projection surfaces. This increases the vividness and immersion of the driver's visual effect and improves the user experience.

[0057] Referring to FIG. 4 , a display control method provided by the present disclosure is shown. The method can be applied to the display system 10 shown in FIG. 1 , and in particular to the display control unit 110 in the display system 10 . The method includes steps S401 to S402 .

[0058] In step S401 , based on the depth information of the image to be displayed, the image to be displayed is divided into image areas corresponding to the projection surface formed by the display unit.

[0059] In some examples, as shown in FIG. 1 , the projection surfaces formed by the display portion 120 are arranged in sequence from a proximal end on a side close to the eye box position to a distal end on a side away from the eye box position.

[0060] In some examples, the image to be displayed may be image data provided by other in-vehicle systems on the vehicle, such as a navigation system, an infotainment system, etc., or may be each image frame of video data provided by these in-vehicle systems. Specifically, these in-vehicle systems may transmit the image to be displayed to the display control unit 110 in the display system 10 via a system bus within the vehicle.

[0061] In some examples, the image to be displayed may be transmitted to the display control unit 110 in the display system 10 by other devices not mounted on the vehicle. For example, a mobile terminal carried by the driver or passenger of the vehicle may be connected to the display system 10 of the vehicle via a wired or wireless connection. Based on these connections, the mobile terminal may transmit the image to be displayed to the display control unit 110 in the display system 10.

[0062] In the above examples, the image to be displayed can be a two-dimensional image or a depth image with depth information. Depth information is represented by the distance between elements within the image and the image collector, using the eyebox as the image collector. In some examples, based on the granularity of image analysis, elements within the image can be pixels within the image or objects represented in the image, etc., although this disclosure does not impose specific limitations on this.

[0063] For a two-dimensional image, although all elements within the image are located on the same plane, based on the actual distance information present during the image acquisition process, the elements within the two-dimensional image will reflect their corresponding distance information through certain specific image features. These specific image features can be called clues in some examples, such as linear perspective, focus / defocus, atmospheric scattering, shadows, texture, occlusion, relative height, and motion clues. Based on these specific image features, the depth information of the elements within the two-dimensional image can be estimated.

[0064] Furthermore, in some exemplary two-dimensional image generation processes, the distance information actually present during image acquisition corresponds to elements within the image. For example, as shown in FIG5(A), using a vehicle HUD system as an example, in an image of building markers in a street scene, the distance information between each building and the image collector can be obtained using map information after the navigation system locates the image collector. In the two-dimensional marker image shown in FIG5(B), each marker has corresponding distance information from the image collector based on its correspondence with the building. For example, in FIG5(A), the building indicated by marker 1 is closest to the image collector, followed by the building indicated by marker 2, followed by the building indicated by marker 3 and the building indicated by marker 4, and the building indicated by marker 5 is farthest from the image collector. Therefore, for the image including building markers shown in FIG5(B), depth information of elements within the two-dimensional image can be directly obtained based on the aforementioned correspondences without having to estimate depth information based on image features.

[0065] For a depth image, each pixel of the image includes not only RGB channel values ​​but also a distance channel value related to the distance between image collectors. The distance channel value can be considered as the depth information of the depth image.

[0066] Based on the depth information of the image to be displayed, the image to be displayed can be divided into the same number of image regions as the number of projection surfaces formed by the display unit 120 shown in FIG1 . Since these image regions are divided based on depth information, and the eyebox position is considered to be the position of the image collector, these image regions are also arranged sequentially from the proximal end of the side close to the eyebox position to the distal end of the side away from the eyebox position, consistent with the arrangement order of the projection surfaces. Based on this consistency, a corresponding projection surface can be determined for each image region across all projection surfaces.

[0067] For example, assume that the number of projection surfaces formed by the display system shown in Figure 1 is five. As shown in Figure 6(A), these projection surfaces are arranged in order from the proximal end near the eyebox position 60 to the distal end away from the eyebox position 60 and are labeled 31-1, 31-2, 31-3, 31-4, and 31-5, respectively. Taking Figures 5(A) and 5(B) as an example, the distance between the buildings indicated by the respective markers and the image collector corresponds to the number of projection surfaces and is divided into five image regions, as shown in Figure 6(B). In Figure 6(B), these image regions are labeled 51, 52, 53, 54, and 55, respectively, from near to far, based on their distance from the image collector. The correspondence between the image regions and the projection surfaces is as follows: image region 51 corresponds to projection surface 31-1, image region 52 corresponds to projection surface 31-2, image region 53 corresponds to projection surface 31-3, image region 54 corresponds to projection surface 31-4, and image region 55 corresponds to projection surface 31-5.

[0068] It should be noted that when the image to be displayed is a depth image, since the depth image carries a distance channel value that can be used as depth information, the depth image can be divided into multiple image areas with distances from the image collector from near to far according to the number of projection surfaces based on the distance channel value, and the corresponding projection surface is determined for each image area in the order of the distances from the image collector from near to far. The specific implementation process is the same as that shown in Figures 5 (A), 5 (B), 6 (A) and 6 (B), and will not be repeated in this disclosure.

[0069] In step S402, each image area is controlled to be displayed on the corresponding projection surface.

[0070] In the present disclosure, referring to the examples shown in Figures 5(A), 5(B), 6(A), and 6(B), and referring to Figure 7, after determining a corresponding projection surface for each image region, each image region is projected onto the corresponding projection surface by controlling the multiple light paths provided by the optical path assembly 122 in the display unit 120. When a viewer observes the displayed image, this provides a more vivid and immersive visual experience compared to a display solution in which a two-dimensional image is projected onto a single projection surface.

[0071] It should be noted that when the image to be displayed is a depth image, since the depth image carries a distance channel value that can be used as depth information, the depth image can be divided into multiple image regions with distances from the image collector from near to far according to the number of projection surfaces based on the distance channel value, and the corresponding projection surface is determined for each image region in the order of the distances from the image collector from near to far. The specific implementation process is the same as that shown in Figures 5 (A), 5 (B), 6 (A) and 6 (B), and will not be repeated in this disclosure. After the depth image is divided into image regions corresponding to the projection surfaces, each image region can still be projected onto the corresponding projection surface by controlling the multiple light paths provided by the optical path component 122 in the display unit 120.

[0072] The technical solution shown in Figure 4 is based on the number of projection surfaces formed by the display unit 120 and distributed from near to far distances from the viewer. After the image data to be displayed is divided into image areas corresponding to the projection surfaces according to depth information, the projection surfaces are used for display, which increases the vividness and immersion of the visual effects of the viewer when viewing the image data, thereby improving the user experience.

[0073] For the technical solution shown in Figure 4, in some possible implementations, when the image to be displayed is a two-dimensional image, since the elements in the two-dimensional image reflect the distance information between them and the image collector during the image acquisition process through specific image features, the present disclosure uses these image features as clues to estimate the depth information of the elements in the two-dimensional image. The estimated value can more accurately characterize the relative depth information between each element and the image collector. After obtaining the relative depth information, the image to be displayed can be divided into image areas consistent with the number of projection surfaces according to the relative distance between them and the image collector, and the projection surface corresponding to each image area can be determined based on the relative distance between them and the image collector.

[0074] In the above implementation, with a larger image analysis granularity, the elements in the image to be displayed may be the target objects presented in the image. Based on this, in some examples, referring to FIG8 , the process of dividing the image to be displayed into image regions corresponding to the projection surface formed by the display unit based on the depth information of the image to be displayed includes steps S801 to S804.

[0075] In step S801 , a target area and a background area in an image to be displayed are obtained.

[0076] In the present disclosure, taking the image to be displayed shown in FIG. 9(A) as an example, target recognition can be performed on the image to be displayed to obtain target objects present in the image to be displayed. Each target object corresponds to a target region. Unrecognizable target objects in the image to be displayed, or image content in the image to be displayed that does not belong to any target object, are considered to be background regions. In the present disclosure, each target region can be considered an image region, while the background region is considered to be the image region farthest from the image collector during the image acquisition process. In other words, the projection surface corresponding to the background region is the projection surface farthest from the eye box position. In conjunction with the example image to be displayed shown in FIG. 9(A), the target object is shown as the target selected by the solid line in FIG. 9(B). Since the projection surface formed by the display unit 120 corresponds to the target region and the background region, in the present disclosure, the number of target regions is the number of projection surfaces minus one. Assuming the number of projection surfaces in this example is 4, the number of target regions is 3.

[0077] In step S802 , an average depth information estimation value of each target area is obtained according to the depth information estimation value of each pixel in each target area.

[0078] In this example, depth information is estimated for each target area using transmittance as a clue. Specifically, as shown in FIG10 , obtaining an average depth information estimation value of each target area based on the depth information estimation value of each pixel in each target area in step S802 includes:

[0079] S8021: Obtaining the pixel transmittance of each pixel in each target area according to the global atmospheric light value of the image to be displayed;

[0080] Regarding step S8021, in the specific implementation process, for the i-th target area O among all target areas i , first we can get the i-th target area O i The dark channel value of each pixel in . For example, each pixel includes RGB channel values. The channel corresponding to the minimum value of these three channel values ​​can be used as the dark channel of the pixel. Correspondingly, the minimum value of these three channel values ​​is also the dark channel value of the pixel.

[0081] Next, the global atmospheric light value of the image to be displayed is obtained based on the maximum value of the dark channel values ​​of all pixels in the background area. For example, within the background area, the dark channel values ​​of all pixels in the background area can be counted, and the maximum value of these dark channel values ​​is used as the global atmospheric light value of the image to be displayed.

[0082] Finally, according to the i-th target area O iThe dark channel value of each pixel in the image and the global atmospheric light value of the image to be displayed are used to obtain the i-th target area O i For example, after obtaining the global atmospheric light value A, for the i-th target area O i The pixel transmittance of the pixel can be obtained based on the following formula: :

[0083]

[0084] in, Represents the dark channel value of the p-th pixel.

[0085] S8022: Obtain the average transmittance of each target area according to the pixel transmittance of each pixel in each target area.

[0086] S8023: Determine an average depth information estimation value of each target area according to the average transmittance of each target area and the distance of the farthest projection surface from the eye box position side among all projection surfaces.

[0087] For step S8023, specifically, the transmittance of the elements in the image is mapped to the distance between them and the image collector. Since the background area is considered to be the image area farthest from the image collector, and the background area corresponds to the projection surface farthest from the eye box position, based on this, for the i-th target area O i For example, the distance between the farthest projection surface on one side of the eye box position and the mapping relationship can be used. To determine the i-th target area O i The average depth estimate of ,Right now:

[0088]

[0089] in, represents the i-th target area O i The mean transmittance.

[0090] It should be noted that although the average depth information estimation value of each target area obtained through the process shown in Figure 10 above cannot accurately obtain the distance between each target area and the image collector, it can accurately characterize the relative distance between each target area relative to the image collector. Through the characterization of this relative distance, it can provide a basis for determining the corresponding projection surface for each target area.

[0091] In step S803 , based on the average depth information estimation value of each target area, a corresponding projection surface is determined from all projection surfaces in order from the proximal end close to the eye box position to the distal end far from the eye box position.

[0092] In this example, in order to determine the projection surface corresponding to each target area, the average depth information estimation value of each target area can be compared with the distance to the projection surface, and the average depth information estimation value of the i-th target area O can be compared with the average depth information estimation value of the i-th target area O. i For example, when the distance of the projection surface closest to the average depth information estimation value of the target area is , the projection surface can be determined as the i-th target area O i Specifically, the step S803 of determining the corresponding projection surface from all projection surfaces in the order from the proximal end of the side close to the eye box position to the distal end of the side far from the eye box position based on the average depth information estimation value of each target area includes:

[0093] For the i-th target area O i , according to the distance F of the j-th projection surface j And the i-th target area O i The average depth information estimate D i , determine the i-th target area O according to the following formula i The corresponding projection surface identifier k:

[0094]

[0095] Where n represents the number of all projection surfaces.

[0096] In step S804, the projection surface farthest from the eye box position among all the projection surfaces is determined as the projection surface corresponding to the background area.

[0097] For the example shown in FIG8 , combined with the image to be displayed and the target object shown in FIG9 (A) and FIG9 (B), the acquired image regions are shown in FIG11 . Image regions 11-1, 11-2, and 11-3 are all target regions, and the average depth information estimated values ​​of image regions 11-1, 11-2, and 11-3 indicate that the target regions are gradually moving away from the image collector. Image region 11-4 is the background region. Furthermore, considering that the number of projection surfaces set in this example is 4, as shown in FIG12 (A) and FIG12 (B), from near to far from eye box position 60, image region 11-1 corresponds to projection surface 31-1, image region 11-2 corresponds to projection surface 31-2, image region 11-3 corresponds to projection surface 31-3, and image region 11-4 corresponds to projection surface 31-4.

[0098] In the example shown in FIG8 above, the image area is divided at a larger image analysis granularity, and a corresponding projection surface is determined for each image area. For smaller image analysis granularity, for example, the elements in the image to be displayed can be pixels in the image, and the image area can also be divided based on the depth information of each pixel, and the corresponding projection surface can be determined for each image area. Based on this, in some examples, referring to FIG13, the image to be displayed is divided into image areas corresponding to the projection surface formed by the display unit based on the depth information of the image to be displayed, including steps S1301 to S1303.

[0099] In step S1301 , depth information of each pixel in the image to be displayed is estimated.

[0100] Specifically, regarding step S1301, in addition to using transmittance as a clue for depth information estimation as in the example shown in FIG8 , the present disclosure can also perform depth information estimation based on other image features presented by each pixel, such as linear perspective, focus / defocus, atmospheric scattering, shadows, textures, occlusions, relative height, and motion clues. Furthermore, the depth information of each pixel can be estimated through deep learning, thereby estimating the depth information of each pixel based on the image to be displayed.

[0101] In step S1302 , a depth interval corresponding to the projection surface is determined.

[0102] In step S1302, due to the correspondence between the image area and the projection surface, in order to divide the image area according to the depth information, depth intervals corresponding to the projection surfaces can be set. For example, if the number of projection surfaces arranged in sequence from the proximal end near the eye box position to the distal end away from the eye box position is set to 8, then 8 depth intervals representing the depth gradually moving away from the image collector can be constructed. Each depth interval corresponds to a projection surface.

[0103] In step S1303 , all pixels in the image to be displayed are divided into image areas corresponding to the projection surface according to the depth interval of the depth information of each pixel.

[0104] In a specific implementation, each depth interval corresponds to an upper depth limit and a lower depth limit for the interval. By comparing the depth information of each pixel with the upper and lower depth limits of each depth interval, it is possible to determine the depth interval that each pixel is in. For example, taking pixel M as an example, when the depth information of the pixel is less than the upper depth limit of depth interval A and greater than the lower depth limit of depth interval A, it can be confirmed that the pixel M is in depth interval A.

[0105] In step S1303, after obtaining the eight depth intervals, each pixel in the image to be displayed can be traversed to determine the depth interval in which the depth information of each pixel lies. After traversing all pixels, all pixels in the same depth interval can be divided into the same image region. Each image region then corresponds to a depth interval. Combined with the correspondence between depth intervals and projection surfaces, each image region corresponds to a projection surface.

[0106] For the example shown in Figure 13, taking the image to be displayed shown in Figure 14 (A) as an example, the number of projection surfaces is set to 8, and the number of depth intervals is also set to 8. After estimating the depth information of all pixels in Figure 14, 8 image regions as shown in Figure 14 (B) can be divided according to the depth intervals in which the depth information of all pixels lies. As shown in Figure 15, these 8 image regions are respectively identified as image region 15-1, image region 15-2, image region 15-3, image region 15-4, image region 15-5, image region 15-6, image region 15-7, and image region 15-8, in order of increasing distance from the image acquisition device. In this example, the number of projection planes set to 8 is shown in FIG16(A). From the eyebox position 60, from near to far, image area 15-1 corresponds to projection plane 31-1, image area 15-2 corresponds to projection plane 31-2, image area 15-3 corresponds to projection plane 31-3, image area 15-4 corresponds to projection plane 31-4, image area 15-5 corresponds to projection plane 31-5, image area 15-6 corresponds to projection plane 31-6, image area 15-7 corresponds to projection plane 31-7, and image area 15-8 corresponds to projection plane 31-8. As shown in FIG16(B), starting from the eyebox position (i.e., the eyepoint), the image area closest to the image collector is displayed on the focal plane closest to the eyepoint, and the image area farthest from the image collector is displayed on the focal plane farthest from the eyepoint.

[0107] With the above technical solution, after the image regions are divided according to the depth information and the corresponding projection surfaces are determined for each image region for display, when viewing the image to be displayed, as the viewer's eye gaze position changes, the image position displayed on each projection surface will shift, resulting in a visual discontinuity in the image. To avoid this, in some possible implementations, the display control method provided by the present disclosure may further include:

[0108] determining the sight movement information of the eye box based on the change of the gaze point of the eye box;

[0109] The position of the image area displayed on each projection surface is moved according to the sight line movement information so that the position of the moved image area on each projection surface is consistent with the sight line after the eye box moves.

[0110] Regarding the above implementation, specifically, an exemplary eye movement monitoring system can be additionally provided in the display system 10 to monitor the viewer's (i.e., eye box) gaze point information in real time and promptly capture changes in the gaze point to determine line of sight movement information. After determining the line of sight movement information, the position of the image area displayed on each projection surface is adjusted based on the line of sight movement information. In some examples, each of the projection surfaces includes a canvas for displaying a corresponding image area; the method of moving the position of the corresponding image area displayed on each projection surface according to the line of sight movement information so that the position of the moved image area on each projection surface is consistent with the line of sight after the eye box is moved includes:

[0111] The position of the canvas on each projection surface is moved according to the sight line movement information, so that the position of the canvas after the movement on each projection surface is consistent with the sight line after the eye box moves.

[0112] Specifically, for the above example, as shown in Figure 17(A), a virtual 3D camera can be introduced, and the canvas used to display the image area on each projection surface can be considered the image captured by the virtual 3D camera. The position of the 3D camera can be considered the viewer's gaze point. Therefore, the image area displayed on the canvas on each projection surface as seen by the viewer can be considered the image captured by the virtual 3D camera, as shown in Figure 17(B). Based on this, when the viewer's gaze point shifts, the position of the 3D camera can be considered the viewer's gaze point. The position of the 3D camera can be moved in accordance with the shift in the gaze point position to maintain consistency with the gaze point position, thereby causing the position of the canvas on each projection surface to shift accordingly, and thus the position of the image area displayed on the canvas to shift accordingly. Based on this consistency, the position of the moved canvas is kept consistent with the line of sight after the eye box shifts, avoiding visual discontinuities in the image.

[0113] For example, taking any one of the multiple projection surfaces as an example, as shown in FIG18(A), before the viewer's gaze point moves, the image content displayed on the canvas in the projection surface that the viewer is focused on is indicated by the solid arrow. When the viewer's line of sight moves, after the canvas position is moved accordingly based on the above implementation method and its example, as shown in FIG18(B), the image content displayed on the canvas in the projection surface that the viewer is focused on remains as indicated by the solid line in FIG18(B). It can be seen that through the above implementation method and its example, after the canvas position is moved accordingly according to the viewer's line of sight, the image content that the viewer is focused on can be maintained without shifting, so that the image display position meets the viewer's eye position requirements in real time, avoiding visual discontinuities in the image.

[0114] Based on the same inventive concept as the aforementioned technical solution, FIG. 19 shows a display control device 1900 provided by the present disclosure. The device 1900 may be the display control unit 110 shown in FIG. 1 . In other words, the functional structure of the display control device 1900 shown in FIG. 19 can also be implemented by the display control unit 110. The display control device 1900 includes: a dividing part 1901 and a control part 1902; wherein,

[0115] The dividing portion 1901 is configured to divide the image to be displayed into image areas corresponding to the projection surface formed by the display unit based on depth information of the image to be displayed; wherein the projection surface formed by the display unit is arranged in sequence from a proximal end on a side close to the eye box position to a distal end on a side away from the eye box position;

[0116] The control part 1902 is configured to control each of the image areas to be displayed on the corresponding projection surface.

[0117] In some examples, the partitioning portion 1901 is configured to:

[0118] Estimate the depth information of each pixel in the image to be displayed; determine the depth interval corresponding to the projection surface; and divide all pixels in the image to be displayed into image areas corresponding to the projection surface according to the depth interval in which the depth information of each pixel is located.

[0119] In some examples, the partitioning portion 1901 is configured to:

[0120] Acquire a target area and a background area in the image to be displayed; wherein the number of the target areas is the number of the projection surfaces minus one;

[0121] Obtain an average depth information estimation value of each target area according to the depth information estimation value of each pixel in each target area;

[0122] Based on the average depth information estimation value of each target area, determining a corresponding projection surface from all projection surfaces in order from a proximal end on a side close to an eye box position to a distal end on a side far from the eye box position;

[0123] The projection surface farthest from the eye box position among all the projection surfaces is determined as the projection surface corresponding to the background area.

[0124] In some examples, the partitioning portion 1901 is configured to:

[0125] The pixel transmittance of each pixel in each target area is obtained according to the global atmospheric light value of the image to be displayed; the average transmittance of each target area is obtained according to the pixel transmittance of each pixel in each target area; and the average depth information estimation value of each target area is determined according to the average transmittance of each target area and the distance of the farthest projection surface from the eye box position side among all projection surfaces.

[0126] In some examples, the partitioning portion 1901 is configured to:

[0127] For the i-th target area O i , obtain the i-th target area O i The dark channel value of each pixel in the background area; obtaining the global atmospheric light value of the image to be displayed according to the maximum value of the dark channel values ​​of all pixels in the background area; according to the i-th target area O i The dark channel value of each pixel in the image and the global atmospheric light value of the image to be displayed are used to obtain the i-th target area O i The pixel transmittance of each pixel in .

[0128] In some examples, the partitioning portion 1901 is configured to:

[0129] For the i-th target area O i , according to the distance F of the j-th projection surface j And the i-th target area O i The average depth information estimate D i , determine the i-th target area O according to the following formula i The corresponding projection surface identifier k:

[0130]

[0131] Where n represents the number of all projection surfaces.

[0132] In some examples, the control part 1902 is further configured to: determine the line of sight movement information of the eye box based on the change of the gaze point of the eye box; and move the position of the image area corresponding to the display on each projection surface according to the line of sight movement information so that the position of the moved image area on each projection surface is consistent with the line of sight after the eye box moves.

[0133] In some examples, each of the projection surfaces includes a canvas for displaying a corresponding image area; the control portion 1902 is further configured to move the position of the canvas on each projection surface according to the line of sight movement information so that the position of the canvas after movement on each projection surface is consistent with the line of sight after the eye box moves.

[0134] Please refer to Figure 20, which shows a block diagram of a display control device 1900 according to an exemplary embodiment of the present disclosure. In some examples, the display control device 1900 has communication capabilities and can access a wired or wireless network. In some examples, the display control device 1900 can receive data based on the accessed wired or wireless network. It is understood that the display control device 1900 is responsible for the calculation and processing of the technical solution of the present disclosure, and this disclosure is not limited to this.

[0135] As shown in FIG. 20 , the display control device 1900 in the present disclosure may include one or more of the following components: a processor 2010 and a memory 2020 .

[0136] Optionally, the processor 2010 utilizes various interfaces and lines to connect various parts of the entire computing device, and performs various functions of the computing device and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 2020, and calling data stored in the memory 2020. Optionally, the processor 2010 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 2010 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to handle wireless communication. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 2010, but may be implemented by a separate chip.

[0137] The memory 2020 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 2020 includes a non-transitory computer-readable storage medium. The memory 2020 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 2020 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above various method embodiments, etc.; the data storage area may store data created according to the use of the computing device, etc.

[0138] In addition, those skilled in the art will understand that the structure of the computing device shown in the above figures does not constitute a limitation of the computing device. The computing device may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components. For example, the computing device also includes a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, an input unit, sensors (such as an accelerometer, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, and other components, which will not be described in detail here.

[0139] It is understandable that the exemplary technical solution of the display control device 1900 described above is based on the same concept as the technical solution of the aforementioned display control method. Therefore, any details not described in detail in the technical solution of the display control device 1900 described above can be referred to the description of the technical solution of the aforementioned display control method. This embodiment of the present disclosure does not elaborate on this.

[0140] The present disclosure also provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction is used to be executed by a processor to implement the display control method described in the above embodiments.

[0141] The present disclosure also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium; a processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes to implement the display control method described in each of the above embodiments.

[0142] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in this disclosure can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0143] It should be noted that the technical solutions described in this disclosure can be combined arbitrarily without conflict.

[0144] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will appreciate that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure. Industrial Applicability

[0145] In this embodiment, based on the number of projection surfaces formed by the display unit and distributed from near to far away from the viewer, the image data to be displayed is divided into image areas corresponding to the projection surfaces according to depth information, and then displayed using the projection surfaces, which increases the vividness and immersion of the visual effects when the viewer views the image data, thereby improving the user experience.

Claims

1. A display control method, characterized in that, The method includes: Based on the depth information of the image to be displayed, dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit; wherein, the projection planes formed by the display unit are arranged in sequence from the proximal end on the side close to the eyebox position to the distal end on the side away from the eyebox position; Controlling each of the image regions to be displayed on the corresponding projection plane.

2. The method according to claim 1, wherein The step of dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed includes: Estimating the depth information of each pixel in the image to be displayed; Determining the depth interval corresponding to the projection plane; Dividing all the pixels in the image to be displayed into image regions corresponding to the projection plane according to the depth interval where the depth information of each pixel is located.

3. The method according to claim 1, characterized in that, The step of dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed includes: Obtaining the target regions and the background region in the image to be displayed; wherein, the number of the target regions is one less than the number of the projection planes; Obtaining the average depth information estimation value of each target region according to the depth information estimation value of each pixel in each target region; Based on the average depth information estimation value of each target region, determining the corresponding projection plane from all the projection planes in the order from the proximal end on the side close to the eyebox position to the distal end on the side away from the eyebox position; Determining the projection plane at the farthest end on one side of the eyebox position among all the projection planes as the projection plane corresponding to the background region.

4. The method according to claim 3, wherein The step of obtaining the average depth information estimation value of each target region according to the depth information estimation value of each pixel in each target region includes: Obtaining the pixel transmittance of each pixel in each target region according to the global atmospheric light value of the image to be displayed; Obtaining the average transmittance of each target region according to the pixel transmittance of each pixel in each target region; Determining the average depth information estimation value of each target region according to the average transmittance of each target region and the distance of the projection plane at the farthest end on one side of the eyebox position among all the projection planes.

5. The method according to claim 4, characterized in that, The step of obtaining the pixel transmittance of each pixel in each target region according to the global atmospheric light value of the image to be displayed includes: For the i-th target region O i , obtain the dark channel value of each pixel in the i-th target region O i ; Obtaining the global atmospheric light value of the image to be displayed according to the maximum value among the dark channel values of all the pixels in the background region; According to the dark channel value of each pixel in the i-th target region O i and the global atmospheric light value of the image to be displayed, obtain the pixel transmittance of each pixel in the i-th target region O i ​ 6. The method according to claim 4, characterized in that, The step of determining the corresponding projection plane from all the projection planes in the order from the proximal end on the side close to the eyebox position to the distal end on the side away from the eyebox position based on the average depth information estimation value of each target region includes: For the i-th target region O i , according to the distance F of the j-th projection plane j and the estimated value D of the average depth information of the i-th target region O i , determine the projection plane identifier k corresponding to the i-th target region O i according to the following formula: i ​ , where n represents the number of all the projection planes.

7. The method according to claim 1, characterized in that, The method further includes: Determining the line-of-sight movement information of the eyebox based on the change of the fixation point of the eyebox; Moving the position of the image region corresponding to each projection plane according to the line-of-sight movement information so that the position of the moved image region on each projection plane is consistent with the moved line of sight of the eyebox.

8. The method according to claim 7, wherein Each of the projection surfaces includes a canvas for displaying a corresponding image area; moving the positions of the image areas correspondingly displayed on each projection surface according to the line-of-sight movement information so that the positions of the moved image areas on each projection surface are consistent with the line of sight after the movement of the eye box includes: Moving the positions of the canvases on each projection surface according to the line-of-sight movement information so that the positions of the moved canvases on each projection surface are consistent with the line of sight after the movement of the eye box.

9. A display control device, characterized in that, The display control device includes: a partitioning part and a control part; wherein, The partitioning part is configured to partition the image to be displayed into image areas corresponding to the projection surfaces formed by the display part based on the depth information of the image to be displayed; wherein, the projection surfaces formed by the display part are arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side far from the eye box position; The control part is configured to control each of the image areas to be displayed on the corresponding projection surface.

10. A display control device, characterized in that, The device includes: a processor and a memory; the processor is used to execute the instructions stored in the memory to implement the display control method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the display control method according to any one of claims 1 to 8.

12. A display system, characterized in that, The display system includes a display control part and a display part; wherein, The display control part is configured to partition the image to be displayed into image areas corresponding to the projection surfaces formed by the display part based on the depth information of the image to be displayed; and control each of the image areas to be displayed on the corresponding projection surface; The display part is configured to form projection surfaces arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side far from the eye box position; and display each of the image areas on the corresponding projection surface based on the control of the display control part.

13. A vehicle, characterized in that, The vehicle includes the display system according to claim 12.

Images