Display system, display control method and device thereof, medium and vehicle

Abstract

The invention provides a display system, a display control method and device thereof, a medium and a vehicle. The display control method comprises the following steps: dividing an image to be displayed into an image area corresponding to a projection surface formed by a display part based on depth information of the image to be displayed; projection planes formed by the display part are sequentially arranged from the near end of one side close to the eye box position to the far end of one side far away from the eye box position; and controlling each image area to be displayed on the corresponding projection surface.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image display, and in particular, to a display system, a display control method, a device, a medium, and a vehicle thereof. Background Art

[0002] A head-up display (HUD) device projects the light of a display image output by an image source onto an imaging window (such as an imaging plate, a windshield, etc.) through, for example, a reflective optical design, so as to display vehicle status information such as vehicle speed and fuel level, as well as indication information such as navigation and danger warnings at an appropriate position in front of the driver. Thereby, the driver can obtain relevant information such as vehicle speed and fuel level without deviating the line of sight from the road surface ahead, and thus the safety factor and driving experience of driving can be improved. Summary of the Invention

[0003] The present disclosure provides a display system, a display control method, a device, a medium, and a vehicle thereof; which can increase the vividness and immersion of the driver's visual effect and improve the user experience.

[0004] The technical solution of the present disclosure is realized as follows:

[0005] In a first aspect, the present disclosure provides a display control method, the method including:

[0006] Based on the depth information of the image to be displayed, dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit; wherein, the projection planes formed by the display unit are arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side away from the eye box position;

[0007] Controlling each of the image regions to be displayed on the corresponding projection plane.

[0008] In a second aspect, the present disclosure provides a display control device, the display control device including: a dividing part and a controlling part; wherein,

[0009] The dividing part is configured to divide the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed; wherein, the projection planes formed by the display unit are arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side away from the eye box position;

[0010] The controlling part is configured to control each of the image regions to be displayed on the corresponding projection plane.

[0011] In a third aspect, the present disclosure provides a display control device, the device comprising: a processor and a memory; the processor is configured to execute instructions stored in the memory to implement the display control method as described in the first aspect.

[0012] In a fourth aspect, the present disclosure provides a computer-readable storage medium, the computer-readable storage medium storing at least one instruction, the at least one instruction being configured to be executed by a processor to implement the display control method as described in the first aspect.

[0013] In a fifth aspect, the present disclosure provides a display system, the display system comprising a display control unit and a display unit; wherein,

[0014] the display control unit is configured to divide the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed;

[0015] and control each of the image regions to be displayed on the corresponding projection plane;

[0016] the display unit is configured to form projection planes arranged in sequence from a proximal end on a side close to the eye box position to a distal end on a side away from the eye box position; and display each of the image regions on the corresponding projection plane based on the control of the display control unit.

[0017] In a sixth aspect, the present disclosure provides a vehicle, the vehicle comprising the display system as described in the fifth aspect.

[0018] The present disclosure provides a display system, a display control method, a device, a medium and a vehicle thereof; based on the number of projection planes formed by the display unit and distributed from near to far from the viewer, the image data to be displayed is divided into image regions corresponding to the projection planes according to the depth information, and then displayed using the projection planes, which increases the vividness and immersion of the visual effect when the viewer views the image data and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of the architecture of a display system provided by the present disclosure.

[0020] Figure 2 is an exemplary top view of a vehicle provided by the present disclosure.

[0021] Figure 3 is an exemplary perspective view from the driver's seat of a vehicle provided by the present disclosure.

[0022] Figure 4 is a schematic flowchart of a display control method provided by the present disclosure.

[0023] FIG. 5(A) is a schematic diagram of an image for identifying a building in a street view provided by the present disclosure.

[0024] FIG. 5(B) is a schematic diagram of a two-dimensional identification image provided by the present disclosure.

[0025] FIG. 6(A) is a side view schematic diagram of an image area corresponding to a projection plane provided by the present disclosure.

[0026] FIG. 6(B) is a schematic diagram of an image area provided by the present disclosure.

[0027] Figure 7 is an oblique view schematic diagram of an image area corresponding to a projection plane provided by the present disclosure.

[0028] Figure 8 is a schematic diagram of a process for dividing an image area corresponding to a projection plane provided by the present disclosure.

[0029] Figure 9 (A) is a schematic diagram of an image to be displayed provided by the present disclosure.

[0030] Figure 9 (B) is a schematic diagram of a target object provided by the present disclosure.

[0031] Figure 10 is a schematic diagram of a process for obtaining an estimated value of the average depth information of each target area provided by the present disclosure.

[0032] Figure 11 is a schematic diagram of another image area provided by the present disclosure.

[0033] FIG. 12(A) is a side view schematic diagram of another image area corresponding to a projection plane provided by the present disclosure.

[0034] FIG. 12(B) is an oblique view schematic diagram of another image area corresponding to a projection plane provided by the present disclosure.

[0035] Figure 13 is a schematic diagram of another process for dividing an image area corresponding to a projection plane provided by the present disclosure.

[0036] Figure 14 (A) is a schematic diagram of yet another image to be displayed provided by the present disclosure.

[0037] Figure 14 (B) is a schematic diagram of yet another division of an image area for an image to be displayed provided by the present disclosure.

[0038] Figure 15 is a schematic diagram of yet another image area provided by the present disclosure.

[0039] FIG. 16(A) is a side view schematic diagram of another image area corresponding to the display on the projection plane provided by the present disclosure.

[0040] FIG. 16(B) is an oblique view schematic diagram of another image area corresponding to the display on the projection plane provided by the present disclosure.

[0041] FIG. 17(A) is a schematic diagram of a virtual three-dimensional camera capturing a picture provided by the present disclosure.

[0042] FIG. 17(B) is a schematic diagram of the image area seen by the viewer provided by the present disclosure.

[0043] FIG. 18(A) is a schematic diagram of the picture content concerned before the fixation point moves provided by the present disclosure.

[0044] FIG. 18(B) is a schematic diagram of the picture content concerned after the fixation point moves provided by the present disclosure.

[0045] Figure 19 It is a schematic diagram of the composition of a display control device provided by the present disclosure.

[0046] Figure 20 It is a schematic diagram of the structure of a display control device provided by the present disclosure. Detailed implementation manners

[0047] Next, the technical solutions in the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the present disclosure.

[0048] Referring to Figure 1 , which shows an exemplary implementation architecture of a display system 10 provided by the present disclosure. The display system 10 includes: a display control unit 110, a display unit 120, and a display component 130. Among them, the display control unit 110 can process the received data and the display information that needs to be displayed, and project the display information onto the display component 130 through the display unit 120 for display. Specifically, after receiving the data 320, the display control unit 110 generates a signal 312. The display unit 120 may include a light source 121 and an optical path component 122. The light source 121 outputs light (such as a virtual image) based on the signal 312 from the display control unit 150 for display on the display component 130. For example, the light source 121 may include one or more lasers and output red light, green light, and blue light.

[0049] The optical path component 122 can reflect the output of the light source 121 onto the display component 130, and viewers can view the display information 212 in the display area where the display information 212 is projected onto the display component 130. In some examples, the optical path component 122 can include one or more mirrors (plane mirrors) and concave mirrors (magnifying glasses). In some examples, the optical path component 122 can reflect the output of the light source 121 onto the display component 130 in the form of two or more (such as N) optical paths (such as the optical paths L-1, …, L-N in Figure 1 ) and form a virtual image 30 in front of the viewer. The virtual image 30 can be projected and displayed on one of the N projection planes 31-1, …, 31-N corresponding to the N optical paths (optical paths L-1, …, L-N) based on the control of the display control unit 110. In some examples, the optical path component 122 can form two or more (such as N) optical paths by polarization processing of the output of the light source 121. Depending on the application scenario, the image display component 13 will also be different. For example, when the application scenario is movie projection in a cinema, then the image display component 13 is a projection screen or a display screen. When the application scenario is to display information on the windshield of a vehicle, then the image display component 13 is the windshield of the vehicle.

[0050] In some examples, these N projection planes (which can also be referred to as focal planes) are distributed in the interval of 5 m to 17 m between the viewer's viewing point (which can also be referred to as the eye point or the eye box position) in the order of increasing distance from the viewer. In some examples, the display control unit 110 can control the virtual image 30 formed by the display unit 120 to be displayed on a projection plane that is farther or closer to the viewer's viewing point, so that the viewer can produce a visual effect of AR display when observing.

[0051] In the present disclosure, taking the display system 10 shown in Figure 1 as an example of being applied to display information on the windshield of a vehicle, refer to Figure 2 and Figure 3 , which respectively show an exemplary top view of the vehicle and an exemplary perspective view from the driver's seat of the vehicle. In the present disclosure, the vehicle is equipped with a display system 10 that can be applied to the technical solution of the present disclosure (refer to Figure 1 ). In some examples, the vehicle can be an internal combustion engine vehicle powered by an engine, a hybrid vehicle powered by an engine and an electric motor, an electric vehicle powered by an electric motor, and other types of vehicles. In the subsequent content of this specification, the vehicle equipped with the display system 10 is referred to as the present vehicle. In Figure 2 , the present vehicle includes a windshield 204 located at the front of the vehicle. The driver and passengers in the passenger compartment 208 of the present vehicle can see the front of the present vehicle through the windshield 204.

[0052] In Figure 2 , the windshield 204 is visually located above the vehicle instrument panel 206. The driver can turn the steering wheel 210 in the passenger compartment 208 to steer the vehicle, such as changing lanes, merging, and parking the vehicle. In some embodiments, the steering wheel 210 can be retracted or omitted.

[0053] In Figure 2 , the display system 10 projects display information 212 (e.g., a virtual image) onto a portion of the windshield 204 through one or more holes (e.g., hole 216) in the instrument panel 206, i.e., the windshield 204 serves as the image display component 13 of the display system 10. Although Figure 3 an example size of the display information 212 is shown, the display information 212 can be presented in a larger or smaller area. Example of the display information 212 includes various vehicle information, as well as streaming media information such as images and videos provided by an in-vehicle entertainment system (not shown in the figure). The display system 10 provides this information to the vehicle driver as an example viewer, and the driver does not need to shift their line of sight from the object in front of the vehicle. In some examples, in Figure 1 the scenario where the display system 10 shown is applied to display information on the windshield of a vehicle, the display system 10 can also be referred to as a Head Up Display (HUD) system.

[0054] In the above related display control scheme, although the virtual image 30 is displayed on projection planes at different distances from the driver's viewing point to produce the visual effect of AR display, the virtual image 30 is still only displayed on a single projection plane, and the visual effect is still not vivid enough and lacks a sense of immersion. Based on this, the present disclosure expects to divide the image data to be displayed into image regions corresponding to the projection planes according to the depth information based on the number of projection planes formed by the display unit 120 and distributed from near to far from the viewer, and then use the projection planes for display, which increases the vividness and immersion of the driver's visual effect and improves the user experience.

[0055] See Figure 4 , which shows a display control method provided by the present disclosure. This method can be applied to Figure 1 the display system 10 shown, especially applied to the display control unit 110 in the display system 10. The method includes step S401 to step S402.

[0056] In step S401, based on the depth information of the image to be displayed, the image to be displayed is divided into image regions corresponding to the projection planes formed by the display unit.

[0057] In some examples, in combination with Figure 1As shown, the projection planes formed by the display unit 120 are arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side away from the eye box position.

[0058] In some examples, the image to be displayed can be image data provided by other in-vehicle systems mounted on the vehicle, such as a navigation system, an infotainment system, etc., or each image frame of the video data provided by these in-vehicle systems. Specifically, these in-vehicle systems can transmit the image to be displayed to the display control unit 110 in the display system 10 through the system bus in the vehicle.

[0059] In some examples, the image to be displayed can also be transmitted to the display control unit 110 in the display system 10 by other devices not mounted on the vehicle. For example, a mobile terminal carried by the driver or passenger of the vehicle can be connected to the display system 10 of the vehicle in a wired or wireless manner. Based on these connections, the mobile terminal can transmit the image to be displayed to the display control unit 110 in the display system 10.

[0060] In the above examples, the image to be displayed can be a two-dimensional image or a depth image with depth information. The depth information is represented as the distance information between the elements in the image and the image acquirer with the eye box as the image acquirer. In some examples, based on the analysis granularity of the image, the elements in the image can be either the pixels in the image or the target objects presented in the image, etc. The present disclosure does not make specific limitations in this regard.

[0061] For a two-dimensional image, although the elements in the two-dimensional image are all in the same plane, based on the actual distance information existing in the image acquisition process, the elements in the two-dimensional image will reflect their corresponding distance information on some specific image features. These specific image features can be called cues in some examples, such as linear perspective, focus / defocus, atmospheric scattering, shadow, texture, occlusion, relative height, and motion cues. Based on these specific image features, the depth information of the elements in the two-dimensional image can be estimated.

[0062] In addition, in some exemplary two-dimensional image generation processes, the distance information actually existing in the image acquisition process has a corresponding relationship with the elements in the image. As shown in FIG. 5(A), taking the HUD system of a vehicle as an example, in the image for identifying buildings in the street view, the distance information between each building and the image collector can be obtained by positioning the image collector through the navigation system and then using the map information. In the two-dimensional identification image shown in FIG. 5(B), each identification corresponds to a distance information with the image collector based on the corresponding relationship with the building. For example, in FIG. 5(A), the building indicated by Identification 1 is the closest to the image collector, followed by the building indicated by Identification 2, then the buildings indicated by Identification 3 and Identification 4 in sequence, and the building indicated by Identification 5 is the farthest from the image collector. Based on this, for the image including building identifications shown in FIG. 5(B), there is no need to estimate the depth information based on image features, but directly obtain the depth information of the elements in the two-dimensional image according to the above corresponding relationship.

[0063] For a depth image, each pixel of the image includes, in addition to the RGB channel values, a distance channel value related to the distance from the image collector, and this distance channel value can be regarded as the depth information of the depth image.

[0064] According to the depth information of the to-be-displayed image, the to-be-displayed image can be divided into the same number of image regions according to the number of projection planes formed by the display unit 120 as shown. Since these image regions are divided according to the depth information, considering the eye box position as the position of the image collector, then these image regions are also arranged in sequence from the proximal side close to the eye box position to the distal side far from the eye box position, which is consistent with the arrangement order of the projection planes. Based on this consistency, a corresponding projection plane can be determined for each image region in all projection planes. Figure 1

[0065] For example, it is set that Figure 1 ​The number of projection planes formed by the display system shown is 5. As shown in Fig. 6(A), these projection planes are arranged in sequence from the proximal end on the side close to the eye box position 60 to the distal end on the side far from the eye box position 60, and are sequentially labeled as 31-1, 31-2, 31-3, 31-4, and 31-5. Taking Fig. 5(A) and Fig. 5(B) as examples, according to the distance between the building indicated by each identifier and the image collector, it is divided into 5 image regions as shown in Fig. 6(B). In Fig. 6(B), these image regions can be sequentially labeled as 51, 52, 53, 54, and 55 according to the distance from the image collector from near to far. The corresponding relationship between the image regions and the projection planes is as follows: image region 51 corresponds to projection plane 31-1, image region 52 corresponds to projection plane 31-2, image region 53 corresponds to projection plane 31-3, image region 54 corresponds to projection plane 31-4, and image region 55 corresponds to projection plane 31-5.

[0066] It should be noted that when the image to be displayed is a depth image, since the depth image carries the distance channel values that can be used as depth information, therefore, the depth image can be divided into multiple image regions with the distance from the image collector from near to far according to the number of projection planes based on the distance channel values, and the corresponding projection planes are determined for each image region respectively according to the order of the distance from the image collector from near to far. The specific implementation process is the same as that shown in Fig. 5(A), Fig. 5(B), Fig. 6(A), and Fig. 6(B), and the present disclosure will not be elaborated here.

[0067] In step S402, control each image region to be displayed on the corresponding projection plane.

[0068] In the present disclosure, still taking the examples shown in Fig. 5(A), Fig. 5(B), Fig. 6(A), and Fig. 6(B) as references, see Figure 7 , after determining the corresponding projection plane for each image region, project each image region onto the corresponding projection plane by controlling the multiple optical paths provided by the optical path component 122 in the display unit 120. When the viewer views the image to be displayed, compared with the display scheme of projecting a two-dimensional image on a single projection plane, it can provide a more vivid and immersive visual experience for the viewer.

[0069] It should be noted that when the image to be displayed is a depth image, since the depth image carries the distance channel values that can serve as depth information, the depth image can be divided into multiple image regions with increasing distances from the image collector based on the distance channel values according to the number of projection planes. Then, the corresponding projection planes are determined for each image region in the order of increasing distances from the image collector to the image regions. The specific implementation process is the same as that shown in FIGS. 5(A), 5(B), 6(A) and 6(B), and will not be elaborated in this disclosure. After dividing the depth image into image regions corresponding to the projection planes, the multiple optical paths provided by the optical path component 122 in the display unit 120 can still be controlled to project each image region onto the corresponding projection plane.

[0070] Figure 4 For the technical solution shown, based on the number of projection planes formed by the display unit 120 and distributed from near to far from the viewer, the image data to be displayed is divided into image regions corresponding to the projection planes according to the depth information, and then the projection planes are used for display, which increases the vividness and immersion of the visual effect when the viewer views the image data and improves the user experience.

[0071] For Figure 4 the technical solution shown, in some possible implementation manners, when the image to be displayed is a two-dimensional image, since the elements in the two-dimensional image reflect their distance information from the image collector during the image acquisition process through specific image features, this disclosure estimates the depth information of the elements in the two-dimensional image using these image features as clues. The estimated value can more accurately represent the relative depth information between each element and the image collector. After obtaining the relative depth information, the image to be displayed can be divided into image regions with the same number as the number of projection planes according to the relative distance from the image collector, and the projection plane corresponding to each image region is determined based on the relative distance from the image collector.

[0072] For the above implementation manner, with a relatively large image analysis granularity, the elements in the image to be displayed can be the target objects presented in the image. Based on this, in some examples, refer to Figure 8 , dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed includes steps S801 to S804.

[0073] In step S801, the target region and the background region in the image to be displayed are acquired.

[0074] In this disclosure, taking Figure 9Taking the image to be displayed shown in (A) as an example, object recognition can be performed on the image to be displayed to obtain the target objects existing in the image to be displayed. Each target object corresponds to a target area. The target objects that cannot be recognized in the image to be displayed or the image content that does not belong to any target object in the image to be displayed are considered as background areas. In the present disclosure, each target area can be considered as an image area, and the background area is considered as the image area that is farthest from the image collector during the image acquisition process. That is to say, the projection plane corresponding to the background area is the projection plane that is farthest from the eye box position. Combining Figure 9 the example of the image to be displayed shown in (A), the target objects are as Figure 9 shown by the targets enclosed by the solid line in (B). Since the projection planes formed by the display unit 120 correspond to the target areas and the background area, therefore, in the present disclosure, the number of target areas is one less than the number of projection planes. Assuming that the number of projection planes in this example is 4, then the number of target areas is 3.

[0075] In step S802, the average depth information estimation value of each target area is obtained according to the depth information estimation value of each pixel in each target area.

[0076] In this example, the depth information of each target area is estimated using the transmittance as a clue. Specifically, as Figure 10 shown, obtaining the average depth information estimation value of each target area according to the depth information estimation value of each pixel in each target area described in step S802 includes:

[0077] S8021: Obtain the pixel transmittance of each pixel in each target area according to the global atmospheric light value of the image to be displayed;

[0078] For step S8021, in the specific implementation process, for the i-th target area O i in all target areas, first, the dark channel value of each pixel in the i-th target area O i can be obtained. For example, each pixel point includes RGB channel values. The channel corresponding to the minimum value among these three channel values can be used as the dark channel of the pixel point. Correspondingly, the minimum value among these three channel values is the dark channel value of the pixel point.

[0079] Next, the global atmospheric light value of the image to be displayed is obtained according to the maximum value among the dark channel values of all pixels in the background area. For example, in the background area, the dark channel values of all pixel points in the background area can be statistically analyzed, and the maximum value among these dark channel values is used as the global atmospheric light value of the image to be displayed.

[0080] Finally, according to the i-th target area O iObtain the average depth information estimation value of the \(i\)-th target area \(O\) based on the dark channel value of each pixel in the image and the global atmospheric light value of the image to be displayed i The pixel transmittance of each pixel in it. For example, after obtaining the global atmospheric light value \(A\), for the \(i\)-th target area \(O\) i For the \(p\)-th pixel in it, the pixel transmittance of this pixel can be obtained according to the following formula

[0081]

[0082] where \(j\) dark (p) represents the dark channel value of the \(p\)-th pixel

[0083] S8022: Obtain the average transmittance of each target area according to the pixel transmittance of each pixel in each target area

[0084] S8023: Determine the average depth information estimation value of each target area according to the average transmittance of each target area and the distance of the most distal projection plane on one side of the eye box position among all projection planes

[0085] For step S8023, specifically, the transmittance of the elements in the image has a mapping relationship with the distance between them and the image collector. Since the background area is considered to be the image area farthest from the image collector, and the background area corresponds to the most distal projection plane on one side of the eye box position. Based on this, for the \(i\)-th target area \(O\) i it can be determined according to this mapping relationship and the distance \(F_n\) of the most distal projection plane on one side of the eye box position for the \(i\)-th target area \(O\) i The average depth estimation value \(D\) of i , that is

[0086]

[0087] where represents the average transmittance of the \(i\)-th target area \(O\) i

[0088] It should be noted that although the average depth information estimation value of each target area obtained through the above Figure 10 shown process cannot accurately obtain the distance between each target area and the image collector, it can accurately represent the relative distance between each target area relative to the image collector. Through the representation of this relative distance, it can provide a basis for determining the corresponding projection plane for each target area

[0089] ​In step S803, based on the estimated average depth information of each target region, the corresponding projection plane is determined from all the projection planes in the order from the proximal end on the side close to the eye box position to the distal end on the side away from the eye box position.

[0090] In this example, in order to determine the projection plane corresponding to each target region, the estimated average depth information value of each target region can be compared with the distance of the projection plane. Taking the i-th target region O i as an example, when the distance of the projection plane closest to the estimated average depth information value of this target region, this projection plane can be determined as the projection plane corresponding to the i-th target region O i Specifically, the step S803 of determining the corresponding projection plane from all the projection planes based on the estimated average depth information of each target region in the order from the proximal end on the side close to the eye box position to the distal end on the side away from the eye box position includes:

[0091] For the i-th target region O i , according to the distance F j of the j-th projection plane and the estimated average depth information value D i of the i-th target region O i , the projection plane identifier k corresponding to the i-th target region O i is determined according to the following formula:

[0092]

[0093] where n represents the number of all projection planes.

[0094] In step S804, the projection plane at the farthest end on one side of the eye box position among all the projection planes is determined as the projection plane corresponding to the background region.

[0095] For the example shown above Figure 8 , combined with Figure 9 (A) and Figure 9 (B) of the image to be displayed and the target object, the obtained image region is as Figure 11As shown, among them, image regions 11-1, 11-2, and 11-3 are all target regions, and the estimated average depth information values of image regions 11-1, 11-2, and 11-3 indicate that the respective target regions are gradually moving away from the image collector, and image region 11-4 is the background region. Combining with the number of projection planes set in this example being 4, referring to FIGS. 12(A) and 12(B), from the position 60 near the eye box in order of increasing distance, they are the projection plane 31-1 corresponding to image region 11-1, the projection plane 31-2 corresponding to image region 11-2, the projection plane 31-3 corresponding to image region 11-3, and the projection plane 31-4 corresponding to image region 11-4.

[0096] Through the above Figure 8 shown example, the image regions are divided with a relatively large image analysis granularity, and a corresponding projection plane is determined for each image region. For a relatively small image analysis granularity, for example, the elements in the image to be displayed can be pixels in the image. Similarly, the image regions can be divided based on the depth information of each pixel, and a corresponding projection plane is determined for each image region. Based on this, in some examples, referring to Figure 13 , the method of dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed includes steps S1301 to S1303.

[0097] In step S1301, the depth information of each pixel in the image to be displayed is estimated.

[0098] For step S1301, specifically, in addition to using the transmittance as a clue for estimating the depth information in the aforementioned Figure 8 shown example, the present disclosure can also estimate the depth information based on other image features presented by each pixel, such as linear perspective, focus / blur, atmospheric scattering, shadows, textures, occlusions, relative height, and motion cues. In addition, the depth information of each pixel can also be estimated through deep learning, so as to estimate the depth information of each pixel according to the image to be displayed.

[0099] In step S1302, a depth interval corresponding to the projection plane is determined.

[0100] For step S1302, due to the corresponding relationship between the image region and the projection plane, in order to divide the image region according to the depth information, a depth interval corresponding to the projection plane can be set. For example, if the number of projection planes arranged in sequence from the proximal side near the eye box position to the distal side far from the eye box position is set to 8, then 8 depth intervals representing a gradual distance from the image collector can be correspondingly constructed. Each depth interval corresponds to a projection plane.

[0101] In step S1303, all pixels in the to-be-displayed image are divided into image regions corresponding to the projection planes according to the depth intervals in which the depth information of each pixel is located.

[0102] In the specific implementation process, each depth interval correspondingly includes the upper limit of the depth information of the interval and the lower limit of the depth information of the interval. By comparing the depth information of each pixel with the upper limit of the depth information and the lower limit of the depth information of each depth interval respectively, it is possible to determine the depth interval in which each pixel is located. For example, taking pixel M as an example, when the depth information of this pixel is less than the upper limit of the depth information of depth interval A and greater than the lower limit of the depth information of depth interval A, it can be confirmed that pixel M is in depth interval A.

[0103] For step S1303, after obtaining 8 depth intervals, each pixel in the to-be-displayed image can be traversed, and the depth interval in which the depth information of each pixel is located can be determined. After completing the traversal of all pixels, all pixels in the same depth interval can be divided into the same image region. Then each image region corresponds to a depth interval, and combined with the corresponding relationship between the depth interval and the projection plane, each image region corresponds to a projection plane.

[0104] For Figure 13 the example shown, taking Figure 14 the to-be-displayed image shown in (A) as an example, it is set that the number of projection planes is 8, and the number of depth intervals is also 8. After estimating and obtaining Figure 14 the depth information of all pixels in, 8 image regions as shown in Figure 14 (B) can be divided according to the depth intervals in which the depth information of all pixels is located. In the order of gradually moving away from the image collector, as Figure 15 shown, these 8 image regions are respectively labeled as image region 15-1, image region 15-2, image region 15-3, image region 15-4, image region 15-5, image region 15-6, image region 15-7, and image region 15-8. Then combined with the setting in this example that the number of projection planes is 8, referring to Fig. 16(A), from the position of the eye box 60 from near to far, in turn, image region 15-1 corresponds to projection plane 31-1, image region 15-2 corresponds to projection plane 35-2, image region 15-3 corresponds to projection plane 31-3, image region 15-4 corresponds to projection plane 31-4, image region 15-5 corresponds to projection plane 31-5, image region 15-6 corresponds to projection plane 35-6, image region 15-7 corresponds to projection plane 31-7, and image region 15-8 corresponds to projection plane 31-8. As shown in Fig. 16(B), starting from the eye box position (i.e., the eye point), the image region closest to the image collector is displayed on the focal plane closest to the eye point, and the image region farthest from the image collector is displayed on the focal plane farthest from the eye point.

[0105] Through the above technical solution, after dividing the image area according to the depth information and determining the corresponding projection plane for each image area for display, during the process of viewing the image to be displayed, as the viewing position of the viewer's eyes changes, the positions of the images displayed on each projection plane will all shift, resulting in a visual break in the picture. To avoid this situation, in some possible implementation manners, the display control method provided by the present disclosure may further include:

[0106] Determining the line-of-sight movement information of the eye box based on the change of the fixation point of the eye box;

[0107] Moving the positions of the image areas corresponding to be displayed on each projection plane according to the line-of-sight movement information so that the positions of the moved image areas on each projection plane are consistent with the line of sight of the moved eye box.

[0108] For the above implementation manner, specifically, an exemplary eye movement monitoring system may be additionally set in the display system 10 to monitor the fixation point information of the viewer (i.e., the eye box) in real time and capture the change of the fixation point in a timely manner to determine the line-of-sight movement information. After determining the line-of-sight movement information, the positions of the image areas displayed on each projection plane are adjusted based on the line-of-sight movement information. In some examples, each of the projection planes includes a canvas for displaying the corresponding image area; the moving the positions of the image areas corresponding to be displayed on each projection plane according to the line-of-sight movement information so that the positions of the moved image areas on each projection plane are consistent with the line of sight of the moved eye box includes:

[0109] Moving the positions of the canvases on each projection plane according to the line-of-sight movement information so that the positions of the moved canvases on each projection plane are consistent with the line of sight of the moved eye box.

[0110] For the above example, specifically, as shown in Fig. 17(A), a virtual three-dimensional camera can be introduced, and the canvas for displaying the image area on each projection plane can be regarded as the picture captured by the virtual three-dimensional camera. The position where the three-dimensional camera is located can be regarded as the position of the viewer's fixation point. Then, the image area displayed on the canvas of each projection plane that the viewer sees can be considered as the image captured by the virtual three-dimensional camera, as shown in Fig. 17(B). Based on this, when the position of the viewer's fixation point shifts, since the position where the three-dimensional camera is located can be regarded as the position of the viewer's fixation point, the position of the three-dimensional camera can be moved accordingly to the shift of the fixation point position to keep it consistent with the fixation point position, so that the position of the canvas on each projection plane also moves accordingly. Furthermore, the position of the image area displayed on the canvas also moves accordingly, and based on the above consistency, the position of the moved canvas is kept consistent with the line of sight after the eye box moves, avoiding the occurrence of visual discontinuity of the picture.

[0111] For example, taking any one of the multi-projection planes as an example, as shown in Fig. 18(A), before the viewer's fixation point moves, the picture content displayed on the canvas in the projection plane that the viewer pays attention to is as shown by the solid arrow. When the viewer's line of sight moves, after the position of the canvas is moved accordingly based on the above implementation manner and its example, as shown in Fig. 18(B), the picture content displayed on the canvas in the projection plane that the viewer pays attention to is still as shown by the solid line in Fig. 18(B). It can be seen that through the above implementation manner and its example, after the position of the canvas is moved accordingly according to the viewer's line of sight movement, it can be ensured that the picture content that the viewer pays attention to does not shift, enabling the display position of the image to meet the viewer's eye position requirements in real time and avoiding the occurrence of visual discontinuity of the picture.

[0112] Based on the same inventive concept as the foregoing technical solution, refer to Figure 19 , which shows a display control device 1900 provided by the present disclosure. The device 1900 may be Figure 1 the display control unit 110 shown in Figure 19 That is to say,

[0113] The function structure shown by the display control device 1900 shown in

[0114] The control part 1902 is configured to control each of the image regions to be displayed on a corresponding projection plane.

[0115] In some examples, the partitioning part 1901 is configured to:

[0116] Estimate the depth information of each pixel in the image to be displayed;

[0117] Determine the depth interval corresponding to the projection plane;

[0118] Divide all the pixels in the image to be displayed into image regions corresponding to the projection plane according to the depth interval in which the depth information of each pixel is located.

[0119] In some examples, the partitioning part 1901 is configured to:

[0120] Obtain the target region and the background region in the image to be displayed; wherein, the number of the target regions is one less than the number of the projection planes;

[0121] Obtain the average depth information estimation value of each target region according to the depth information estimation value of each pixel in each target region;

[0122] Based on the average depth information estimation value of each target region, determine the corresponding projection plane from all the projection planes in the order from the proximal end on the side close to the eye box position to the distal end on the side away from the eye box position;

[0123] Determine the projection plane at the farthest end on one side of the eye box position among all the projection planes as the projection plane corresponding to the background region.

[0124] In some examples, the partitioning part 1901 is configured to:

[0125] Obtain the pixel transmittance of each pixel in each target region according to the global atmospheric light value of the image to be displayed;

[0126] Obtain the average transmittance of each target region according to the pixel transmittance of each pixel in each target region;

[0127] Determine the average depth information estimation value of each target region according to the average transmittance of each target region and the distance of the projection plane at the farthest end on one side of the eye box position among all the projection planes.

[0128] In some examples, the partitioning part 1901 is configured to:

[0129] For the i-th target region O i , obtain the i-th target region O iThe dark channel value of each pixel in;

[0130] Obtain the global atmospheric light value of the image to be displayed according to the maximum value among the dark channel values of all pixels in the background area;

[0131] According to the i-th target area O i Obtain the pixel transmittance of each pixel in the i-th target area O according to the dark channel value of each pixel in and the global atmospheric light value of the image to be displayed i in each pixel.

[0132] In some examples, the partitioning part 1901 is configured to:

[0133] For the i-th target area O i , according to the distance F of the j-th projection plane j and the estimated average depth information D of the i-th target area O i , determine the projection plane identifier k corresponding to the i-th target area O according to the following formula: i where n represents the number of all projection planes. i corresponding projection plane identifier k:

[0134]

[0135] where n represents the number of all projection planes.

[0136] In some examples, the control part 1902 is further configured to:

[0137] Determine the line-of-sight movement information of the eye box based on the change of the fixation point of the eye box;

[0138] Move the position of the image area corresponding to each projection plane according to the line-of-sight movement information so that the position of the moved image area on each projection plane is consistent with the line of sight of the moved eye box.

[0139] In some examples, each projection plane includes a canvas for displaying the corresponding image area; the control part 1902 is further configured to:

[0140] Move the position of the canvas on each projection plane according to the line-of-sight movement information so that the position of the moved canvas on each projection plane is consistent with the line of sight of the moved eye box.

[0141] Please refer to Figure 20, which shows a structural block diagram of a display control device 1900 provided by an exemplary embodiment of the present disclosure. In some examples, the display control device 1700 has a communication function and can access a wired network or a wireless network. In some examples, the display control device 1700 can receive data based on the accessed wired network or wireless network. It can be understood that the display control device 1700 undertakes the computing and processing work of the technical solution of the present disclosure, and the present disclosure does not limit this.

[0142] As Figure 20 shown, the display control device 1900 in the present disclosure may include one or more of the following components: a processor 2010 and a memory 2020.

[0143] Optionally, the processor 2010 connects various parts within the entire computing device using various interfaces and lines, and executes various functions of the computing device and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 2020, and by calling data stored in the memory 2020. Optionally, the processor 2010 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 2010 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), and a baseband chip, etc. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the NPU is used to implement artificial intelligence (AI) functions; the baseband chip is used to process wireless communication. It can be understood that the above baseband chip may not be integrated into the processor 2010 and may be implemented separately by a single chip.

[0144] The memory 2020 may include a Random Access Memory (RAM), or may also include a Read-Only Memory (ROM). Optionally, the memory 2020 includes a non-transitory computer-readable storage medium. The memory 2020 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 2020 may include a program storage area and a data storage area. Among them, the program storage area can store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above method embodiments, etc.; the data storage area can store data created according to the use of the computing device, etc.

[0145] In addition, those skilled in the art can understand that the structure of the computing device shown in the above drawings does not limit the computing device. The computing device may include more or fewer components than shown in the drawings, or combine some components, or have different component arrangements. For example, the computing device may also include components such as a display screen, a camera module, a microphone, a speaker, a radio frequency circuit, an input unit, sensors (such as an acceleration sensor, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, etc., which will not be elaborated here.

[0146] The present disclosure also provides a computer-readable storage medium storing at least one instruction for being executed by a processor to implement the display control method as described in the above embodiments.

[0147] The present disclosure also provides a computer program product including computer instructions stored in a computer-readable storage medium. The processor of the computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computing device to implement the display control method as described in the above embodiments.

[0148] Those skilled in the art should be able to realize that in the above one or more examples, the functions described in the present disclosure can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer storage media and communication media, where the communication media includes any medium facilitating the transmission of a computer program from one place to another. The storage media can be any available medium accessible by a general or special-purpose computer.

[0149] It should be noted that: among the technical solutions recorded in this disclosure, they can be arbitrarily combined without conflict.

[0150] The above is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A display control method, characterized in that, The method includes: Based on the depth information of the image to be displayed, dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit; wherein, the projection planes formed by the display unit are arranged in sequence from the proximal end on the side close to the eyebox position to the distal end on the side away from the eyebox position; Controlling each of the image regions to be displayed on the corresponding projection plane.

2. The method according to claim 1, wherein The step of dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed includes: Estimating the depth information of each pixel in the image to be displayed; Determining the depth intervals corresponding to the projection planes; Dividing all the pixels in the image to be displayed into image regions corresponding to the projection planes according to the depth intervals where the depth information of each pixel is located.

3. The method according to claim 1, wherein The step of dividing the image to be displayed into image regions corresponding to the projection planes formed by the display unit based on the depth information of the image to be displayed includes: Obtaining the target regions and the background region in the image to be displayed; wherein, the number of the target regions is one less than the number of the projection planes; Obtaining the average depth information estimation value of each target region according to the depth information estimation values of each pixel in each target region; Based on the average depth information estimation value of each target region, determining the corresponding projection plane from all the projection planes in the order from the proximal end on the side close to the eyebox position to the distal end on the side away from the eyebox position; Determining the projection plane at the farthest end on one side of the eyebox position among all the projection planes as the projection plane corresponding to the background region.

4. The method according to claim 3, characterized in that, The step of obtaining the average depth information estimation value of each target region according to the depth information estimation values of each pixel in each target region includes: Obtaining the pixel transmittance of each pixel in each target region according to the global atmospheric light value of the image to be displayed; Obtaining the average transmittance of each target region according to the pixel transmittance of each pixel in each target region; Determining the average depth information estimation value of each target region according to the average transmittance of each target region and the distance of the projection plane at the farthest end on one side of the eyebox position among all the projection planes.

5. The method according to claim 4, wherein The step of obtaining the pixel transmittance of each pixel in each target region according to the global atmospheric light value of the image to be displayed includes: For the i-th target region O i , obtain the dark channel value of each pixel in the i-th target region O i ; Obtaining the global atmospheric light value of the image to be displayed according to the maximum value among the dark channel values of all the pixels in the background region; Obtain the pixel transmittance of each pixel in the i-th target region O according to the dark channel value of each pixel in the i-th target region O and the global atmospheric light value of the image to be displayed i and the global atmospheric light value of the image to be displayed i in the i-th target region O 6. The method according to claim 4, characterized in that, The step of determining the corresponding projection plane from all the projection planes in the order from the proximal end on the side close to the eyebox position to the distal end on the side away from the eyebox position based on the average depth information estimation value of each target region includes: For the i-th target region O i , according to the distance F j of the j-th projection plane and the estimated average depth information D i of the i-th target region O i , determine the projection plane identifier k i corresponding to the i-th target region O according to the following formula: wherein, n represents the number of all the projection planes.

7. The method according to claim 1, characterized in that, The method further includes: Determining the line-of-sight movement information of the eyebox based on the change of the fixation point of the eyebox; Moving the positions of the image regions corresponding to each projection plane according to the line-of-sight movement information so that the positions of the moved image regions on each projection plane are consistent with the moved line of sight of the eyebox.

8. The method according to claim 7, wherein Each of the projection surfaces includes a canvas for displaying a corresponding image area; moving the positions of the image areas corresponding to each projection surface according to the line-of-sight movement information so that the positions of the moved image areas on each projection surface are consistent with the line of sight after the movement of the eye box includes: Moving the positions of the canvases on each projection surface according to the line-of-sight movement information so that the positions of the moved canvases on each projection surface are consistent with the line of sight after the movement of the eye box.

9. A display control device, characterized in that, The display control device includes: a partitioning part and a control part; wherein, The partitioning part is configured to partition the image to be displayed into image areas corresponding to the projection surfaces formed by the display part based on the depth information of the image to be displayed; wherein, the projection surfaces formed by the display part are arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side far from the eye box position; The control part is configured to control the display of each of the image areas on the corresponding projection surface.

10. A display control device, characterized in that, The device includes: a processor and a memory; the processor is used to execute the instructions stored in the memory to implement the display control method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the display control method according to any one of claims 1 to 8.

12. A display system, characterized in that, The display system includes a display control part and a display part; wherein, The display control part is configured to partition the image to be displayed into image areas corresponding to the projection surfaces formed by the display part based on the depth information of the image to be displayed; and control the display of each of the image areas on the corresponding projection surface; The display part is configured to form projection surfaces arranged in sequence from the proximal end on the side close to the eye box position to the distal end on the side far from the eye box position; and display each of the image areas on the corresponding projection surface based on the control of the display control part.

13. A vehicle, characterized in that, The vehicle includes the display system according to claim 12.

Images