Method and apparatus for adjusting a display device

Abstract

This application provides a method and an apparatus for adjusting a display device. The method includes: obtaining a first face image of a first user; determining first spatial coordinates of facial feature points of the first user based on the first face image; and adjusting an orientation of a first side of an image displayed by the display device based on the first spatial coordinates, where the first side of the image includes information to be transmitted to the first user. According to the method for adjusting a display device of this application, the orientation of the image displayed by the display device can be accurately adjusted based on the position of the user, so that the user can experience the interaction with the display device. This helps to improve the user experience at the interaction level.

Description

Technical Field

[0001] This application claims the priority of Chinese Patent Application No. 202210505723.1, titled "METHOD AND APPARATUS FOR ADJUSTING DISPLAY DEVICE", filed with the China National Intellectual Property Administration on May 10, 2022, the entire content of which is incorporated herein by reference.

[0002] This application relates to the field of human-computer interaction, and more specifically, to a method and apparatus for adjusting a display device.

Background Art

[0003] With the development of smart vehicles, more imaging display devices, such as in-vehicle large screens, head up display (HUD) devices, and in-vehicle holographic images, are integrated into the cockpit. However, the display range of existing display devices is limited. When the user is in different locations, for the user, these display devices are not necessarily at a good viewing angle or position, and there may be cases where imaging is incomplete and the display cannot be seen.

[0004] Therefore, there is an urgent need to develop a method and apparatus that can accurately adjust the display device based on the user's position.

Summary of the Invention

Problems to be Solved by the Invention

[0005] This application provides a method and apparatus for adjusting a display device that can accurately adjust the orientation of an image displayed by a display device based on the user's position and help improve the user experience at the human-computer interaction level.

Means for Solving the Problems

[0006] According to a first aspect, a method for adjusting a display device is provided, which may be executed by a vehicle, or may be executed by an in-vehicle terminal of a vehicle such as an in-vehicle head unit, or may be executed by a chip or circuit used in a vehicle. This is not limited in the present application. Hereinafter, for ease of explanation, an example in which a vehicle executes the method will be used for the explanation.

[0007] The method may include the steps of obtaining a first face image of a first user, determining first spatial coordinates of face feature points of the first user based on the first face image, and adjusting an orientation of a first side of an image displayed by a display device based on the first spatial coordinates, where the first side of the image includes information to be transmitted to the first user.

[0008] In the above technical solution, the orientation of the image displayed by the display device is accurately adjusted based on the position of the user, thereby improving the user experience at the interaction level.

[0009] In some possible implementations, the first spatial coordinates are three-dimensional coordinates of the face feature points of the user in a vehicle coordinate system.

[0010] For example, the face feature points may be located at any one or more of the center of the eyebrows, eyes, eyebrows, or nose of the first user. Alternatively, the face feature points may be points at other positions on the face of the first user. This is not particularly limited in the embodiments of the present application.

[0011] In some possible implementations, the image may be a three-dimensional image, such as a holographic projection. It should be understood that different angles of the holographic projection contain different information. Alternatively, the image may be a two-dimensional image that interacts with the first user and has three-dimensional features, such as a digital human. It should be understood that different interaction effects can be implemented when the face of the digital human faces different directions. For example, when the face of the digital human faces the first user, the first user can feel the effect of face-to-face communication with the digital human.

[0012] For example, the image may be a three-dimensional image of a virtual assistant, and the first side of the image may be the side that includes the front of the face of the virtual assistant. In another example, the image is a three-dimensional image of a vehicle. To display the style of the front housing of the vehicle to the first user, the first side of the image may be the side that includes the front housing of the vehicle.

[0013] For example, adjusting the orientation of the first side of the image may include adjusting the pose of the image. For example, when the image is a three-dimensional image of a virtual assistant, the pose of the head of the image may be adjusted so that the face of the image faces the first user.

[0014] In connection with the first aspect, in some implementations of the first aspect, after adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, the method further includes obtaining a second face image of the second user, determining second spatial coordinates of the facial feature points of the second user based on the second face image, and adjusting the orientation of the first side of the image displayed by the display device based on the second spatial coordinates.

[0015] In some possible implementations, determining the second spatial coordinates may be actively triggered by the second user. For example, the second user may trigger based on a voice instruction. For example, the second user sends a voice instruction such as "Holographic image, please face me". Alternatively, the second user may trigger by using a related button. For example, the second user taps the "Open holographic image" button. Further, the vehicle controls the camera device to acquire the second face image of the second user, and determines the second spatial coordinates of the facial feature points of the second user.

[0016] In some possible implementations, after the orientation of the first side of the image displayed by the display device is adjusted based on the first spatial coordinates, after a preset time, the second face image of the second user is acquired. For example, the preset time may be 3 seconds or 5 seconds, or other preset times. This is not particularly limited in the embodiments of the present application.

[0017] In the above technical solution, when there are two or more users, the orientation of the image displayed by the display device may be adjusted based on the three-dimensional coordinates of the facial feature points of each user among the plurality of users, and the interaction experience in the multi-user scenario can be improved.

[0018] In connection with the first aspect, in some implementations of the first aspect, after adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, the method further includes the steps of acquiring a third face image of the first user, determining the third spatial coordinates of the facial feature points of the first user based on the third face image, and when the distance between the first spatial coordinates and the third spatial coordinates is greater than or equal to a preset threshold, adjusting the orientation of the first side of the image displayed by the display device based on the third spatial coordinates.

[0019] In some possible implementations, when a first user interacts with an image and the head position of the first user changes, a second face image of the first user may be acquired, and a second coordinate position of facial feature points may be determined based on the second face image.

[0020] Furthermore, when the moving distance of the head of the first user exceeds a preset threshold, the orientation of the first side of the image is adjusted based on the second spatial coordinates.

[0021] For example, the preset threshold may be 15 centimeters, 20 centimeters, or another distance, which is not particularly limited in the embodiments of this application.

[0022] In the above technical solution, when the moving distance of the user's head position exceeds a preset distance, the orientation of the image is adjusted based on the user's moving destination position, thereby avoiding frequent adjustment of the image orientation and improving the user's driving experience.

[0023] In connection with the first aspect, in some implementations of the first aspect, before acquiring the first face image of the first user, the method includes the step of acquiring audio information, where the audio information includes a voice instruction of the first user, and the voice instruction instructs to start and / or adjust a display device, the step of determining the sound source position of the audio information, and the step of determining the position of the first user based on the sound source position.

[0024] In the above technical solution, in a multi-user scenario, when a user sends a voice instruction, based on the sound source position, it may be determined whether the user sending the instruction is the driver, the passenger in the co-driver seat, or the passenger in the rear seat. Accordingly, the display device is adjusted so that the side including the information to be transmitted to the user among the images displayed by the display device faces the user, and the interaction between the user and the images displayed by the display device can be improved.

[0025] In connection with the first aspect, in some implementations of the first aspect, the audio information further includes voice instructions of a second user, and the method further includes controlling a display device to display a first image and a second image, and adjusting the display device so that a first side of the first image faces the first user and a first side of the second image faces the second user, where the image displayed by the display device includes the first image and / or the second image, a first side of the first image includes information to be transmitted to the first user, and a first side of the second image includes information to be transmitted to the second user.

[0026] In the above technical solution, in a multi-user scenario, the display device may further copy the image so that each image faces one of the plurality of users, which can further improve the user's interaction experience and interest in driving.

[0027] In connection with the first aspect, in some implementations of the first aspect, the method further includes adjusting the position of the image displayed by the display device based on the first spatial coordinates.

[0028] In some possible implementations, the position of the image in the y-axis direction in the vehicle coordinate system may be adjusted based on the first spatial coordinates. For example, after the position of the user is determined, the display device may be adjusted so that the image projected by the display device moves to a position corresponding to the position of the user. For example, if the position of the user is the passenger seat, the display device may be adjusted so that the position of the image moves to the passenger seat.

[0029] In some possible implementations, the position of the image in the z-axis direction in the vehicle coordinate system may be adjusted based on the first spatial coordinates. For example, after the position of the user is determined, the display device may be adjusted so that the height of the image projected by the display device moves to a height suitable for the user to view.

[0030] It should be understood that the y-axis direction in the vehicle coordinate system is perpendicular to the longitudinal symmetry plane of the vehicle, and the z-axis direction in the vehicle coordinate system is parallel to the longitudinal symmetry plane of the vehicle.

[0031] In the above technical solution, the display device is adjusted to control the image to move to the user's position or to control the image to move to an appropriate height so that the user does not need to change the orientation to interact with the image displayed by the display device. This helps to improve the user's driving experience.

[0032] In connection with the first aspect, in some implementations of the first aspect, before adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, the method further includes the step of obtaining the initial pose information of the display device, where the initial pose information indicates the pose angle and position of the display device. The step of adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates includes the step of determining the rotation angle and rotation direction of the display device based on the first spatial coordinates and the initial pose information, and the step of adjusting the orientation of the first side of the image displayed by the display device based on the rotation angle and rotation direction.

[0033] In some possible implementations, before the orientation of the first side surface of the image displayed by the display device is adjusted based on the first spatial coordinates, the initial pose information of the image is obtained, where the initial pose information of the image indicates the pose angle and position of the image, and the rotation angle and rotation direction of the image are determined based on the first spatial coordinates and the initial pose information of the image. Further, the rotation angle and rotation direction of the display device are determined based on the mapping relationship between the angle change of the display device and the angle change of the image, and then the display device is adjusted. For example, when the image is a virtual assistant, the pose angle of the image may be the head pose angle of the virtual assistant.

[0034] In relation to the first aspect, in some implementations of the first aspect, the first spatial coordinates of the facial feature points of the first user indicate the three-dimensional position of the facial feature points of the first user within the vehicle, and the vehicle includes a display device.

[0035] In the above technical solution, the display device can be adjusted based on the three-dimensional coordinates of the facial feature points of the user in the vehicle coordinate system. As a result, the orientation of the image displayed by the display device can be accurately adjusted, providing a good interaction effect for the user and improving the user's interaction experience.

[0036] In relation to the first aspect, in some implementations of the first aspect, the display device includes a holographic projection device, and the image includes a three-dimensional 3D image.

[0037] According to the second aspect, an apparatus for adjusting a display device is provided. The apparatus includes an acquisition unit configured to acquire a first facial image of a first user, and a processing unit configured to determine first spatial coordinates of facial feature points of the first user based on the first facial image and adjust the orientation of the image on the first side displayed by the display device based on the first spatial coordinates.

[0038] In relation to the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to acquire a second facial image of a second user after adjusting the orientation of the image on the first side displayed by the display device based on the first spatial coordinates. The processing unit is further configured to determine second spatial coordinates of facial feature points of the second user based on the second facial image and adjust the orientation of the image on the first side displayed by the display device based on the second spatial coordinates.

[0039] In connection with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to, based on the first spatial coordinates, adjust the orientation of the first side of the image displayed by the display device, and then acquire the third face image of the first user. The processing unit is configured to, based on the third face image, determine the third spatial coordinates of the facial feature points of the first user, and when the distance between the first spatial coordinates and the third spatial coordinates is greater than or equal to a preset threshold, further configured to adjust the orientation of the first side of the image displayed by the display device based on the third spatial coordinates.

[0040] In connection with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to acquire audio information before acquiring the first face image of the first user, the audio information includes a voice instruction of the first user, and the voice instruction instructs to start and / or adjust the display device. The processing unit is configured to determine the sound source position of the audio information and further configured to determine the position of the first user based on the sound source position.

[0041] In connection with the second aspect, in some implementations of the second aspect, the audio information further includes a voice instruction of the second user, and the processing unit is configured to control the display device to display the first image and the second image, and further configured to adjust the display device so that the first side of the first image faces the first user and the first side of the second image faces the second user. The image displayed by the display device includes the first image and / or the second image, the first side of the first image includes the information to be transmitted to the first user, and the first side of the second image includes the information to be transmitted to the second user.

[0042] In connection with the second aspect, in some implementations of the second aspect, the processing unit is further configured to adjust the position of the image displayed by the display device based on the first spatial coordinates.

[0043] In connection with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to acquire initial posture information of the display device before adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, where the initial posture information indicates the posture angle and position of the display device. The processing unit is further configured to determine the rotation angle and rotation direction of the display device based on the first spatial coordinates and the initial posture information, and to adjust the orientation of the first side of the image displayed by the display device based on the rotation angle and the rotation direction.

[0044] In connection with the second aspect, in some implementations of the second aspect, the first spatial coordinates of the facial feature points of the first user indicate the three-dimensional position of the facial feature points of the first user within the vehicle, and the vehicle includes a display device.

[0045] In connection with the second aspect, in some implementations of the second aspect, the display device includes a holographic projection device, and the image includes a three-dimensional (3D) image.

[0046] According to a third aspect, there is provided an apparatus for adjusting a display device, the apparatus including a memory configured to store a computer program, and a processor configured to execute the computer program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute a method according to any one of the possible implementations of the first aspect.

[0047] According to a fourth aspect, a vehicle is provided. The vehicle includes a device according to any one of the possible implementations of the second aspect or a device according to any one of the possible implementations of the third aspect, and a display device. The display device may be a holographic projection device, and the displayed image may be a three-dimensional 3D image. Alternatively, the display device may be an in-vehicle display, and the displayed image may be a digital human. Alternatively, the display device may be another display device. This is not particularly limited in the present application.

[0048] According to a fifth aspect, a computer program product is provided. The computer program product includes computer program code. When the computer program code is executed by a computer, the computer is capable of executing a method according to any one of the possible implementations of the first aspect.

[0049] It should be noted that all or part of the computer program code may be stored in a first storage medium. The first storage medium may be encapsulated together with the processor or may be encapsulated separately from the processor. This is not particularly limited in the embodiments of the present application.

[0050] According to a sixth aspect, a computer-readable medium is provided. The computer-readable medium stores program code. When the computer program code is executed by a computer, the computer is capable of executing a method according to any one of the possible implementations of the first aspect.

[0051] According to a seventh aspect, a chip is provided. The chip includes a processor configured to call a computer program or computer instructions stored in a memory, as a result of which the processor is capable of executing a method according to any one of the possible implementations of the first aspect.

[0052] In relation to the seventh aspect, in a possible implementation, the processor is coupled to the memory via an interface.

[0053] In relation to the seventh aspect, in a possible implementation, the chip system further includes a memory, and the memory stores a computer program or computer instructions.

[0054] According to the method for adjusting a display device according to an embodiment of the present application, based on the position of the user, the orientation of the image displayed by the display device can be accurately adjusted, and the user experience at the interaction level can be improved. The display device can be adjusted based on the position where the user's head moves. When the moving distance of the user's head exceeds a preset distance, the orientation of the image is adjusted based on the position where the user's head has moved. Thereby, frequent adjustment of the orientation of the image can be avoided, and it can help improve the user's driving experience. When there are two or more users, based on the three-dimensional coordinates of the facial feature points of each of the plurality of users, the orientation of the image displayed by the display device can be adjusted to improve the interaction experience in a multi-user scenario. Alternatively, based on the voice instruction sent by the user, the position of the user can be determined, and then the display device is adjusted based on the position of the user. In a multi-user scenario, when the user sends a voice instruction, based on the sound source position, it is determined whether the user sending the instruction is the driver, the passenger in the front passenger seat, or the passenger in the rear seat, and accordingly, the display device is adjusted so that the side including the information to be transmitted to the user among the images displayed by the display device faces the user, and the interaction between the user and the image displayed by the display device can be improved. In a multi-user scenario, the display device can further copy the image so that each image faces one of the plurality of users, and further improve the user's interaction experience and interest in driving. Furthermore, the display device is adjusted to control the image to move to the position of the user or to move to an appropriate height so that the user does not need to change the orientation to interact with the image displayed by the display device. This helps improve the user's driving experience.

Brief Description of the Drawings

[0055]

Figure 1

[0056]

Figure 2

[0057]

Figure 3

[0058]

Figure 4

[0059]

Figure 5

[0060]

Figure 6(a)

Figure 6(b)

Figure 6(c)

Figure 6(d)

Figure 6(e)

[0061]

Figure 7

[0062]

Figure 8(a)

Figure 8(b)

[0063]

Figure 9

[0064]

Figure 10

[0065]

Figure 11

Embodiments for Carrying Out the Invention

[0066] Hereinafter, with reference to the accompanying drawings, the technical solutions of the embodiments in the present application will be described.

[0067] Figure 1 is a schematic of a vehicle cockpit scenario according to an embodiment of the present application. One or more cameras, such as a camera of a driver monitor system (DMS), a camera of a cabin monitor system (CMS), and a camera of a dashcam, are installed inside the smart cockpit and can capture images inside or outside the cockpit. In Figure 1(a), a camera arranged on the A-pillar is used as an example. The cameras used to capture inside and outside the cockpit may be the same camera or different cameras. In addition, an in-vehicle display is further arranged inside the cockpit. In Figure 1(a), a display arranged in the central control area is used as an example. The vehicle can display information to the user by using at least one of an in-vehicle display, a HUD (not shown in Figure 1(a)), and an in-vehicle holographic projection device (not shown in Figure 1(a)). For example, by using an in-vehicle holographic projection device, a three-dimensional image of a virtual assistant can be displayed to the user. Details are shown in Figure 1(a). In some possible implementations, the in-vehicle holographic projection device can further perform imaging on the rear of the headrest of the front seat. Details are shown in Figure 1(b). The three-dimensional image shown in Figure 1(a) is only an example for explanation, and it should be understood that the specific content of the three-dimensional image may alternatively be another three-dimensional object, for example, alternatively a three-dimensional image of the vehicle. It should be understood that in the embodiment of the present application, the position of the camera that collects image information inside the cockpit is not particularly limited. The camera may be located on the A-pillar shown in Figure 1(a), on the B-pillar, under the steering wheel, near the rearview mirror, etc.

[0068] FIG. 2 is a functional block diagram of a vehicle 100 according to an embodiment of the present application. The vehicle 100 may include a sensing system 120, a display device 130, and a computing platform 150. The sensing system 120 may include several types of sensors that sense information regarding the environment around the vehicle 100 and the environment of the vehicle cockpit. For example, the sensing system 120 may include a positioning system. The positioning system may be a global positioning system (GPS), or may be a BeiDou system or another positioning system, an inertial measurement unit (IMU), a lidar, a millimeter wave radar, an ultrasonic radar, a vision sensor, a sound sensor, and one or more of a camera device. The camera device may include a red, green, and blue / infrared (RGB / IR) camera, or may include a depth camera, such as a time of flight (TOF) camera, a stereo camera, or a structured light camera.

[0069] Some or all of the functions of vehicle 100 may be controlled by computing platform 150. Computing platform 150 may include processors 151 to 15n (n is a positive integer). A processor is a circuit having signal processing capabilities. In one implementation, a processor is a circuit having the ability to read and execute instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), a digital signal processor (DSP), etc. In another implementation, a processor may implement a specific function based on the logical relationship of hardware circuits. The logical relationship of the hardware circuits is fixed or reconfigurable. For example, a processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), such as a field programmable gate array (FPGA). In a reconfigurable hardware circuit, the process by which a processor loads a document to implement a hardware circuit configuration may be understood as the process by which a processor loads instructions to implement some or all of the functions of the aforementioned units. Additionally, a processor may be a hardware circuit designed for artificial intelligence, and may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU). Additionally, computing platform 150 may further include a memory. The memory is configured to store instructions.Some or all of Processors 151 to 15n may call instructions in the memory and execute the instructions to implement corresponding functions.

[0070] The display device 130 in the cockpit is mainly classified into three categories. The first category is the in-vehicle display. The second category is a projection display such as an HUD. The third category is a holographic projection device. The in-vehicle display is a physical display and an important part of the in-vehicle infotainment system. Multiple displays can be arranged in the cockpit, such as a digital instrument display, a central control screen, a display in front of the co-driver (also called the passenger seat), a display in front of the left rear passenger, and a display in front of the right rear passenger. The window can also be used as a display for display. The head-up display, also called a head-up display system, includes a combiner-type head-up display (C-HUD) system, a windshield-type head-up display (W-HUD) system, and an augmented reality head-up display (AR-HUD) system. The holographic projection system is a display device that can display a three-dimensional (3D) image of an object. In some possible implementations, the holographic projection device may record and reproduce a 3D image of an object based on the principles of light interference and diffraction, or may further implement a 3D effect by methods such as edge blanking or Pepper's ghost. In the embodiments of the present application, the holographic projection device may display an image of a 3D virtual assistant, or a 3D image of a vehicle, or another 3D image. This is not particularly limited in the embodiments of the present application. In some possible implementations, the display device 130 may further display a digital human. A digital human may be a virtual image that has human characteristics such as a human appearance and personality, has language, facial expression, and body movement expression capabilities, and can communicate and interact with humans. The image can be displayed on an in-vehicle display or the like.

[0071] It should be understood that the image displayed by the HUD is a two-dimensional image projected onto the HUD screen. The digital humans displayed on the in-vehicle display are also included in two-dimensional images with a three-dimensional effect and depth of field. The image displayed by the holographic projection device is a three-dimensional image. Specifically, when viewing a holographic image from different angles, different sides of the object can be seen, and a parallax effect and a depth effect can be obtained.

[0072] In an embodiment of the present application, the processor may obtain user image information collected by the sensing system 120 and determine the position of the user's eyes and the user's fixation point with reference to the three-dimensional coordinate system of the vehicle 100. When it is determined that the user's fixation point is on the display device and / or the image displayed by the display device, the display device is adjusted based on the actual position of the user within the vehicle 100. In some possible implementations, the processor may further determine the position of the user based on the sound information collected by the sound sensor in the sensing system 120, and then adjust the angle of the image projected by the holographic projection device based on the position of the user. In some possible implementations, the user image information and the sound information may alternatively be stored in the memory of the computing platform 150 in the form of data. In some possible implementations, the processor may process the user image information and the sound information to obtain a parameterized indicator, and then adjust the display device based on the parameterized indicator. It should be understood that the above operations may be executed by the same processor or alternatively by one or more processors. This is not particularly limited in the embodiments of the present application.

[0073] Hereinafter, with reference to FIG. 3, the workflow of the system for adjusting the display device will be described in detail. FIG. 3 is a diagram of the architecture of the system for adjusting the display device according to an embodiment of the present application. The system 200 includes a sensing module, a calibration module, a display device adjustment module, and a display module. The sensing module may include one or more camera devices and one or more sensors within the sensing system 120 shown in FIG. 2. The display module may include one or more display devices within the display device 130 shown in FIG. 2. The calibration module and the display device adjustment module may include one or more processors within the computing platform 150 shown in FIG. 2. Specifically, the sensing module may detect the face information and / or sound information of the user. The display device adjustment module determines adjustment parameters based on the face image information obtained from the sensing module and the extrinsic parameter in the vehicle coordinate system and obtained from the calibration module, and then may determine the actual three-dimensional coordinates of any position of the face of the user in the vehicle with reference to the extrinsic parameter in the vehicle coordinate system and the face image information. Further, the display device adjustment module may adjust the display device within the display module based on the three-dimensional coordinates of one or more feature points of the user's face. For example, the feature point of the user's face may be the eye, or may be the point between the two eyes, or may be another point. This is not particularly limited in the embodiments of the present application.

[0074] It should be understood that the above-mentioned modules and devices are merely examples. In actual applications, the above-mentioned modules and devices may be added or deleted according to actual requirements. For example, the display module and the display device adjustment module in FIG. 3 may alternatively be combined into one module. In other words, the functions of the display module and the display device adjustment module are implemented by one module.

[0075] As described above, in the current technical background, the display range of display devices is limited. When a user is in a different location, for the user, these display devices are not necessarily in a good viewing angle or position, and imaging may be incomplete and the display may not be visible. In particular, in a holographic projection device, the direction of the three-dimensional image projected by the holographic projection device is fixed, and the images seen by the user at different positions are different. As a result, a user who wants to obtain information transmitted by the image cannot obtain the relevant information timely and comprehensively. In this regard, embodiments of the present application can adjust the holographic projection device based on the user's position, and can accurately adjust the three-dimensional image projected by the holographic projection device based on the user's position, so as to improve the interaction with the user and improve the user's driving experience, a method and an apparatus for adjusting a display device are provided.

[0076] FIG. 4 shows a method for adjusting a display device according to an embodiment of the present application. Method 400 may be applied to the scenario shown in FIG. 1, or may be applied to vehicle 100 shown in FIG. 2. Also, the method may be executed by the system shown in FIG. 3. Method 400 includes the following steps.

[0077] S401: Obtain a first face image of the user.

[0078] For example, after the display device is turned on, a first face image of the user may be obtained by using a camera disposed inside the vehicle, where the first face image includes an image of the user's eyes. For example, in order to facilitate obtaining a face image of the driver, a camera may be disposed on the A-pillar, so that the camera can capture both eyes of the user. It should be understood that the user may be any user in the cockpit of the vehicle, for example, the driver, the user in the passenger seat, or the user in the rear passenger seat in the cockpit.

[0079] S402: Determine whether the user's fixation focus is the displayed image.

[0080] Specifically, if it is determined that the user's fixation focus is the displayed image, S403 is executed. Otherwise, S405 is executed.

[0081] In some possible implementations, if the display device is a holographic projection device and the image projected by the holographic projection device is a virtual 3D image, determine whether the user's fixation focus is on the virtual 3D image.

[0082] For example, the user's fixation focus may be determined by using gaze estimation technology, gaze tracking technology, or the like.

[0083] S403: Determine the first position information of the facial feature points based on the first facial image.

[0084] For example, the first position information indicates the first position of the facial feature points in the vehicle coordinate system.

[0085] In some possible implementations, one facial feature point may be determined based on the first facial image, or two or more facial feature points may be determined based on the first facial image. This is not particularly limited in the embodiments of the present application.

[0086] For example, the facial feature points may be located at any one or more of the center of the eyebrows, eyes, eyebrows, or nose. Alternatively, the facial feature points may be points at other positions on the user's face. This is not particularly limited in the embodiments of the present application.

[0087] Furthermore, based on the image coordinates of the facial feature points in the first facial image, the three-dimensional coordinates of the facial feature points in the vehicle cockpit, that is, the first position information, are determined.

[0088] In a specific implementation process, alternatively, a face feature area may be determined based on the first face image, and then the first position information may be determined based on the face feature area. Alternatively, the first position information may be determined based on another feature point or area determined from the first face image. This is not particularly limited in this application.

[0089] For details on how to determine the first position information based on image coordinates, please refer to the following description.

[0090] The image coordinate system is a coordinate system established for an image captured by a camera. The image coordinate system may use the center of the image captured by the camera as the origin, and the image coordinate system is a two-dimensional coordinate system. The camera coordinate system uses the optical center of the camera as the origin of the coordinate system. The Xc axis and the Yc axis are parallel to the x axis and the y axis of the image coordinate system. The optical axis of the camera is the Zc axis. The coordinate system follows the right-hand rule. The optical center of the camera may be understood as the geometric center of the camera lens, and the camera coordinate system is a three-dimensional coordinate system. The conversion relationship between the image coordinate system and the camera coordinate system can be shown in FIG. 5. In FIG. 5, the two-dimensional coordinate system o-xy is the image coordinate system of the image q captured by the camera. The intersection of the x axis and the y axis is o, that is, o is the origin of the image coordinate system, and the coordinates of any point on the image plane q may be represented by image coordinates (x, y). The three-dimensional coordinate system O-XYZ is the camera coordinate system, and any plane in the camera coordinate system may be called a spatial plane. The spatial plane Q is any plane in the spatial coordinate system (that is, any plane in O-XYZ), and the coordinates of any point on the spatial plane Q may also be represented by three-dimensional coordinates (X, Y, Z). The image plane q and the spatial plane Q have a conversion relationship of q = sHQ. q represents the image coordinates of a point on the image plane, Q represents the three-dimensional coordinates of a point on the spatial plane, s represents a scale factor, and the homography matrix H = MW.

[0091]

Number

[0092] W = [R, t], where W is the external parameter matrix of the camera. R is a 3×3 orthogonal unit matrix also called the rotation matrix, and t is a 3D translation vector. When the image plane q has the above transformation relationship with the spatial plane Q and the image coordinates, scale factor, and homography matrix H of the points on q are known, the spatial coordinates of the points on Q can be calculated based on the above transformation relationship between the image plane and the spatial plane.

[0093] For example, in FIG. 5, the image coordinates of the points on the plane q are (x 1 , y 1 ). When the intrinsic parameter matrix of the camera is M, the physical transformation matrix of the external parameters of the camera is W, and the scale factor is s, based on the transformation relationship between the image plane and the world plane, an equation with X 1 , Y 1 , Z 1 as unknowns can be obtained:

Equation

[0094] Furthermore, the coordinates (X 1 , Y 1 , Z 1 ) of the points on the spatial plane Q are converted into the vehicle coordinate system, and the 3D coordinates (X 2 , Y 2 , Z 2 ) of the points in the vehicle coordinate system are obtained.

[0095] For example, the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) may be determined by a perspective-n-point (PnP) algorithm based on a conventional head model. The conventional head model includes the relationship between the coordinates of specific points in the vehicle coordinate system and the image coordinate system. Specifically, the coordinates of specific points in the vehicle coordinate system and the camera coordinate system have the following relationship: [Number] Here, p is the coordinate of the point in the pixel coordinate system, P c is the coordinate of the point in the camera coordinate system, P W is the coordinate of the point in the vehicle coordinate system, ω is the depth of the point, K is the intrinsic parameter matrix of the camera, R cw and [Number] is the pose transformation from the vehicle coordinate system to the camera coordinate system. More specifically, R cw is the rotation matrix from the vehicle coordinate system to the camera coordinate system (converting the representation of the same vector in the vehicle coordinate system to the representation in the camera coordinate system), [Number] is the corresponding translation vector (i.e., the representation in the camera coordinate system of the vector pointing from the origin of the camera coordinate system to the origin of the vehicle coordinate system). Furthermore, based on the coordinates of the n points of the conventional head model in the vehicle coordinate system and the coordinates of the n points in the image coordinate system, R cw and [Number] can be solved. Furthermore, the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) of the facial feature points in the vehicle coordinate system are transformed into the coordinates (X 1 , Y1 , Z 1 can be obtained based on

[0096] In some possible implementations, by using a face keypoint detection algorithm, the 3D position of the corresponding position of a human face can be obtained, and then the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) of the face feature points in the vehicle coordinate system can be determined. Alternatively, by using a head reconstruction algorithm, the user's head can be directly reconstructed in real time, and the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) of the face feature points in the vehicle coordinate system can be determined.

[0097] It should be understood that the three-dimensional coordinates of the face feature points in the cockpit can provide a more accurate reference for adjusting the display device.

[0098] S404: Adjust the display direction of the display device based on the first position information.

[0099] For example, the display device in the embodiments of the present application is a holographic projection device, and the image displayed by the display device is a 3D image. For example, the direction of the holographic image displayed by the display device may be adjusted based on the first position information so that the side of the holographic image containing the content to be transmitted to the user faces the user. For example, when the holographic image is a 3D image of a virtual assistant, the face of the virtual assistant or the body and face of the virtual assistant can be adjusted to face the user. As shown in FIG. 6(a), when the user is the user in the passenger seat, the displayed holographic image of the virtual assistant may be adjusted to face the user in the passenger seat. In some possible implementations, the orientation of the digital human face displayed by the display device may be adjusted based on the first position information.

[0100] In some possible implementations, the preset direction of the holographic image displayed by the display device may be shown in FIG. 6(b). For example, when the holographic image is a virtual assistant, the preset direction may be the direction of the face of the virtual assistant. When the holographic image is another image, the preset direction may be the direction of the side containing the content to be transmitted to the user.

[0101] Furthermore, adjusting the display direction of the display device based on the first position information may include adjusting the posture of the holographic image such that the side of the holographic image containing the content to be transmitted to the user faces the first position of the user's facial feature points, that is, the preset direction shown in FIG. 6(b) faces the first position of the facial feature points specifically shown in FIG. 6(c).

[0102] In some possible implementations, the posture of the holographic image can be adjusted by adjusting the angle of the holographic projection device. For example, the holographic projection device includes a projection device and an imaging device. Here, the projection device projects light onto the imaging device, and as a result, the imaging device can render a three-dimensional stereoscopic image. In some possible implementations, when the projection device rotates, the three-dimensional stereoscopic image rotates. In this case, the rotation angle of the projection device in a plane perpendicular to the longitudinal symmetry plane of the vehicle can be adjusted based on the first position information such that the side of the holographic image containing the content to be transmitted to the user faces the user. In some possible implementations, there is a mapping relationship between the angle conversion of the holographic projection device and the angle conversion of the holographic image. For example, when the holographic projection device rotates clockwise by α degrees (degree, °) in a plane perpendicular to the longitudinal symmetry plane of the vehicle, the holographic image rotates clockwise by β degrees in a plane perpendicular to the longitudinal symmetry plane of the vehicle. α and β may or may not be equal. This is not limited to the embodiments of the present application.

[0103] For example, the preset direction shown in FIG. 6(b) indicates the initial posture of the holographic image. This preset direction indicates the posture angle of the holographic image as 0 degrees, and the coordinates of the holographic image in the vehicle coordinate system are (X 3 , Y 3 , Z 3 ). The coordinates of the first position shown in FIG. 6(c) in the vehicle coordinate system are (X 4 , Y 4 , Z 4 ). In this case, based on the first position information and the initial posture of the holographic image, it can be determined that the first rotation angle θ 1 is arctan((X4 - X3) / (Y4 - Y3)). Here, the first rotation direction is clockwise rotation in a plane perpendicular to the longitudinal symmetry plane of the vehicle. Further, based on the mapping relationship between the angle conversion of the holographic projection device and the angle conversion of the holographic image, the rotation angle θ 2 of the holographic projection device is determined. Next, the holographic projection device is controlled to rotate clockwise by θ 2 in a plane perpendicular to the longitudinal symmetry plane of the vehicle so that the side of the holographic image containing the content to be transmitted to the user faces the user.

[0104] For example, the projection device may implement synchronous playback of a digital high-definition multi-channel hard disk. The imaging device may include a spectroscope. Alternatively, the projection device and the imaging device may include another device. This is not particularly limited in the embodiments of the present application.

[0105] In some possible implementations, based on the first position information, the posture of the three-dimensional stereoscopic image may be further adjusted. For example, when the three-dimensional stereoscopic image is a virtual assistant, the orientation of the head and / or body of the virtual assistant may be adjusted so that the head of the virtual assistant faces the first position.

[0106] In some possible implementations, the height of the image displayed by the holographic projection device may be further adjusted based on the first position information. For example, the height of the image displayed by the holographic projection device is adjusted based on the height information of the first position information, and the holographic projection device is controlled to display a holographic image at the corresponding height. For example, the rotation angle of the projection device in a plane parallel to the longitudinal symmetry plane of the vehicle may be controlled based on the height information at the first position, and thus, the height of the holographic image can be controlled.

[0107] S405: Obtain the second face image of the user, and determine the second position information of the facial feature points based on the second face image.

[0108] For example, the second position information indicates the second position of the facial feature points in the vehicle coordinate system.

[0109] In some possible implementations, when the user interacts with the holographic image and the head position of the user changes, the second face image of the user can be obtained, and the second position information of the facial feature points is determined based on the second face image. Specifically, for the method of determining the second position information of the facial feature points, please refer to the description of S403. Details will not be described again here.

[0110] S406: Determine whether the distance difference between the first position and the second position is greater than or equal to the first threshold.

[0111] Specifically, when it is determined that the distance difference between the first position and the second position is greater than or equal to the first threshold, S407 is executed. Otherwise, S408 is executed.

[0112] When the distance difference is greater than or equal to a preset threshold, the adjustment of the display device is triggered, and as a result, frequent adjustment of the display device can be avoided.

[0113] For example, the first threshold value may be 15 centimeters (cm), 20 cm, or another distance, which is not particularly limited in the embodiments of the present application.

[0114] S407: Adjust the display direction of the display device based on the second position information.

[0115] In some possible implementations, both the first position and the second position are the positions of the facial feature points of the first user (for example, the user in the driver's seat) in the vehicle coordinate system. Therefore, adjusting the display direction of the display device based on the second position information can be shown in FIG. 6(d).

[0116] In some possible implementations, the first position is the position of the facial feature points of the first user (for example, the user in the driver's seat) in the vehicle coordinate system, and the second position is the position of the facial feature points of the second user (for example, the user in the passenger seat) in the vehicle coordinate system. Therefore, adjusting the display direction of the display device based on the second position information can be shown in FIG. 6(e). For example, when the second position and the first position indicate different users, the determination of the second position may be actively triggered by the second user. For example, the second user may trigger based on a voice instruction, such as "Holographic image, turn towards me". Alternatively, the second user may trigger by using a related button. For example, the second user taps the "Open Holographic Image" button.

[0117] For example, the angle and direction by which the display device needs to rotate may be determined based on the first position information and the second position information, and then the display device is controlled to rotate so that the display direction of the display device faces the second position. In some possible implementations, the display device rotates within a plane perpendicular to the longitudinal symmetry plane of the vehicle.

[0118] S408: End.

[0119] It should be understood that the above "end" means ending the process of adjusting the display device.

[0120] The steps and operations of the method for controlling the display device shown in FIG. 4 are merely examples for illustration. In the embodiments of the present application, other operations or variations of each operation in FIG. 4 may be further executed. In addition, the steps in FIG. 4 may be executed in a sequence different from the sequence shown in FIG. 4, and some operations in FIG. 4 may not need to be executed. For example, S405 to S407 may not be executed. For example, S402 may be skipped and S403 may be directly executed.

[0121] According to the method for adjusting the display device provided in the embodiments of the present application, the display direction of the display device can be adjusted based on the three-dimensional coordinates of the feature points of the user's head in the vehicle coordinate system. As a result, the orientation of the image displayed by the display device can be accurately adjusted, providing a better interaction effect for the user and improving the user's interaction experience.

[0122] FIG. 7 shows a method for adjusting a display device according to an embodiment of the present application. Method 700 may be applied to the scenario shown in FIG. 1, or may be applied to the vehicle 100 shown in FIG. 2. Also, the method may be executed by the system shown in FIG. 3. Each step or operation of the method for controlling the display device shown in FIG. 7 is merely an example for illustration. In the embodiments of the present application, other operations or variations of each operation in FIG. 4 may be further executed. Method 700 includes the following steps.

[0123] S701: Obtain audio data inside the vehicle and determine the sound source position of the audio data.

[0124] For example, the audio information may be audio information obtained after excluding various invalid audio information from the audio information collected inside the vehicle, and the invalid audio information may be audio information with an overly low volume. The sound source position may be the position of the sound source corresponding to the audio information. The sound source position may also be the relative position with respect to an optical display device based on the positioning of the sound source, or may be specific position coordinates. This is not particularly limited in the embodiments of the present application.

[0125] For example, the sound source position may be determined based on the time difference of arrival (TDOA) principle and the audio information collected by a plurality of sound sensors. For example, sound sensor A and sound sensor B each detect that audio has been emitted from sound source S. The time when the sound source signal of sound source S reaches sound sensor A is t1, and the time when the sound source signal of sound source S reaches sound sensor B is t2. Therefore, the time difference is dt = |t1 - t2|. When the distance between sound source S and sound sensor A is set as AS, the distance between sound source S and sound sensor B is set as BS, and the speed of sound is set as c, dt = t1 - t2 = AS / c - BS / c can be obtained. Then, one of the sensors is selected as a reference point, and the position of the sound source can be determined based on the distance a between the two sound sensors.

[0126] In some possible implementations, the audio information may be voice including specific activation words such as, for example, "Activate the display device" or "Virtual assistant, turn towards me" or other voice information.

[0127] S702: Determine the user's position based on the sound source position.

[0128] In some possible implementations, within a preset time, two sound source positions are determined based on the acquired audio data. For example, if the sound source positions are determined to be at the driver's seat and the passenger seat respectively, the display device may be controlled to execute an image "copy". As shown in FIG. 8(a), the two images acquired after the copy are respectively facing the user at the driver's seat and the user at the passenger seat. For example, the image is a 3D image of a virtual assistant, and the two copied images are controlled by the same command.

[0129] In some possible implementations, within a preset time, two sound source positions are determined based on the acquired audio data. For example, if the sound source positions are determined to be at the driver's seat and the passenger seat respectively, the display device may be controlled to display two images, and the two images are respectively controlled to face the user at the driver's seat and the user at the passenger seat.

[0130] For example, the preset time may be 3 seconds or 5 seconds, or other preset times.

[0131] In some possible implementations, the display device is disposed on the armrest at the center of the front seats. Also, when the sound source positions are determined to be at the left rear seat and the right rear seat respectively, the display device may be controlled to project two images, and the two images are respectively facing the user at the left rear seat and the user at the right rear seat.

[0132] In some possible implementations, when the number of users is more than two, the display device may be controlled to project a plurality of images, and the plurality of images may be adjusted to face each of the plurality of users.

[0133] In some possible implementations, after the position of the user is determined based on the sound source position, the display device may be adjusted such that the image projected by the display device moves to the sound source position. For example, when the sound source position is the passenger seat, as shown in FIG. 8(b), the display device may be adjusted to move the position of the image to the passenger seat.

[0134] In some possible embodiments, the holographic projection device includes a projection device and an imaging device, where the projection device projects light onto the imaging device, and as a result, the imaging device can render a three-dimensional stereoscopic image. In some possible implementations, the position of the imaging device is adjusted to the passenger seat, and then the projection angle of the projection device is adaptively adjusted to form a holographic image at the passenger seat.

[0135] According to the method for adjusting a display device according to an embodiment of the present application, in a multi-user scenario, when a user sends a voice instruction, based on the sound source position, it may be determined whether the user sending the instruction is the driver, a passenger in the passenger seat, or a passenger in the rear seat, and accordingly, the display device is adjusted such that the 3D image projected by the display device faces the user, improving the interaction between the user and the image projected by the display device. In a multi-user scenario, the display device may further copy the image, and as a result, each image faces one of the plurality of users, further improving the user's interaction experience and interest in driving.

[0136] FIG. 9 shows a method for adjusting a display device according to an embodiment of the present application. Method 900 may be applied to the scenario shown in FIG. 1, or may be applied to the vehicle 100 shown in FIG. 2. Also, this method may be executed by the system shown in FIG. 3. The steps and operations of the method for controlling the display device shown in FIG. 9 are merely examples for illustration. In an embodiment of the present application, another operation or a variation of each operation in FIG. 4 may be further executed. Method 900 includes the following steps.

[0137] S901: Obtain the first face image of the first user.

[0138] Specifically, for the method of obtaining the first face image of the first user, please refer to the description of the above-described embodiments. Details will not be described again here.

[0139] S902: Based on the first face image, determine the first spatial coordinates of the facial feature points of the first user.

[0140] In some possible implementations, the first spatial coordinates may be the three-dimensional coordinates of the facial feature points of the first user in the vehicle coordinate system.

[0141] For example, the facial feature points of the first user may be the points in the above-described embodiments. Furthermore, for the method of determining the first spatial coordinates, please refer to the description of the foregoing embodiments. Details will not be described again here.

[0142] S903: Based on the first spatial coordinates, adjust the orientation of the first side of the image displayed by the display device, where the first side of the image contains the information to be transmitted to the first user.

[0143] For example, the image may be a holographic image or a binary image with three-dimensional effects and depth of field. This is not particularly limited in the embodiments of the present application.

[0144] Specifically, for the method of adjusting the orientation of the image displayed by the display device based on the first spatial coordinates, please refer to the description of the above-described embodiments. Details will not be described again here.

[0145] According to the method for adjusting the display device provided in the embodiments of the present application, the user can experience the interaction with the display device, improving the user experience at the interaction level.

[0146] In various embodiments of the present application, unless otherwise specified or there is no logical contradiction, the terms and / or descriptions in each embodiment are consistent and may be cross-referenced to each other. The technical features in different embodiments may be combined based on their internal logical relationships to form new embodiments.

[0147] In the above, with reference to FIGS. 4 to 9, the method provided in the embodiments of the present application has been described in detail. Hereinafter, with reference to FIGS. 10 and 11, the apparatus provided in the embodiments of the present application will be described in detail. It should be understood that the description of the embodiments of the apparatus corresponds to the description of the embodiments of the method. Therefore, for the content not described in detail, please refer to the foregoing embodiments of the method. For the sake of brevity, the details will not be described again here.

[0148] FIG. 10 is a block diagram of an apparatus 2000 for adjusting a display device according to an embodiment of the present application. The apparatus 2000 includes an acquisition unit 2010 and a processing unit 2020. The acquisition unit 2010 may implement the corresponding communication function, and the processing unit 2020 is configured to process data.

[0149] Optionally, the apparatus 2000 may further include a storage unit. The storage unit may be configured to store instructions and / or data. The processing unit 2020 may read the instructions and / or data in the storage unit. As a result, the apparatus implements the embodiments of the foregoing method.

[0150] The apparatus 2000 may include units configured to execute the methods of FIGS. 4, 7, and 9. In addition, the units in the apparatus 2000 and the other operations and / or functions described above are separately used to implement the corresponding procedures of the embodiments of the methods of FIGS. 4, 7, and 9.

[0151] When the device 2000 is configured to execute the method 900 of FIG. 9, the acquisition unit 2010 may be configured to execute S901 of the method 900, and the processing unit 2020 may be configured to execute S902 and S903 of the method 900.

[0152] The device 2000 includes an acquisition unit 2010 configured to acquire a first face image of a first user, and a processing unit 2020 configured to determine first spatial coordinates of face feature points of the first user based on the first face image and adjust an orientation of a first side of an image displayed by a display device based on the first spatial coordinates. The first side of the image includes information to be transmitted to the first user.

[0153] Optionally, after the acquisition unit 2010 adjusts the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, the acquisition unit 2010 is further configured to acquire a second face image of a second user. The processing unit 2020 is further configured to determine second spatial coordinates of face feature points of the second user based on the second face image and adjust the orientation of the first side of the image displayed by the display device based on the second spatial coordinates.

[0154] Optionally, after the acquisition unit 2010 adjusts the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, the acquisition unit 2010 is further configured to acquire a third face image of the first user. The processing unit 2020 is configured to determine third spatial coordinates of face feature points of the first user based on the third face image, and when the distance between the first spatial coordinates and the third spatial coordinates is greater than or equal to a preset threshold, the processing unit 2020 is further configured to adjust the orientation of the first side of the image displayed by the display device based on the third spatial coordinates.

[0155] Optionally, the acquisition unit 2010 is further configured to acquire audio information before acquiring the first face image of the first user, and the audio information includes a voice instruction of the first user, and the voice instruction instructs to start and / or adjust a display device. The processing unit 2020 is further configured to determine the sound source position of the audio information and determine the position of the first user based on the sound source position.

[0156] Optionally, the audio information further includes a voice instruction of the second user, and the processing unit 2020 controls the display device to display the first image and the second image, and further configures the display device to adjust such that the first side of the first image faces the first user and the first side of the second image faces the second user. The image displayed by the display device includes the first image and / or the second image, the first side of the first image includes information to be transmitted to the first user, and the first side of the second image includes information to be transmitted to the second user.

[0157] Optionally, the processing unit 2020 is further configured to adjust the position of the image displayed by the display device based on the first spatial coordinates.

[0158] Optionally, the acquisition unit 2010 is further configured to acquire initial pose information of the display device before adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, and the initial pose information indicates the pose angle and position of the display device. The processing unit 2020 determines the rotation angle and rotation direction of the display device based on the first spatial coordinates and the initial pose information, and is further configured to adjust the orientation of the first side of the image displayed by the display device based on the rotation angle and rotation direction.

[0159] Optionally, the first spatial coordinates of the facial feature points of the first user indicate the three-dimensional position of the facial feature points of the first user inside the vehicle, and the vehicle includes a display device.

[0160] Optionally, the display device includes a holographic projection device, and the image includes a three-dimensional 3D image.

[0161] It should be understood that the division of the units within the device is only a logical functional division. During actual implementation, all or part of the units may be integrated into one physical entity or physically separated. Additionally, the units within the device may be implemented in the form of software called by a processor. For example, the device includes a processor, the processor is connected to a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any one of the aforementioned methods or the functions of each unit of the device. The processor is a general-purpose processor such as a CPU or a microprocessor. The memory is an internal memory of the device or an external memory of the device. Alternatively, the units within the device may be implemented in the form of a hardware circuit, and the functions of some or all of the units may be implemented by designing the hardware circuit. The hardware circuit may be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and some or all of the functions of the aforementioned units are implemented by designing the logical relationships between the elements within the circuit. As another example, in another implementation, the hardware circuit may be implemented by using a PLD. Taking an FPGA as an example, the hardware circuit may include a large number of logic gate circuits, and the connection relationships between the logic gate circuits are configured by using a configuration file to implement some or all of the aforementioned functions. All the units of the aforementioned device may be implemented in the form of software called by a processor, or all the units may be implemented in the form of a hardware circuit, or some of the units may be implemented in the form of software called by a processor, and the remaining part may be implemented in the form of a hardware circuit.

[0162] In an embodiment of the present application, the processor is a circuit having signal processing capabilities. In one implementation, the processor may be a circuit having instruction reading and execution capabilities, such as a CPU, a microprocessor, a GPU, or a DSP. In another embodiment, the processor may implement a specific function by using the logical relationships of hardware circuits. The logical relationships of the hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA. In a reconfigurable hardware circuit, the process by which the processor loads a configuration document to implement the hardware circuit configuration may be understood as the process by which the processor loads instructions for implementing some or all of the functions of the aforementioned units. Additionally, the processor may be a hardware circuit designed for artificial intelligence and may be understood as an ASIC, such as an NPU, a TPU, or a DPU.

[0163] It can be seen that each unit of the aforementioned device may be one or more processors (or processing circuits) configured to implement the above-described method, such as a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0164] In addition, all or some of the units of the above device may be integrated or may be implemented independently. In one implementation, the unit is integrated and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor configured to implement any one of the above-described methods or to implement the functions of each unit of the device. The types of the at least one processor may be different and may include, for example, a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0165] In a specific implementation process, the operations executed by the acquisition unit 2010 and the processing unit 2020 may be executed by the same processor, may be executed by different processors, or may be executed separately by a plurality of processors, for example. In one example, one or more processors may be connected to one or more sensors within the sensing system 120 of FIG. 2, and may acquire and process a user's face image from the one or more sensors. In another example, one or more processors may be further connected to one or more display devices within the display device 130, and may control the orientation of the image displayed by the display device. For example, in a specific implementation process, one or more processors may be arranged in a processor within an in-vehicle head unit, or may be a processor arranged in another in-vehicle terminal. For example, in a specific implementation process, the device 2000 may be a chip arranged in an in-vehicle head unit or another in-vehicle terminal. For example, in a specific implementation process, the device 2000 may be the computing platform 150 shown in FIG. 2 arranged within a vehicle.

[0166] FIG. 11 is a block diagram of an apparatus for adjusting a display device according to an embodiment of the present application. The apparatus 2100 for adjusting the display device shown in FIG. 11 may include a processor 2110, a transceiver 2120, and a memory 2130. The processor 2110, the transceiver 2120, and the memory 2130 are connected via an internal connection path. The memory 2130 is configured to store instructions. The processor 2110 is configured to execute the instructions stored in the memory 2130, and as a result, the transceiver 2120 receives / transmits some parameters. Optionally, the memory 2130 may be coupled to the processor 2110 via an interface, or may be integrated with the processor 2110.

[0167] The transceiver 2120 may include a transceiver device such as an input / output interface, but is not limited thereto, and it should be noted that communication between the device 2100 and another device or a communication network is implemented.

[0168] The processor 2110 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, and is configured to execute related programs to implement a method for adjusting a display device in an embodiment of the method of the present application. The processor 2110 may be an integrated circuit chip and has signal processing capabilities. In a specific implementation process, the steps of the method for adjusting the display device in the present application may be completed by using the integrated logic circuit of the hardware in the processor 2110 or by using instructions in the form of software. Alternatively, the processor 2110 may be a general-purpose processor, DSP, ASIC, FPGA, or another programmable logic device, discrete gate, or transistor logic device, or discrete hardware component. The processor 2110 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly executed and achieved by using a hardware decoding processor, or may be executed and achieved by using a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 2130. The processor 2110 reads the information in the memory 2130 and, in combination with the hardware of the processor 2110, executes a method for adjusting a display device in an embodiment of the method of the present application.

[0169] The memory 2130 may be a read only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM).

[0170] The transceiver 2120 implements communication between the device 2100 and another device or a communication network using a transceiver device, for example, but not limited to these. For example, by using the transceiver 2120, a user's face image and / or voice information can be obtained.

[0171] Embodiments of the present application further provide a vehicle. The vehicle may include the above-described device 2000 or the above-described device 2100 and the above-described display device. The display device may be a holographic projection device, and the displayed image may be a three-dimensional image. Alternatively, the display device may be an in-vehicle display, and the displayed image may be a digital human. Alternatively, the display device may be another display device. This is not particularly limited in the present application.

[0172] Embodiments of the present application further provide a computer program product. The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer can execute a method.

[0173] Embodiments of the present application further provide a computer-readable storage medium. The computer-readable storage medium stores program code. When the computer program code is executed on a computer, the computer can execute any one of the methods in FIGS. 4, 7, and 9.

[0174] Embodiments of the present application further provide a chip including at least one processor and a memory. The at least one processor is coupled to the memory, reads and executes instructions in the memory, and is configured to execute any one of the methods in FIGS. 4, 7, and 9.

[0175] In this application, each aspect, embodiment, and feature is presented with respect to a system that includes a plurality of devices, components, modules, etc. It should be recognized and understood that each system may include another device, component, module, etc., and / or may not include all of the devices, components, modules, etc. described with reference to the accompanying drawings. In addition, combinations of these solutions may be used.

[0176] In addition, in the embodiments of this application, terms such as "for example" and "such as" are used to give examples, illustrations, or explanations. Any embodiment or design scheme described as an "example" in this application should not be described as being more preferable or having more advantages than another embodiment or design scheme. Specifically, the term "example" is used to present a specific concept.

[0177] In the embodiments of this application, the terms "associated (corresponding, related)" and "corresponding (corresponding)" may sometimes be mixed. It should be noted that when the difference between the two is not emphasized, the meanings represented by the two are the same.

[0178] The network architectures and service scenarios described in the embodiments of this application are intended to more clearly explain the technical solutions in the embodiments of this application and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will recognize that due to the development of network architectures and the emergence of new service scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0179] The term "one embodiment" or "some embodiments" described in this specification and the like means that one or more embodiments of this application include specific features, structures or characteristics described in combination with the embodiments. Therefore, descriptions such as "in an embodiment", "in some embodiments", "in some other embodiments" and "in other embodiments" that appear in different parts of this specification do not necessarily mean references to the same embodiment. Instead, unless otherwise emphasized, they mean "one or more, but not all" of the embodiments. The terms "include", "comprise", "have" and their variants all mean "include but are not limited to" unless otherwise emphasized.

[0180] In this application, "at least one" refers to one or more, and "a plurality of" refers to two or more. The term "and / or" describes the relationship for describing related objects and indicates that three relationships can exist. For example, A and / or B may represent the following cases, that is, only A exists, both A and B exist, and only B exists, where A and B may be singular or plural. The character " / " usually indicates the "or" relationship between related objects. "At least one of the following items (pieces)" or a similar expression refers to any combination of these items, including any combination of a single item (piece) or multiple items (pieces). For example, at least one item (piece) among a, b, and c indicates any one of a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c may be singular or plural.

[0181] Those skilled in the art can recognize that, in combination with the examples described in the embodiments disclosed herein, the units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is executed by hardware or by software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but the implementation should not be considered to exceed the scope of this application.

[0182] As will be apparent to those skilled in the art, for the detailed operation processes of the above systems, devices and units, for the sake of convenience and brevity of description, reference is made to the corresponding processes in the embodiments of the above methods, and the details will not be described again here.

[0183] In some embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the described embodiments of the device are merely examples. For example, the division into units is only a logical function division, and in actual implementation, other divisions may be used. For example, a plurality of units or components may be combined and integrated into another system, and some features may be ignored or not executed. In addition, the mutual connection, direct connection or communication connection shown or discussed may be implemented through some interfaces. The indirect connection or communication connection between devices or units may be implemented in electrical, mechanical or other forms.

[0184] The units described as individual components may or may not be physically separate, and the parts shown as units may or may not be physical units. They may be arranged in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected based on the actual requirements for achieving the objectives of the embodiment solution.

[0185] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, each of the functional units may exist physically independently, or two or more functional units may be integrated into one processing unit.

[0186] When these functions are implemented in the form of software functional units and sold or used as independent products, these functions may be stored in a computer-readable storage medium. Based on such an understanding, all or part of the technical solutions of the present application may be implemented in the form of software products. The software product includes several instructions stored in a storage medium for instructing a computer device (which may be a personal computer, a server or a network device) to execute all or part of the steps of the method described in the embodiment. The above storage medium includes any medium that can store program codes, such as a USB flash drive, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk.

[0187] The foregoing description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any changes or substitutions that can be easily understood by those skilled in the art within the technical scope disclosed in the present application shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for adjusting a display device, comprising: obtaining a first face image of a first user; determining first spatial coordinates of face feature points of the first user based on the first face image; adjusting, based on the first spatial coordinates, an orientation of a first side of an image displayed by the display device, wherein the first side of the image includes information to be transmitted to the first user; A method comprising the above.

2. After adjusting, based on the first spatial coordinates, the orientation of the first side of the image displayed by the display device, the method further comprises: obtaining a second face image of a second user; determining second spatial coordinates of face feature points of the second user based on the second face image; adjusting, based on the second spatial coordinates, the orientation of the first side of the image displayed by the display device; The method according to claim 1, further comprising the above.

3. After adjusting, based on the first spatial coordinates, the orientation of the first side of the image displayed by the display device, the method further comprises: obtaining a third face image of the first user; determining third spatial coordinates of the face feature points of the first user based on the third face image; when a distance between the first spatial coordinates and the third spatial coordinates is greater than or equal to a preset threshold, adjusting, based on the third spatial coordinates, the orientation of the first side of the image displayed by the display device; The method according to claim 1, further comprising the above.

4. Before obtaining the first face image of the first user, the method further comprises: obtaining audio information, wherein the audio information includes a voice instruction of the first user, and the voice instruction instructs to start and / or adjust the display device; determining a sound source position of the audio information; determining a position of the first user based on the sound source position; The method according to any one of claims 1 to 3, further comprising the above.

5. The audio information further includes a voice instruction of a second user, and the method further comprises: controlling the display device to display a first image and a second image; Adjusting the display device such that the first side of the first image faces the first user and the first side of the second image faces the second user, wherein the images displayed by the display device include the first image and / or the second image, the first side of the first image includes information to be transmitted to the first user, and the first side of the second image includes information to be transmitted to the second user. The method according to claim 4, further comprising.

6. The method includes Adjusting the position of the image displayed by the display device based on the first spatial coordinates. The method according to any one of claims 1 to 5, further comprising.

7. Before adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates, the method further includes Obtaining initial pose information of the display device, wherein the initial pose information indicates the pose angle and position of the display device. The step of adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates is Determining the rotation angle and rotation direction of the display device based on the first spatial coordinates and the initial pose information; and Adjusting the orientation of the first side of the image displayed by the display device based on the rotation angle and the rotation direction. The method according to any one of claims 1 to 6, comprising.

8. The first spatial coordinates of the facial feature points of the first user indicate the three-dimensional position of the facial feature points of the first user in the vehicle, and the vehicle includes the display device. The method according to any one of claims 1 to 7.

9. The display device includes a holographic projection device, and the image includes a three-dimensional 3D image. The method according to any one of claims 1 to 8.

10. An apparatus for adjusting a display device, comprising An acquisition unit configured to acquire a first facial image of a first user. Based on the first face image, a processing unit configured to determine first spatial coordinates of facial feature points of the first user and adjust an orientation of a first side of an image displayed by the display device based on the first spatial coordinates, wherein the first side of the image includes information to be transmitted to the first user. Device. **Claim 11** The acquisition unit is further configured to acquire a second face image of a second user after adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates. The processing unit is further configured to determine second spatial coordinates of facial feature points of the second user based on the second face image and adjust the orientation of the first side of the image displayed by the display device based on the second spatial coordinates. The device according to claim 10. **Claim 12** The acquisition unit is further configured to acquire a third face image of the first user after adjusting the orientation of the first side of the image displayed by the display device based on the first spatial coordinates. The processing unit is further configured to determine third spatial coordinates of the facial feature points of the first user based on the third face image, and when a distance between the first spatial coordinates and the third spatial coordinates is equal to or greater than a preset threshold, adjust the orientation of the first side of the image displayed by the display device based on the third spatial coordinates. The device according to claim 10. **Claim 13** The acquisition unit is further configured to acquire audio information before acquiring the first face image of the first user, wherein the audio information includes a voice instruction of the first user, and the voice instruction instructs to start and / or adjust the display device. The processing unit is further configured to determine a sound source position of the audio information and determine a position of the first user based on the sound source position. The device according to any one of claims 10 to 12. **Claim 14** The audio information further includes a voice instruction of a second user, and the processing unit controls the display device to display a first image and a second image. The display device is further configured to adjust the display device such that the first side of the first image faces the first user and the first side of the second image faces the second user, the image displayed by the display device includes the first image and / or the second image, the first side of the first image includes information to be transmitted to the first user, and the first side of the second image includes information to be transmitted to the second user. The apparatus according to claim 13.

15. The processing unit is further configured to adjust the position of the image displayed by the display device based on the first spatial coordinates. The apparatus according to any one of claims 10 to 14.

16. The first spatial coordinates of the facial feature points of the first user indicate the three-dimensional position of the facial feature points of the first user in the vehicle, and the vehicle includes the display device. The apparatus according to any one of claims 10 to 15.

17. The acquisition unit is further configured to acquire initial pose information of the display device based on the first spatial coordinates before adjusting the orientation of the first side of the image displayed by the display device, and the initial pose information indicates the pose angle and position of the display device. The processing unit is further configured to determine the rotation angle and rotation direction of the display device based on the first spatial coordinates and the initial pose information, and to adjust the orientation of the first side of the image displayed by the display device based on the rotation angle and the rotation direction. The apparatus according to any one of claims 10 to 16.

18. The display device includes a holographic projection device, and the image includes a three-dimensional 3D image. The apparatus according to any one of claims 10 to 17.

19. An apparatus for adjusting a display device, comprising: a memory configured to store a computer program; a processor configured to execute the computer program stored in the memory to execute the method according to any one of claims 1 to 9. An apparatus comprising the above.

20. A vehicle comprising the apparatus according to any one of claims 10 to 18 and a display device.

21. A computer-readable storage medium storing a computer program, which when executed by a computer, enables the method according to any one of claims 1 to 9 to be executed. **Claim 22** A chip comprising a processor and a data interface, wherein the processor reads instructions stored in a memory via the data interface and executes the method according to any one of claims 1 to 9.

Images