Screen control methods, vehicles and storage media

By acquiring facial images of people inside the vehicle and adjusting parameters, the position and orientation of the in-vehicle screen are automatically adjusted, solving the problem that a fixed in-vehicle screen cannot meet user needs and improving the user experience.

CN117389405BActive Publication Date: 2026-11-17GREAT WALL MOTOR CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210784150.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-05
Publication Date
2026-11-17
Estimated Expiration
2042-07-05

AI Technical Summary

Technical Problem

Fixed-location in-vehicle screens are unable to meet people's ever-increasing demands for a higher quality of life.

Method used

By acquiring facial images inside the vehicle, determining facial pose information, and generating adjustment parameters based on screen pose information, the pose of the in-vehicle screen is automatically adjusted to adapt to the viewing needs of different passengers.

Benefits of technology

It enables flexible adjustment of the in-vehicle screen, improves the user experience, and meets the viewing needs of different passengers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117389405B_ABST
    Figure CN117389405B_ABST
Patent Text Reader

Abstract

This application relates to the field of equipment control technology and provides a screen control method, a vehicle, and a storage medium. The method includes: acquiring an in-vehicle face image; determining the face pose information corresponding to the in-vehicle face image based on its image features; generating pose adjustment parameters for adjusting the pose of the in-vehicle screen based on the screen pose information and the face pose information; and adjusting the pose of the in-vehicle screen according to the pose adjustment parameters. This application can automatically adjust the pose of the in-vehicle screen based on the situation of people inside the vehicle, enabling timely and flexible adjustment of the in-vehicle screen's pose, thus improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of equipment control technology, and in particular relates to a screen control method, a vehicle, and a storage medium. Background Technology

[0002] With the diversification of vehicle functions, in-vehicle screens are becoming increasingly common. When parked, in-vehicle screens can play movies and TV shows for passengers, and while driving, they can provide navigation information to the driver.

[0003] In related technologies, in-vehicle screens are typically fixedly installed between the driver's and passenger's seats. However, with technological advancements and rising living standards, fixed-position in-vehicle screens are struggling to meet people's increasingly sophisticated demands for quality of life. Summary of the Invention

[0004] This application provides a screen control method, a vehicle, and a storage medium, aiming to solve the problem in the related art that vehicle screens with fixed locations cannot meet people's ever-increasing demands for quality of life.

[0005] In a first aspect, embodiments of this application provide a screen control method, the method comprising:

[0006] Acquire facial images inside the vehicle, and determine the facial pose information corresponding to the facial images inside the vehicle based on the image features of the facial images inside the vehicle;

[0007] Based on the screen pose information and face pose information of the vehicle screen, pose adjustment parameters are generated to adjust the pose of the vehicle screen.

[0008] The pose of the in-vehicle screen is adjusted according to the pose adjustment parameters.

[0009] In some embodiments, determining the facial pose information corresponding to the in-vehicle facial image based on image features includes:

[0010] Extract face region images from in-vehicle face images, and perform eye detection on the face region images to obtain eye detection results;

[0011] Based on the results of human eye detection, key points of the face region image are determined, and the face pose information of the face image inside the vehicle is determined based on the key points of the image.

[0012] In some embodiments, determining image key points of a face region image based on human eye detection results includes:

[0013] When the human eye detection result indicates that an eye has been detected, the detected eye location is determined as an image key point in the face region image;

[0014] When the human eye detection result indicates that no eye has been detected, the preset location points of the face region image are determined as image key points of the face region image, wherein the preset location points include the center point of the face region image.

[0015] In some embodiments, when the eye detection result indicates that an eye has been detected, the detected eye location is determined as an image key point of the face region image, including:

[0016] When the human eye detection result indicates that an eye has been detected, the location of the detected eye is determined as an image key point in the face region image;

[0017] When the human eye detection results indicate that two eyes have been detected, the midpoint between the two detected eye positions is determined as the image key point of the face region image.

[0018] In some embodiments, when multiple face region images exist in the in-vehicle face image, there are multiple face pose information; and based on the screen pose information of the in-vehicle screen and the face pose information, pose adjustment parameters for adjusting the pose of the in-vehicle screen are generated, including:

[0019] Based on multiple facial pose information, the target facial pose information is determined, and pose adjustment parameters are generated based on the target facial pose information and screen pose information.

[0020] In some embodiments, when a first number of face region images exist in the in-vehicle face image, the method further includes:

[0021] Collect in-vehicle audio information. If the in-vehicle audio information indicates the existence of a second number of users, and the second number is greater than the first number, extract the target audio for each user from the in-vehicle audio information.

[0022] Each target audio is input into a pre-trained audio localization model to obtain the audio location of the corresponding audio.

[0023] Based on the user positions indicated by each face region image and the obtained audio positions, the missed users and their positions are determined. The positions of the missed users are used to correct the pose adjustment parameters.

[0024] In some embodiments, acquiring an in-vehicle face image includes: acquiring an in-vehicle face image at a target location;

[0025] The target location includes at least one of the following: the driver's seat, other in-vehicle locations besides the driver's seat, and the target location corresponds to the content playback mode of the in-vehicle screen.

[0026] In some embodiments, the method further includes:

[0027] When a light spot area is detected on the vehicle screen, the light intensity of the windows on both sides is collected, and the position of the vehicle screen is adjusted according to the collected light intensity.

[0028] Secondly, embodiments of this application provide a screen control device, including:

[0029] The image acquisition unit is used to acquire in-vehicle face images and determine the face pose information corresponding to the in-vehicle face images based on the image features of the in-vehicle face images.

[0030] The parameter generation unit is used to generate pose adjustment parameters for adjusting the pose of the vehicle screen based on the screen pose information and face pose information of the vehicle screen.

[0031] The pose adjustment unit is used to adjust the pose of the vehicle screen according to the pose adjustment parameters.

[0032] In some embodiments, the image acquisition unit includes an eye detection module and an information determination module.

[0033] The human eye detection module is used to extract face region images from in-vehicle face images and to perform human eye detection on the face region images to obtain human eye detection results.

[0034] The information determination module is used to determine the key points of the face region image based on the human eye detection results, and to determine the face pose information of the face image inside the vehicle based on the key points.

[0035] In some embodiments, the information determination module is specifically used for:

[0036] When the human eye detection result indicates that an eye has been detected, the detected eye location is determined as an image key point in the face region image;

[0037] When the human eye detection result indicates that no eye has been detected, the preset location points of the face region image are determined as image key points of the face region image, wherein the preset location points include the center point of the face region image.

[0038] In some embodiments, in the information determination module, when the human eye detection result indicates that an eye has been detected, the detected eye location is determined as an image key point of the face region image, including:

[0039] When the human eye detection result indicates that an eye has been detected, the location of the detected eye is determined as an image key point in the face region image;

[0040] When the human eye detection results indicate that two eyes have been detected, the midpoint between the two detected eye positions is determined as the image key point of the face region image.

[0041] In some embodiments, when there are multiple face region images in the in-vehicle face image, there are multiple face pose information; and a parameter generation unit is specifically used to: determine the target face pose information based on the multiple face pose information, and generate pose adjustment parameters based on the target face pose information and the screen pose information.

[0042] In some embodiments, when a first number of facial region images exist in the in-vehicle facial image, the device further includes an audio acquisition unit, a location determination unit, and a parameter correction unit.

[0043] An audio acquisition unit is used to acquire in-vehicle audio information. When the in-vehicle audio information indicates the existence of a second number of users, and the second number is greater than the first number, the unit extracts the target audio for each user from the in-vehicle audio information.

[0044] The location determination unit is used to input each target audio into a pre-trained audio localization model to obtain the audio location of the corresponding audio.

[0045] The parameter correction unit is used to determine the missed users and their locations based on the user positions indicated by each face region image and the obtained audio positions. The locations of the missed users are used to correct the pose adjustment parameters.

[0046] In some embodiments, the image acquisition unit is specifically used to: acquire an image of a person's face inside the vehicle at a target location;

[0047] The target location includes at least one of the following: the driver's seat, other in-vehicle locations besides the driver's seat, and the target location corresponds to the content playback mode of the in-vehicle screen.

[0048] In some embodiments, the device further includes a screen adjustment unit for acquiring the illumination intensity of the side windows when a light spot area is detected on the vehicle screen, and adjusting the pose of the vehicle screen according to the acquired illumination intensity.

[0049] Thirdly, embodiments of this application provide a vehicle including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described screen control methods.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described screen control methods.

[0051] Fifthly, embodiments of this application provide a computer program product that, when run on a vehicle, causes the vehicle to execute any of the aforementioned screen control methods.

[0052] Compared with related technologies, the beneficial effects of the embodiments of this application are: it can automatically acquire in-vehicle facial images and automatically adjust the position and pose of the in-vehicle screen based on the in-vehicle personnel situation indicated by the in-vehicle facial images, which can realize timely and flexible adjustment of the position and pose of the in-vehicle screen and help improve the user experience.

[0053] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating a screen control method provided in an embodiment of this application;

[0056] Figure 2 A schematic diagram illustrating the relative positional relationship between a face and an in-vehicle screen, provided for an embodiment of this application;

[0057] Figure 3 This is a flowchart illustrating the process of determining the facial pose information corresponding to a face image inside a vehicle, provided in an embodiment of this application.

[0058] Figure 4(a) is a diagram showing the optimal positional relationship between an image key point and the vehicle screen plane provided in an embodiment of this application;

[0059] Figure 4(b) is a diagram showing the optimal positional relationship between multiple image key points and the vehicle screen plane provided in an embodiment of this application;

[0060] Figure 5 This is a flowchart illustrating the process for determining the location of a missed user, provided in an embodiment of this application.

[0061] Figure 6 This is a schematic diagram of the screen control device provided in the embodiments of this application;

[0062] Figure 7 This is a schematic diagram of the vehicle structure provided in the embodiments of this application. Detailed Implementation

[0063] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0064] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0065] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0066] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0067] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0068] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0069] To illustrate the technical solution of this application, the following embodiments will be used for explanation.

[0070] Example 1

[0071] Please see Figure 1 This application provides a screen control method, including:

[0072] Step 101: Obtain the face image inside the vehicle, and determine the face pose information corresponding to the face image inside the vehicle based on the image features of the face image inside the vehicle.

[0073] The aforementioned facial pose information typically indicates the displacement and rotation angle of the face. In practice, the rotation angle of the face usually refers to the rotation angle of the face plane in a reference coordinate system. This reference coordinate system can be the camera coordinate system or other coordinate systems. The camera coordinate system is typically the coordinate system of the camera that captured the in-vehicle facial image.

[0074] Here, the subject executing the above screen control method is usually a vehicle, and specifically, it can be a controller in the vehicle used to control the in-vehicle screen.

[0075] In practice, the aforementioned executing entity can acquire the in-vehicle facial images using cameras installed inside the vehicle. Subsequently, based on the image features carried by these in-vehicle facial images, the executing entity can obtain the facial pose information of one or more faces presented in the in-vehicle facial images.

[0076] As an example, the aforementioned execution entity can obtain the facial pose information corresponding to the in-vehicle facial image in the following way: The execution entity can input the in-vehicle facial image into a pre-trained facial pose prediction model to obtain the facial pose information of the face presented in the in-vehicle facial image. The aforementioned facial pose prediction model can be used to analyze the correspondence between facial images and the facial pose information of the faces presented in the facial images. Specifically, the facial pose prediction model can be a correspondence table generated based on statistical analysis of a large number of facial images, storing the correspondence between multiple facial images and facial pose information; or it can be a model obtained by training an initial model (such as a Convolutional Neural Network (CNN), ResNet, etc.) using machine learning methods based on training samples.

[0077] As another example, the aforementioned execution entity can also obtain the facial pose information corresponding to the in-vehicle facial image in the following way: First, the execution entity can detect facial region images from the in-vehicle facial image. These facial region images are typically the areas where faces are located in the in-vehicle facial image. In practice, an in-vehicle facial image can contain one or multiple facial region images. For example, when two faces are present in the in-vehicle facial image, two facial region images can be detected. Then, the execution entity can compare the detected facial region images with a pre-stored set of region images to find the region image whose facial pose is most similar to that of the facial region image, and determine the facial pose information corresponding to that region image as the facial pose information of that facial region image. It should be noted that when multiple facial region images exist in the in-vehicle facial image, multiple facial pose information can be obtained; that is, for each facial region image, one facial pose information can be obtained.

[0078] In practical applications, acquiring in-vehicle facial images can be performed when a preset trigger condition is met. This preset trigger condition can be a pre-defined condition used to activate screen adjustment functions. For example, the preset trigger condition could be receiving a user input instructing the user to activate the screen adjustment function. Alternatively, the preset trigger condition could be detecting that the in-vehicle screen has started playing content.

[0079] Step 102: Generate pose adjustment parameters for adjusting the pose of the vehicle screen based on the screen pose information and face pose information of the vehicle screen.

[0080] The aforementioned screen pose information typically indicates the displacement and rotation angle of the vehicle screen. In practice, the rotation angle of the vehicle screen usually refers to the rotation angle of the plane formed by the vehicle screen in the reference coordinate system.

[0081] Here, the aforementioned execution entity can use the aforementioned screen pose information and the aforementioned face pose information to calculate pose adjustment parameters for adjusting the pose of the vehicle screen.

[0082] Figure 2 This is a schematic diagram illustrating the relative positional relationship between a face and a vehicle screen, provided as an embodiment of this application. (Combined with...) Figure 2If the screen pose information and face pose information indicate that the relative pose of the face and the vehicle screen is as follows: the face is located 0.1 meters to the left of the vehicle screen and the angle between the face plane and the vehicle screen plane is 10 degrees, then the vehicle screen can move 0.1 meters to the left and then rotate 10 degrees to the right front. That is, at this time, the pose adjustment parameters can be ΔL = -0.1 and Δθ = 10, indicating a movement of 0.1 meters to the left and a rotation of 10 degrees to the right front.

[0083] In some optional implementations, when multiple face regions exist in the in-vehicle face image, there are multiple face pose information. In this case, step 102 above, which generates pose adjustment parameters for adjusting the pose of the in-vehicle screen based on the screen pose information and the face pose information, may include: determining the target face pose information based on the multiple face pose information, and generating pose adjustment parameters based on the target face pose information and the screen pose information.

[0084] Here, when multiple facial pose information is available, the aforementioned execution entity can first use these multiple facial pose information to obtain a target facial pose information. As an example, the facial pose information corresponding to the driver can be determined as the target facial pose information. As another example, the average of the facial pose information corresponding to at least some of the occupants can be determined as the aforementioned target facial pose information. For instance, if the facial pose information corresponding to the driver is (r1, t1) and the facial pose information corresponding to the passenger is (r2, t2), then the obtained target facial pose information can be (r, t), where r = (r1 + r2) ÷ 2, and t = (t1 + t2) ÷ 2. r is the rotation angle, and t is the displacement.

[0085] It should be noted that when multiple face region images exist, the face pose parameters corresponding to each face region image are used to jointly determine the target face pose information. Based on the target face pose information and the screen pose information, the pose adjustment parameters for adjusting the in-vehicle screen are determined. This takes into account the pose of each person in the vehicle, so that the adjusted in-vehicle screen meets the viewing needs of each person in the vehicle, which helps to further improve the user experience.

[0086] Step 103: Adjust the pose of the vehicle screen according to the pose adjustment parameters.

[0087] Here, the aforementioned execution entity can use the pose adjustment parameters to adjust the pose of the vehicle screen. In practice, the pose adjustment parameters can be used to generate commands to control the motor's movement. The motor's movement then drives the vehicle screen's movement, thereby adjusting the vehicle screen's pose.

[0088] In practical applications, after adjusting the pose of the vehicle screen, the aforementioned execution entity can further adjust the screen pose information based on the pose adjustment parameters. For example, if the current pose information of the vehicle screen is (R1, T1) and the pose adjustment parameters are (Δr, Δt), then the next pose information of the vehicle screen can be (R1+Δr, T1+Δt).

[0089] The method provided in this embodiment can automatically acquire in-vehicle facial images and automatically adjust the position and pose of the in-vehicle screen based on the in-vehicle occupant information indicated by the in-vehicle facial images. This enables timely and flexible adjustment of the in-vehicle screen's position and pose, which helps improve the user experience.

[0090] In some optional implementations of this embodiment, determining the facial pose information corresponding to the in-vehicle facial image based on the image features of the in-vehicle facial image may include the following steps 301 and 302.

[0091] Figure 3 This is a schematic diagram of a process for determining the facial pose information corresponding to a face image inside a vehicle, provided in an embodiment of this application.

[0092] Step 301: Extract the face region image from the face image inside the vehicle, and perform eye detection on the face region image to obtain the eye detection result.

[0093] The aforementioned face region image typically refers to the area containing the face in an in-vehicle face image. In practice, an in-vehicle face image can contain one or multiple face region images. For example, if two faces are present in an in-vehicle face image, two face region images can be detected.

[0094] The aforementioned eye detection results are used to indicate the number of eyes present in the face region image.

[0095] Here, the aforementioned execution entity can detect and extract face regions from in-vehicle face images. Then, eye detection can be performed on the face regions to obtain eye detection results.

[0096] In practice, the aforementioned execution entity can input a face region image into a pre-trained neural network model for detecting human eyes to obtain the human eye detection result for the face region image. Alternatively, the execution entity can compare the face region image with a pre-stored set of face images to obtain the face image most similar to the face region image, and use the number of eyes corresponding to that face image to generate the human eye detection result for that face region image.

[0097] Step 302: Based on the human eye detection results, determine the image key points of the face region image, and determine the face pose information of the face image inside the vehicle based on the image key points.

[0098] Here, for each facial region image in the in-vehicle facial image, the aforementioned execution entity can determine image key points on the facial region image based on the eye detection results. For example, if the eye detection results indicate that there are 0 eyes in the facial region image, meaning no eyes exist, the aforementioned execution entity can use any point on the facial region image as the aforementioned image key point. For instance, if the camera is installed on the left side of the vehicle and the in-vehicle screen is currently on the right side, the occupant may turn their head to look at the screen, causing the camera to only capture a side profile. In this case, when detecting the facial region image of the occupant, no eyes are detected. Since no eyes are detected, it is assumed that no eyes exist in the facial region image, and in this case, any point on the facial region image can be used as the aforementioned image key point.

[0099] After obtaining the key points of the image, the aforementioned execution entity can use the coordinates of the key points to calculate the facial pose information of the face region image.

[0100] In practice, since there is usually a transformation relationship between the coordinates of image keypoints in the face region image coordinate system and their coordinates in the camera coordinate system, after obtaining the coordinates of the image keypoints, their coordinates in the camera coordinate system can be obtained. When the reference coordinate system is the camera coordinate system, if the coordinates of the image keypoints in the camera coordinate system are known, and if there is only one face region in the in-vehicle face image, the face pose information of the in-vehicle face image can be directly calculated using the coordinates of the image keypoints in the camera coordinate system. For example, if the coordinates of the image keypoints in the camera coordinate system are A, the origin of the camera coordinate system is O, and the three coordinate axes of the camera coordinate system are the X-axis, Y-axis, and Z-axis, then the displacement in the calculated face pose information is the displacement between point A and point O, and the rotation angle in the face pose information is the angle between point A and each coordinate axis.

[0101] It should be noted that image key points can accurately describe the position of a face in a reference coordinate system. Furthermore, by performing eye detection on the face region image in the in-vehicle face image, image key points can be selected from the face region image based on the eye detection results, and then used to generate face pose information. Obtaining the face pose information of the corresponding face region image solely from face key points is computationally simple and helps improve data processing efficiency.

[0102] When the facial pose information of the in-vehicle face image is determined based on the image key points, the aforementioned execution entity can determine the pose adjustment parameters of the vehicle screen in the following manner: determine the target distance between the image key points and the center normal of the vehicle screen, and generate the pose adjustment parameters based on the target distance and the coordinates of the image key points. Here, the center normal is the normal to the vehicle screen plane at the center point of the vehicle screen.

[0103] Specifically, if the in-vehicle face image contains only one face region, the distance between the image key points of that face region and the central normal can be calculated and denoted as the target distance. If the target distance is 0, the image key points are located on the central normal, which is the optimal viewing position, and no adjustment to the in-vehicle screen is required. If the target distance is greater than zero, the target distance and the coordinates of the image key points need to be used to generate pose adjustment parameters to adjust the in-vehicle screen to a position where the central normal coincides with the image key points.

[0104] If the in-vehicle facial image contains multiple facial regions, the distance between the midpoint of each keypoint and the central normal can be calculated and denoted as the target distance. If the target distance is 0, the midpoint of each keypoint lies on the central normal, which is the optimal viewing position and no adjustment to the in-vehicle screen is needed. If the target distance is greater than zero, the target distance and the coordinates of the midpoints of the keypoints are used to generate pose adjustment parameters to align the in-vehicle screen with the central normal and the midpoints of the keypoints.

[0105] Figure 4(a) shows the optimal positional relationship between an image key point and the vehicle screen plane provided in an embodiment of this application. Figure 4(b) shows the optimal positional relationship between multiple image key points and the vehicle screen plane provided in an embodiment of this application.

[0106] When the in-vehicle face image indicates that there is only one user in the vehicle, there is only one image keypoint. Referring to Figure 4(a), the dashed line is the center normal of the in-vehicle screen plane, and image keypoint d1 is located on the center normal of the in-vehicle screen plane, which is the optimal screen viewing position. When the in-vehicle face image indicates that there are two users in the vehicle, there are two image keypoints. Referring to Figure 4(a), the dashed line is the center normal of the in-vehicle screen plane, and the midpoint between image keypoints d1 and d2 is located on the center normal of the in-vehicle screen plane, which is the optimal screen viewing position.

[0107] In some optional implementations, determining image key points of the face region image based on the human eye detection results may include: when the human eye detection results indicate that an eye has been detected, determining the detected eye position as an image key point of the face region image; when the human eye detection results indicate that no eye has been detected, determining a preset position point of the face region image as an image key point of the face region image.

[0108] The preset location point includes the center point of the face region image.

[0109] Here, for each facial region image in the in-vehicle facial image, if eyes are detected, the aforementioned execution entity can determine the eye position as an image key point of the facial region image. If eyes are not detected, the execution entity can use a preset location point as the image key point of the facial region image. When the preset location point is the center point of the facial region image, since the center point of the facial region image can more accurately describe the position of the face in the reference coordinate system, more accurate facial pose information can be obtained, thereby further improving the accuracy of adjusting the in-vehicle screen.

[0110] In some alternative implementations, the method of determining the detected eye location as an image key point of the face region image when the eye detection result indicates that an eye has been detected may include: determining the detected eye location as an image key point of the face region image when the eye detection result indicates that one eye has been detected; and determining the midpoint between the detected eye locations as an image key point of the face region image when the eye detection result indicates that two eyes have been detected.

[0111] Here, for each facial region image in the in-vehicle facial image, if only one eye is detected in the facial region image, the eye position can be used as the image key point. If two eyes are detected, the midpoint between the two eyes can be used as the image key point of the facial region image. Determining the image key points of the facial region image based on the eye position can obtain more accurate facial pose information and make the adjusted in-vehicle screen more suitable for user viewing, thus helping to further improve the user experience.

[0112] In some optional implementations of this embodiment, when there is a first number of facial region images in the in-vehicle facial image, the screen control method may further include the following steps 501 to 503.

[0113] Figure 5 This is a schematic diagram of a process for determining the location of a user who was missed during a detection process, provided in an embodiment of this application.

[0114] Step 501: Collect in-vehicle audio information. If the in-vehicle audio information indicates the existence of a second number of users and the second number is greater than the first number, extract the target audio for each user from the in-vehicle audio information.

[0115] The aforementioned in-vehicle audio information typically refers to audio from inside the vehicle. The aforementioned target audio is typically the user's audio segment from the in-vehicle audio information. A specific target audio segment can be assigned to each user.

[0116] Here, the aforementioned executing entity can use an audio acquisition device, such as a microphone, to acquire the aforementioned in-vehicle audio information.

[0117] After obtaining the aforementioned in-vehicle audio information, the executing entity can analyze the in-vehicle audio information to determine the number of users corresponding to the in-vehicle audio information.

[0118] As an example, the aforementioned entity can perform frequency analysis on the in-vehicle audio information to determine if there are multiple voice frequencies included in the in-vehicle audio information. If there are multiple frequencies, it means that the in-vehicle audio information corresponds to multiple users. For example, if the in-vehicle audio information includes three voice frequencies, it means that the in-vehicle audio information corresponds to three users.

[0119] Subsequently, if the in-vehicle audio information indicates the presence of a second number of users, and this second number is greater than the first number, the aforementioned execution entity can extract the target audio for each user from the in-vehicle audio information. For example, if the first number is 2 and the second number is 3, then the second number is greater than the first number, and the aforementioned execution entity can extract the target audio for each user from the in-vehicle audio information.

[0120] Step 502: Input each target audio into the pre-trained audio localization model to obtain the audio location of the corresponding audio.

[0121] The aforementioned audio localization model is typically used to represent the correspondence between audio and audio location. In practice, this audio localization model can be obtained by training an initial model (e.g., CNN, ResNet, etc.) using machine learning methods based on training samples.

[0122] Here, for each user's target audio, the aforementioned execution entity can input the target audio into the audio localization model to obtain the audio position corresponding to the target audio.

[0123] Step 503: Based on the user positions indicated by each face region image and the obtained audio positions, determine the missed users and their locations.

[0124] The location of the missed user is used to correct the pose adjustment parameters.

[0125] Among them, the users who were missed are usually those who were not detected by the facial images inside the vehicle.

[0126] For example, if there are two face region images, and the user positions indicated by these two face region images are L1 and L2 respectively, and there are three target audios, and the three audio positions corresponding to these three target audios are L1, L2 and L3, then there is a missed user and the position of the missed user is L3.

[0127] In practical applications, after obtaining the location of the missed user, the aforementioned execution entity can use this location to correct the pose adjustment parameters obtained in step 102. For example, the execution entity can calculate the offset between the first reference position and the second reference position, and then use this offset to adjust the pose adjustment parameters. Here, the first reference position is the average position of all users except the missed user, and the second reference position is the average position of all users including the missed user.

[0128] For example, if there are 3 users in the vehicle, including one missed user whose position is L3, and the other two users' positions are L1 and L2 respectively, then the aforementioned execution entity can calculate the average of L1 and L2 to obtain the first reference position X1, and calculate the average of L1, L2 and L3 to obtain the second reference position X2. Then, the offset displacement between the first reference position and the second reference position can be calculated as ΔX = X2 - X1. After that, the sum of the offset displacement and the pose adjustment parameter can be used as the adjusted pose adjustment parameter.

[0129] It should be noted that by using the location of missed users to correct the obtained pose adjustment parameters, the pose of the in-vehicle screen can be automatically adjusted based on the location of all occupants in the vehicle. This makes the adjusted in-vehicle screen more suitable for the viewing needs of each occupant, which helps to further improve the user experience.

[0130] In some optional implementations of this embodiment, step 101 above, which involves obtaining the in-vehicle face image, may include: obtaining the in-vehicle face image at the target location.

[0131] The target location includes at least one of the following: the driver's seat, or other in-vehicle locations besides the driver's seat, and the target location corresponds to the content playback mode of the in-vehicle screen. The content playback mode can include a navigation playback mode and an entertainment content playback mode. In navigation mode, the in-vehicle screen displays navigation information. In entertainment content playback mode, the in-vehicle screen displays entertainment content that the user desires to play.

[0132] In practice, during navigation mode, the target location is usually the driver's seat. During entertainment playback mode, the target location can be the driver's seat or other locations inside the vehicle besides the driver's seat.

[0133] It's worth noting that acquiring facial images of different areas inside the vehicle, depending on the content playback mode of the in-vehicle screen, allows for targeted screen adjustments, improving adjustment efficiency. Furthermore, in navigation mode, acquiring only facial images from the driver's seat requires less data processing compared to processing images of the entire vehicle, thus improving data processing efficiency. The driver also doesn't need to turn their head to look at the navigation, enhancing driving safety.

[0134] In some optional implementations of this embodiment, the screen control method may further include the following steps: when a light spot area is detected on the vehicle screen, the light intensity of the two side windows is collected, and the pose of the vehicle screen is adjusted according to the collected light intensity.

[0135] In practice, the aforementioned execution entity can capture images of the vehicle's screen using a camera and perform spot detection on those images to determine whether spot areas exist on the screen. For example, the vehicle screen image can be input into a pre-trained spot detection model to obtain spot detection information, which indicates the presence of spot areas on the screen. This spot detection model can characterize the correspondence between the image and the spot detection information. In practice, this spot detection model can be a model obtained by training an initial model (e.g., CNN, ResNet, etc.) using machine learning methods based on training samples.

[0136] When there are glare areas on the vehicle screen, the aforementioned actuator can collect the light intensity of the windows on both sides using a light intensity sensor. Then, the collected light intensity can be used to adjust the position and orientation of the vehicle screen. For example, if the light intensity at the left window is greater than the light intensity at the right window, the vehicle screen can be moved to the right and / or turned to the right, thus avoiding the side with higher light intensity as much as possible.

[0137] It should be noted that when strong external light shines on a car screen, glare areas often appear, which can affect the user's viewing experience. Adjusting the car screen according to the light intensity on both sides of the vehicle, so that the screen avoids the side with higher light intensity, can make the adjusted screen more suitable for the viewing needs of the occupants, thus further improving the user experience.

[0138] In some alternative implementations, the screen control method described above may further include the following steps: receiving a screen adjustment request from a user that includes target pose information, and adjusting the vehicle screen to match the target pose information based on the screen adjustment request.

[0139] The aforementioned screen adjustment request is typically used to request information for adjusting the in-vehicle screen.

[0140] Here, the aforementioned executing entity can adjust the vehicle screen upon receiving a screen adjustment request, thereby adjusting the vehicle screen to the position desired by the user. This allows for flexible adjustment of the vehicle screen and helps to further improve the user experience.

[0141] Example 2

[0142] Corresponding to the screen control method in the above embodiments, Figure 6 A structural block diagram of a screen control device 600 provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown. (Refer to...) Figure 6 The device includes an image acquisition unit 601, a parameter generation unit 602, and a pose adjustment unit 603.

[0143] The image acquisition unit 601 is used to acquire face images inside the vehicle and determine the face pose information corresponding to the face images inside the vehicle based on the image features of the face images inside the vehicle.

[0144] The parameter generation unit 602 is used to generate pose adjustment parameters for adjusting the pose of the vehicle screen based on the screen pose information and face pose information of the vehicle screen.

[0145] The pose adjustment unit 603 is used to adjust the pose of the vehicle screen according to the pose adjustment parameters.

[0146] In some embodiments, the image acquisition unit 601 includes an eye detection module and an information determination module. The eye detection module is used to extract a face region image from an in-vehicle face image and to perform eye detection on the face region image to obtain an eye detection result. The information determination module is used to determine image key points of the face region image based on the eye detection result, and to determine the face pose information of the in-vehicle face image based on the image key points.

[0147] In some embodiments, the information determination module is specifically configured to: when the human eye detection result indicates that an eye has been detected, determine the detected eye position as an image key point of the face region image; when the human eye detection result indicates that no eye has been detected, determine a preset position point of the face region image as an image key point of the face region image, wherein the preset position point includes the center point of the face region image.

[0148] In some embodiments, in the information determination module, when the eye detection result indicates that an eye has been detected, determining the detected eye position as an image key point of the face region image includes: when the eye detection result indicates that one eye has been detected, determining the detected eye position as an image key point of the face region image; and when the eye detection result indicates that two eyes have been detected, determining the midpoint between the detected eye positions as an image key point of the face region image.

[0149] In some embodiments, when multiple face region images exist in the in-vehicle face image, there are multiple face pose information. The parameter generation unit 602 is specifically used to: determine the target face pose information based on the multiple face pose information, and generate pose adjustment parameters based on the target face pose information and the screen pose information.

[0150] In some embodiments, when a first number of face region images exist in the in-vehicle face images, the device further includes an audio acquisition unit, a location determination unit, and a parameter correction unit. The audio acquisition unit is used to acquire in-vehicle audio information, and when the in-vehicle audio information indicates the presence of a second number of users, and the second number is greater than the first number, it extracts target audio for each user from the in-vehicle audio information. The location determination unit is used to input each target audio into a pre-trained audio localization model to obtain the audio location of the corresponding audio. The parameter correction unit is used to determine missed users and their locations based on the user locations indicated by each face region image and the obtained audio locations, wherein the locations of the missed users are used to correct the pose adjustment parameters.

[0151] In some embodiments, the image acquisition unit 601 is specifically used to: acquire an in-vehicle face image at a target location. The target location includes at least one of the following: the driver's seat, other in-vehicle locations besides the driver's seat, and the target location corresponds to the content playback mode of the in-vehicle screen.

[0152] In some embodiments, the device further includes a screen adjustment unit for acquiring the illumination intensity of the side windows when a light spot area is detected on the vehicle screen, and adjusting the pose of the vehicle screen according to the acquired illumination intensity.

[0153] The device provided in this embodiment can automatically acquire in-vehicle facial images and automatically adjust the position and pose of the in-vehicle screen based on the in-vehicle occupant information indicated by the in-vehicle facial images. This enables timely and flexible adjustment of the in-vehicle screen's position and pose, which helps improve the user experience.

[0154] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0155] Example 3

[0156] Figure 7 This is a structural schematic diagram of a vehicle 700 provided in one embodiment of this application. Figure 7 As shown, the vehicle 700 of this embodiment includes: at least one processor 701 ( Figure 7 The diagram shows only one processor, memory 702, and a computer program 703 stored in memory 702 and executable on at least one processor 701, such as a screen control program. When processor 701 executes computer program 703, it implements the steps in any of the above-described method embodiments. When processor 701 executes computer program 703, it implements the steps in the embodiments of the above-described screen control methods. When processor 701 executes computer program 703, it implements the functions of each module / unit in the above-described device embodiments, such as... Figure 6 The functions of the image acquisition unit 601 to the pose adjustment unit 603 are shown.

[0157] For example, computer program 703 can be divided into one or more modules / units, one or more of which are stored in memory 702 and executed by processor 701 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 703 in vehicle 700. For example, computer program 703 can be divided into an image acquisition unit, a parameter generation unit, and a pose adjustment unit. The specific functions of each unit have been described in the above embodiments and will not be repeated here.

[0158] Vehicle 700 may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of vehicle 700 and does not constitute a limitation on vehicle 700. It may include more or fewer components than shown, or combine certain components, or different components. For example, a vehicle may also include input / output devices, network access devices, buses, etc.

[0159] The processor 701 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0160] The memory 702 can be an internal storage unit of the vehicle 700, such as a hard drive or RAM. The memory 702 can also be an external storage device of the vehicle 700, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 702 can include both internal and external storage units of the vehicle 700. The memory 702 is used to store computer programs and other programs and data required by the vehicle. The memory 702 can also be used to temporarily store data that has been output or will be output.

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0162] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0163] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] In the embodiments provided in this application, it should be understood that the disclosed devices / vehicles and methods can be implemented in other ways. For example, the device / vehicle embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If an integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer-readable storage medium can be non-volatile or volatile. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of a computer-readable storage medium may be appropriately added to or subtracted from the contents as required by the legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable storage medium may not include electrical carrier signals and telecommunication signals.

[0168] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A screen control method, characterized in that, The method includes: Acquire a face image inside the vehicle, and determine the face pose information corresponding to the face image inside the vehicle based on the image features of the face image inside the vehicle; Based on the screen pose information of the vehicle screen and the face pose information, pose adjustment parameters are generated for adjusting the pose of the vehicle screen. The pose of the vehicle screen is adjusted according to the pose adjustment parameters. When a first number of facial region images exist in the in-vehicle facial image, the method further includes: Collect in-vehicle audio information, and when the in-vehicle audio information indicates the existence of a second number of users, and the second number is greater than the first number, extract target audio for each user from the in-vehicle audio information; Each target audio is input into a pre-trained audio localization model to obtain the audio location of the corresponding audio. Based on the user positions indicated by each face region image and the obtained audio positions, the missed users and their positions are determined, wherein the positions of the missed users are used to correct the pose adjustment parameters.

2. The screen control method according to claim 1, characterized in that, The step of determining the facial pose information corresponding to the facial image inside the vehicle based on the image features of the facial image inside the vehicle includes: The face region image is extracted from the face image inside the vehicle, and the face region image is subjected to eye detection to obtain the eye detection result; Based on the human eye detection results, image key points of the face region image are determined, and face pose information of the in-vehicle face image is determined based on the image key points.

3. The screen control method according to claim 2, characterized in that, The step of determining the image key points of the face region image based on the human eye detection results includes: When the human eye detection result indicates that an eye has been detected, the detected eye position is determined as an image key point of the face region image; When the human eye detection result indicates that no eye is detected, a preset location point of the face region image is determined as the image key point of the face region image, wherein the preset location point includes the center point of the face region image.

4. The screen control method according to claim 3, characterized in that, When the human eye detection result indicates that an eye has been detected, determining the detected eye position as an image key point of the face region image includes: When the human eye detection result indicates that an eye has been detected, the location of the detected eye is determined as an image key point of the face region image; When the human eye detection result indicates that two eyes have been detected, the midpoint of the two detected eye positions is determined as the image key point of the face region image.

5. The screen control method according to claim 1, characterized in that, When multiple face region images exist in the in-vehicle face image, there are multiple face pose information; and the generation of pose adjustment parameters for adjusting the pose of the in-vehicle screen based on the screen pose information of the in-vehicle screen and the face pose information includes: Based on multiple facial pose information, target facial pose information is determined, and pose adjustment parameters are generated based on the target facial pose information and the screen pose information.

6. The screen control method according to claim 1, characterized in that, The step of acquiring the in-vehicle facial image includes: acquiring the in-vehicle facial image at the target location; The target location includes at least one of the following: the driver's seat, other in-vehicle locations besides the driver's seat, and the target location corresponds to the content playback mode of the in-vehicle screen.

7. The screen control method according to any one of claims 1-6, characterized in that, The method further includes: When a light spot area is detected on the vehicle screen, the light intensity of the windows on both sides is collected, and the position of the vehicle screen is adjusted according to the collected light intensity.

8. A vehicle comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the screen control method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the screen control method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Method of controlling terminal screen to rotate and terminal

    CN103092344A

  • Motion adjustment method for aligning screen with face

    CN111126142A