Gaze-adaptive automobile interaction part light-emitting adjustment method, computer device and storage medium

Through the gaze-adaptive lighting adjustment method of the automotive interactive component, the line of sight characteristics are used to adjust the lighting of the automotive interactive component, which solves the problem of insufficient light in the cabin and improves the operation convenience and safety of the driver.

CN119018048AActive Publication Date: 2024-11-26GAC HONDA AUTOMOBILE CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411158207.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-26
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

During driving, due to insufficient light in the car compartment, it is difficult for the personnel on the car to accurately identify and operate the car interactive components, resulting in inconvenient operation and safety hazards.

Method used

The luminescence adjustment method of the car interactive component adaptive gaze is adopted. By obtaining the image of the person on the car, extracting the line of sight characteristics, determining the gaze area, and adjusting the luminescence parameters of the target car interactive component according to the gaze area, ensuring that the personnel on the car are in a suitable light environment. operate.

Benefits of technology

Real-time tracking of the sight of people on the vehicle and luminous adjustment of target car interactive components is realized, reducing the risk of wrong operations and distractions caused by insufficient light, and improving the accuracy and safety of operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119018048A_ABST
    Figure CN119018048A_ABST
Patent Text Reader

Abstract

The invention discloses a gaze-adaptive automobile interaction part light-emitting adjustment method, a computer device and a storage medium, which can realize real-time tracking of the sight of a person on an automobile and control a target automobile interaction part to carry out light-emitting adjustment for response. According to the invention, the vehicle-mounted personnel can trigger the light-emitting adjustment of the target vehicle interaction part through the action of watching the target vehicle interaction part, so that the light-emitting parameters of the target vehicle interaction part can meet the visual requirements of the vehicle-mounted personnel, and the vehicle-mounted personnel can observe and operate the target vehicle interaction part in a proper light environment. Therefore, personnel on the automobile can be guaranteed to accurately and efficiently control the automobile interaction part, the danger caused by distraction due to wrong operation of the automobile interaction part or repeated operation of the automobile interaction part is reduced, and the traffic safety is guaranteed. The invention is widely applied to the technical field of automobiles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a gaze-adaptive automobile interactive component luminescence adjustment method, a computer device and a storage medium. Background Art

[0002] When driving a car, it is usually necessary to use the car interactive components to control the functions of the car functional components. Due to the airtightness of the space inside the car cabin, the light inside the car cabin is often insufficient, which makes it difficult for the passengers to identify and accurately touch the car interactive components. Although the passengers can improve the lighting conditions in the cabin space by manually adjusting the lighting parameters of the car interactive components or turning on the lighting equipment inside the car, this requires the passengers to be distracted to operate, which is inconvenient for the passengers to operate and even faces certain dangers. Summary of the invention

[0003] In view of the technical problems in current automobile technology such as automobile interactive components being difficult to identify and use, the purpose of the present invention is to provide a gaze-adaptive automobile interactive component luminescence adjustment method, a computer device and a storage medium.

[0004] In one aspect, an embodiment of the present invention includes a gaze-adaptive method for adjusting the luminescence of an automobile interactive component, the gaze-adaptive method for adjusting the luminescence of an automobile interactive component comprising the following steps:

[0005] Obtain images of people in the vehicle;

[0006] Extracting features from the image of the person in the vehicle to obtain sight line features;

[0007] determining a gaze area according to the sight line characteristics;

[0008] determining a target automobile interactive component according to the gaze area;

[0009] The target automobile interactive component is subjected to luminescence adjustment.

[0010] Furthermore, the extracting features of the image of the person on the vehicle to obtain sight line features includes:

[0011] Running the first feature extraction network and the second feature extraction network;

[0012] Inputting the image of the person on the vehicle into the first feature extraction network for processing, and obtaining background-related features output by the first feature extraction network;

[0013] Inputting the image of the person on the vehicle into the second feature extraction network for processing, and obtaining the sight-related features output by the first feature extraction network;

[0014] The sight line feature is determined according to the background related feature and the sight line related feature.

[0015] Further, determining the sight line feature according to the background related feature and the sight line related feature includes:

[0016] Eliminating the sight-line related features from the image of the person on the vehicle to obtain an image without sight-line related features;

[0017] Obtaining the background feature by intersecting the background-related feature and the line-of-sight-removed feature image;

[0018] The background feature is eliminated from the image of the person in the vehicle to obtain the sight line feature.

[0019] Furthermore, the gaze-adaptive automobile interactive component luminescence adjustment method further includes:

[0020] Acquire a first training image and a second training image; the first training image corresponds to a marked sight feature, and the second training image corresponds to a marked background feature;

[0021] Alternatingly performing multiple rounds of first adversarial training processes and second adversarial training processes on the first feature extraction network and the second feature extraction network;

[0022] In the first adversarial training process, the first training image is input into the first feature extraction network for processing, a first feature output by the first feature extraction network is obtained, the first feature is eliminated from the first training image to obtain an image without the first feature, the image without the first feature is input into the second feature extraction network for processing, a first loss function value is determined according to an output result of the second feature extraction network and the marked line of sight feature, and parameters of the first feature extraction network and / or the second feature extraction network are adjusted according to the first loss function value;

[0023] In the second adversarial training process, the second training image is input into the second feature extraction network for processing, the second feature output by the second feature extraction network is obtained, the second feature is eliminated from the second training image, and an image without the second feature is obtained. The image without the second feature is input into the first feature extraction network for processing, and a second loss function value is determined based on the output result of the first feature extraction network and the marked background feature, and parameters of the first feature extraction network and / or the second feature extraction network are adjusted based on the second loss function value.

[0024] Further, determining the gaze area according to the sight line feature includes:

[0025] Running the third feature extraction network and the fourth feature extraction network;

[0026] Inputting the sight line feature into the third feature extraction network for processing, and obtaining a left eye sight line vector output by the third feature extraction network;

[0027] Inputting the sight line feature into the fourth feature extraction network for processing, and obtaining the right eye sight line vector output by the third feature extraction network;

[0028] An intersection area is determined according to the left eye sight line vector and the right eye sight line vector, and the intersection area is used as the gaze area.

[0029] Further, determining the gaze area according to the sight line feature includes:

[0030] Run the fifth feature extraction network;

[0031] Inputting the sight line feature into the fifth feature extraction network for processing, and obtaining a monocular sight line vector output by the fifth feature extraction network;

[0032] The spatial area through which the monocular sight line vector passes is taken as the gaze area.

[0033] Further, determining the target automobile interactive component according to the gaze area includes:

[0034] Determining candidate vehicle interaction components according to the gaze area; the candidate vehicle interaction components are vehicle interaction components located in the gaze area;

[0035] When the maintaining existence time of the gaze area is greater than a time threshold, the candidate automobile interaction component is determined as the target automobile interaction component.

[0036] Furthermore, the step of adjusting the light emission of the target automobile interactive component includes:

[0037] When the target vehicle interactive component is a self-luminous component, controlling the target vehicle interactive component to adjust brightness;

[0038] When the target vehicle interactive component is a non-self-luminous component, a lighting component is determined according to the target vehicle interactive component, and the lighting component is controlled to adjust the brightness; wherein the target vehicle interactive component is located within the lighting range of the lighting component.

[0039] On the other hand, an embodiment of the present invention also includes a computer device, including a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the gaze-adaptive automobile interactive component lighting adjustment method in the embodiment.

[0040] On the other hand, an embodiment of the present invention further includes a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the gaze-adaptive automobile interactive component lighting adjustment method in the embodiment.

[0041] The beneficial effect of the present invention is that through the gaze-adaptive automobile interactive component lighting adjustment method in the embodiment, it is possible to achieve real-time tracking of the line of sight of the occupants of the vehicle, and control the target automobile interactive component to perform lighting adjustment in response, so that the occupants of the vehicle can trigger the lighting adjustment of the target automobile interactive component by gazing at the target automobile interactive component, so that the lighting parameters of the target automobile interactive component can meet the visual requirements of the occupants of the vehicle, so that the occupants of the vehicle can observe and operate the target automobile interactive component in a suitable lighting environment, thereby facilitating the occupants of the vehicle to accurately and efficiently control the automobile interactive components, reducing the danger of distraction caused by incorrect operation of the automobile interactive components or repeated operation of the automobile interactive components, and ensuring traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the structure of a vehicle system to which the gaze-adaptive automobile interactive component luminescence adjustment method can be applied in the embodiment;

[0043] Figure 2 A schematic diagram of the steps of the gaze-adaptive automobile interactive component luminescence adjustment method in an embodiment;

[0044] Figure 3 Schematic diagram of the structure of the first feature extraction network and the second feature extraction network in the embodiment;

[0045] Figure 4 Schematic diagram of the principle of the first adversarial training process and the second adversarial training process in the embodiment;

[0046] Figure 5 Schematic diagram of the principle of the first feature extraction network and the second feature extraction network in the embodiment;

[0047] Figure 6 Schematic diagram of the structure of the third feature extraction network and the fourth feature extraction network in the embodiment. DETAILED DESCRIPTION

[0048] In this embodiment, the gaze-adaptive automobile interactive component luminescence adjustment method can be applied to Figure 1 In the vehicle system shown.

[0049] Reference Figure 1The vehicle system to which the gaze-adaptive vehicle interactive component luminescence adjustment method can be applied includes a control module, a camera, a vehicle interactive component, and a lighting component. Among them, the control module is a component with functions such as data acquisition, data processing, data output and control, and can be specifically an electronic control unit (ECU); the camera shoots the interior space of the cockpit, thereby capturing information such as facial expressions of people in the car; the vehicle interactive component is a component with interactive functions on the car, wherein the interactive function can be two-way human-machine interaction, or a one-way instruction or information transmission from a person to a machine or from a machine to a person, and can specifically include a central control screen that realizes an intelligent two-way interactive interface, a head-up display system (HUD) that electronically displays information, an instrument or an ambient light, a touch button that electronically receives control instructions, a knob or a gear lever that has a mechanical structure for receiving control instructions, and a handle or a pedal that has a mechanical structure for transmitting control actions.

[0050] Reference Figure 1 , the car interaction components may be self-luminous or non-self-luminous. For example, car interaction components such as the central control screen, head-up display system, instrument or ambient light are luminous when working, and therefore belong to the self-luminous car interaction components in this embodiment; touch buttons, knobs, gear levers, handles or pedals and other car interaction components are not luminous when working, and therefore belong to the non-self-luminous car interaction components in this embodiment. The luminous parameters (including brightness, color, flashing frequency, etc.) of the self-luminous car interaction components are adjustable, and the control module can send control instructions to the self-luminous car interaction components to adjust the luminous parameters of the self-luminous car interaction components; non-self-luminous car interaction components do not emit light themselves, refer to Figure 1 , non-self-luminous automobile interactive components can be equipped with lighting components to illuminate the non-self-luminous automobile interactive components. The lighting components can specifically be indicator lights or light strips installed on the automobile interactive components, or components such as lighting lamps installed on the automobile roof and other locations. The lighting components installed on the automobile interactive components can directly illuminate the automobile interactive components when they emit light, and the lighting lamps have a certain lighting range, and the corresponding automobile interactive components are located within the lighting range; the control module can send control instructions to the lighting components equipped with the non-self-luminous automobile interactive components, thereby adjusting the light-emitting parameters of the lighting components to achieve the adjustment of the light-emitting parameters of the non-self-luminous automobile interactive components.

[0051] In this embodiment, refer to Figure 2 The method for adjusting the light emission of an automobile interactive component based on gaze adaptation includes the following steps:

[0052] S1. Obtaining images of people in the vehicle;

[0053] S2. Extract features from the image of the person in the car to obtain sight features;

[0054] S3. Determine the gaze area according to the sight line characteristics;

[0055] S4. Determine the target vehicle interactive component according to the gaze area;

[0056] S5. Adjust the light emission of the target automobile interactive component.

[0057] In this embodiment, each step of the gaze-adaptive automobile interactive component luminescence adjustment method including steps S1-S5 can be executed by the control module. When the control module needs to obtain certain data or perform certain operations when executing the gaze-adaptive automobile interactive component luminescence adjustment method, the control module can send instructions to the corresponding automobile component, thereby calling the automobile component to collect data or perform operations.

[0058] In step S1, the control module calls the camera to capture an image of the person on the vehicle. The image of the person on the vehicle can be an image of the head of the person on the vehicle such as the driver or the passenger captured by the camera. The image of the person on the vehicle can be one or more still images or a dynamic video. The camera is generally used to capture the person on the vehicle who has a higher control over the vehicle, such as the driver. The camera sends the captured image of the person on the vehicle to the control module.

[0059] In this embodiment, a still image captured by the driver is used as an example of the image of the person in the vehicle for description.

[0060] In step S2, the control module can run an artificial intelligence model to extract features from the image of the person on the vehicle to obtain sight features. The sight features represent feature information contained in the image of the person on the vehicle that can reflect the direction of the sight of the eyes of the person on the vehicle (driver).

[0061] In step S3, since the sight line feature obtained in step S2 can determine the sight line direction of the driver, the spatial position where the driver's eyes are looking when the image of the person in the car is taken, that is, the looking area, can be determined based on the sight line feature.

[0062] In step S4, the spatial coordinates of each vehicle interaction component on the vehicle can be recorded in advance by means of calibration experiments, and the positional relationship between each vehicle interaction component and the gaze area determined in step S3 can be determined based on the spatial coordinates. In this embodiment, if the spatial coordinates of a vehicle interaction component are within the gaze area, then when executing step S4, the vehicle interaction component is determined as the target vehicle interaction component, that is, the vehicle interaction component to be luminous.

[0063] In step S5, the target vehicle interactive component determined in step S4 is illuminated and adjusted so that the illumination of the target vehicle interactive component can meet the visual needs of the occupant (driver) and respond to the gaze action of the occupant (driver).

[0064] In this embodiment, steps S1-S5 are executed in a dynamic cycle, that is, each time after the execution of steps S1-S5, the execution of the next step S1-S5 begins. When the performance of the control module is sufficient, it takes a short time to execute steps S1-S5, so the execution of steps S1-S5 can achieve real-time tracking of the line of sight of the occupants, and control the target vehicle interactive component to adjust the light to respond, so that the occupants can trigger the adjustment of the light of the target vehicle interactive component by watching the action of the target vehicle interactive component, so that the light parameters of the target vehicle interactive component can meet the visual requirements of the occupants, so that the occupants can observe and operate the target vehicle interactive component in a suitable light environment, which is conducive to ensuring that the occupants can accurately and efficiently control the vehicle interactive component, reduce the danger of distraction caused by incorrect operation of the vehicle interactive component or repeated operation of the vehicle interactive component, and ensure traffic safety.

[0065] In this embodiment, when the control module executes step S2, it can run the first feature extraction network and the second feature extraction network to extract features from the image of the person on the vehicle to obtain the sight line features. Specifically, a convolutional neural network can be used as the first feature extraction network and the second feature extraction network. The first feature extraction network and the second feature extraction network both have Figure 3 The structure shown.

[0066] In this embodiment, the control module can locate the positions of both eyes in the image of the person on the vehicle, take the center of the line connecting the positions of both eyes as the position of the center of the eyebrows, take the position of the center of the eyebrows as the origin, take the line connecting the positions of both eyes as the first coordinate axis, take the straight line located in the plane where the image of the person on the vehicle is located, passing through the origin and perpendicular to the first coordinate axis as the second coordinate axis, and take the straight line perpendicular to the plane where the image of the person on the vehicle is located and passing through the origin as the third coordinate axis, thereby establishing a unified coordinate system, in which the coordinates of the feature points in the image or the spatial positions in the car cabin are represented.

[0067] In this embodiment, the following steps may be performed to train the first feature extraction network and the second feature extraction network:

[0068] P1. Obtain a first training image and a second training image;

[0069] P2. Alternately perform multiple rounds of the first adversarial training process and the second adversarial training process on the first feature extraction network and the second feature extraction network.

[0070] The first training image corresponds to a labeled sight feature, and the second training image corresponds to a labeled background feature;

[0071] In step P1, multiple training images can be obtained. The content of each training image is a facial image of a person in a car cabin, including the person's eye image information. Among them, a part of the training images are first training images, each of which has a corresponding marked sight feature, which can be obtained by manual marking or the like, and the marked sight feature represents the sight feature such as the sight direction corresponding to the eye image information of the person in the first training image; another part of the training images are second training images, each of which has a corresponding marked background feature, which can be obtained by manual marking or the like, and the marked background feature represents the features in the second training image other than the sight feature, which can specifically include the background objects in the car cabin, the shooting light, the skin texture of the person, the expression of the person, and the like.

[0072] In this embodiment, when executing step P1, multiple training images can be obtained, each training image has a corresponding marked line of sight feature and a marked background feature, and each training image can be used as both a first training image and a second training image; when executing step P2, the training image used to mark the line of sight feature is the first training image, and the training image used to mark the background feature is the second training image.

[0073] In step P2, multiple rounds of the first adversarial training process and the second adversarial training process are alternately performed on the first feature extraction network and the second feature extraction network, for example, a round of the first adversarial training process is first performed, followed by a round of the second adversarial training process, then a round of the first adversarial training process, then a round of the second adversarial training process, etc., until the total number of rounds reaches a threshold and other stop conditions are met. The total number of rounds of the first adversarial training process and the total number of rounds of the second adversarial training process may be equal or unequal.

[0074] In this embodiment, a first round of adversarial training process and a subsequent second round of adversarial training process in step P2 are taken as examples for description.

[0075] The principle of the first adversarial training process is as follows Figure 4 As shown in part (a) of . When executing step P2, a first training image and its corresponding marked sight line feature are obtained, the first training image is input into the first feature extraction network for processing, and the first feature output by the first feature extraction network is obtained. The first feature is a feature extracted from the first training image and related to the background part of the first training image.

[0076] Reference Figure 4In part (a), after obtaining the first feature, the first feature is eliminated from the first training image to obtain the first feature-removed image. Specifically, the first feature can be subtracted from the first training image to obtain the first feature-removed image, that is, the first feature-removed image lacks the first feature relative to the first training image.

[0077] Reference Figure 4 In part (a), after obtaining the first feature-removed image, the first feature-removed image is input into the second feature extraction network for processing to obtain the output result of the second feature extraction network. The output result of the second feature extraction network is the features related to the sight line portion of the first training image extracted from the first feature-removed image.

[0078] Reference Figure 4 In part (a), the first loss function value is calculated using a loss function such as L2 according to the output result of the second feature extraction network and the marked line of sight feature. Then, according to the first loss function value, at least one of the first feature extraction network and the second feature extraction network is adjusted by a back propagation algorithm or the like, thereby completing the parameter update of the first feature extraction network and / or the second feature extraction network in this round of the first adversarial training process.

[0079] The principle of the second adversarial training process is as follows Figure 4 When executing step P2, a second training image and its corresponding labeled background features are obtained, and the second training image is input into the second feature extraction network for processing to obtain the second feature output by the second feature extraction network. The second feature is a feature extracted from the second training image and related to the sight line portion of the second training image.

[0080] Reference Figure 4 In part (b), after obtaining the second feature, the second feature is eliminated from the second training image to obtain an image without the second feature. Specifically, the second feature can be subtracted from the second training image to obtain an image without the second feature, that is, the image without the second feature lacks the second feature relative to the second training image.

[0081] Reference Figure 4 In part (b), after obtaining the image without the second feature, the image without the second feature is input into the first feature extraction network for processing to obtain the output result of the first feature extraction network. The output result of the first feature extraction network is the features related to the background part of the second training image extracted from the image without the second feature.

[0082] Reference Figure 4In part (b), a second loss function value is calculated using a loss function such as L2 based on the output result of the first feature extraction network and the marked background features. Then, according to the second loss function value, parameters of at least one of the first feature extraction network and the second feature extraction network are adjusted through a back propagation algorithm, etc., thereby completing the parameter update of the first feature extraction network and / or the second feature extraction network in this round of the first adversarial training process.

[0083] After the second adversarial training process is completed, if the stopping condition is not met, the next round of the first adversarial training process can be performed, and a pair of new first training images and their corresponding labeled sight features can be used for training; after the next round of the first adversarial training process is completed, if the stopping condition is not met, the next round of the second adversarial training process can be performed, and a pair of new second training images and their corresponding labeled background features can be used for training...

[0084] In this embodiment, the principle of executing steps P1-P2 is that: in the first adversarial training process, the first feature extraction network is used to process the first training image to obtain a first feature-removed image, and the performance of the first feature extraction network is used to eliminate at least a part of the features related to the background part, and then the second feature extraction network is used to extract features related to the line of sight part from the first feature-removed image, that is, in the first adversarial training process, the first feature extraction network is equivalent to a generator, which can generate a first feature-removed image for the second feature extraction network to train the line of sight feature extraction performance, and the second feature extraction network is equivalent to a discriminator, which can extract features related to the line of sight part from the first feature-removed image; in the second adversarial training process, the second feature extraction network is used to process the second training image to obtain a second feature-removed image. For example, the performance of the second feature extraction network is used to eliminate at least a part of the features related to the line of sight part, and then the first feature extraction network is used to extract features related to the background part from the second feature-removed image, that is, in the second adversarial training process, the second feature extraction network is equivalent to a generator, which can generate a first feature-removed image for the second feature extraction network to train the line of sight feature extraction performance, and the first feature extraction network is equivalent to a discriminator, which can extract features related to the background part from the second feature-removed image; through the alternating execution of the first adversarial training process and the second adversarial training process, the generative adversarial training of the first feature extraction network and the second feature extraction network can be achieved, thereby training the first feature extraction network with background feature extraction performance and the second feature extraction network with line of sight feature extraction performance.

[0085] In this embodiment, steps P1-P2 may be performed before step S2. After the first feature extraction network and the second feature extraction network are obtained by training through steps P1-P2, step S2 may be performed.

[0086] In this embodiment, when the control module executes step S2, that is, extracting features from the image of the person in the vehicle to obtain the sight line features, the control module may specifically execute the following steps:

[0087] S201. Run the first feature extraction network and the second feature extraction network;

[0088] S202. Inputting the image of the person in the vehicle into the first feature extraction network for processing, and obtaining the background-related features output by the first feature extraction network;

[0089] S203. Inputting the image of the person on the vehicle into the second feature extraction network for processing, and obtaining the sight-related features output by the first feature extraction network;

[0090] S204. Determine sight line features according to background related features and sight line related features.

[0091] The principle of steps S201-S204 is as follows Figure 5 shown.

[0092] In step S201, the control module may run the first feature extraction network and the second feature extraction network trained in steps P1-P2.

[0093] In step S202, the control module inputs the vehicle person image img obtained in step S1 into the first feature extraction network for processing, and uses the background feature extraction performance of the first feature extraction network to extract the background-related feature feature in the vehicle person image img. background-related .

[0094] In step S203, the control module inputs the image of the person on the vehicle img obtained in step S1 into the second feature extraction network for processing, and uses the sight feature extraction performance of the second feature extraction network to extract the sight-related feature feature in the image of the person on the vehicle img. sight-related .

[0095] In step S204, the control module obtains the background-related features in step S202. background-related And the sight-related feature obtained in step S203 sight-related , determine the sight feature sight .

[0096] In this embodiment, when the control module executes step S204, that is, the step of determining the sight line feature according to the background related features and the sight line related features, the control module may specifically execute the following steps:

[0097] S20401. Eliminate sight-related features from the image of the person on the vehicle to obtain an image without sight-related features;

[0098] S20402. Obtain the background features by intersecting the background-related features and the image without sight-related features;

[0099] S20403. Eliminate background features from the image of the person in the vehicle to obtain sight features.

[0100] In step S20401, all features contained in the image of the person on the vehicle img are taken as a complete set, and the sight-related features feature are eliminated from the image of the person on the vehicle img sight-related After that, the obtained image without line-of-sight related features is equivalent to

[0101] In step S20402, the background related features are background-related And the image without line of sight related features Find the intersection to get the background feature feature background ,Right now

[0102] In step S20403, all features contained in the image of the person on the vehicle img are taken as a complete set, and the background feature feature is eliminated from the image of the person on the vehicle img background Then, the sight feature feature is obtained sight ,Right now

[0103] In this embodiment, the principle of executing steps S20401-S20403 is that the image of the person on the vehicle img can be regarded as composed of background features feature background and sight feature sight The first feature extraction network has the performance of background feature extraction, but due to errors or deviations in training, the background-related features directly extracted by the first feature extraction network are background-related May contain only background features background Most of them also contain sight features sight Similarly, the second feature extraction network directly extracts the sight-related features sight-related May only contain line of sight features sight Most of them, including background features background A small part of the line of sight feature can be known according to the set principle sight satisfy By executing steps S20401-S20403, it can be calculated that So as to obtain the sight feature feature sight Therefore, executing steps S20401-S20403 can reduce the influence of practical factors such as errors or deviations in training, so that the final result obtained is close to or equal to the line of sight feature feature sight .

[0104] In this embodiment, the sight line feature sight Specifically, it can be expressed in the form of coordinates of multiple key points in the human eye in the image img of the person in the car.

[0105] In this embodiment, when the control module executes step S3, that is, the step of determining the gaze area according to the sight line feature, the control module may specifically execute the following steps:

[0106] S301A. Run the third feature extraction network and the fourth feature extraction network;

[0107] S302A. Inputting the sight feature into the third feature extraction network for processing, obtaining the left eye sight vector output by the third feature extraction network;

[0108] S303A. Input the sight feature into the fourth feature extraction network for processing, and obtain the right eye sight vector output by the third feature extraction network;

[0109] S304A. Determine an intersection area according to the left eye sight line vector and the right eye sight line vector, and use the intersection area as the gaze area.

[0110] Steps S301A-S304A are the first execution mode of step S3.

[0111] In this embodiment, when the control module executes step S301A, a convolutional neural network can be used as the third feature extraction network and the fourth feature extraction network. The third feature extraction network and the fourth feature extraction network both have Figure 6 The structure shown.

[0112] In this embodiment, the third feature extraction network and the fourth feature extraction network are both trained networks, wherein the third feature extraction network has the performance of extracting the left eye sight vector based on the sight feature (all or part of the left eye), and the fourth feature extraction network has the performance of extracting the right eye sight vector based on the sight feature (all or part of the right eye).

[0113] Specifically, in steps S302A-S303A, the left eye sight vector represents the sight feature feature sight The corresponding left eye sight direction of the person in the car, and the right eye sight vector represents the sight feature feature sightThe corresponding line of sight of the right eye of the person in the car.

[0114] In step S304A, the control module calculates the intersection point (extension line) based on the left eye sight vector and the right eye sight vector through a vector algorithm, and determines a space as the intersection area with a certain radius extending from the intersection point as the center. This intersection area is the gaze area of ​​the people in the car.

[0115] By executing steps S301A-S304A, it is possible to sight The determined binocular sight line vector is used to calculate the area that the person in the car is looking at.

[0116] In this embodiment, when the control module executes step S3, that is, the step of determining the gaze area according to the sight line feature, the control module may specifically execute the following steps:

[0117] S301B. Run the fifth feature extraction network;

[0118] S302B inputs the sight feature into the fifth feature extraction network for processing, and obtains the monocular sight vector output by the fifth feature extraction network;

[0119] S303B. The spatial area through which the monocular sight line vector passes is regarded as the fixation area.

[0120] Steps S301B-S304B are a second execution mode of step S3.

[0121] In step S301B, the fifth feature extraction network used may be one of the third feature extraction network and the fourth feature extraction network. sight They are input into the third feature extraction network and the fourth feature extraction network respectively. The third feature extraction network cannot extract the line of sight feature. sight The fourth feature extraction network can process the line of sight feature sight Then, in step S301B, the fourth feature extraction network is used as the fifth feature extraction network, and in step S302B, the right eye sight vector output by the fourth feature extraction network is used as the monocular sight vector.

[0122] In step S303B, a spatial region is determined with a certain radius extending from the point on the monocular sight line vector obtained in step S302B as the center. This spatial region is the gaze area of ​​the occupants of the vehicle.

[0123] By executing steps S301B-S303B, when the occupant has monocular vision impairment or both eyes in the image of the occupant are unclear, the gaze area can be determined based on the recognition of the monocular eye, thereby expanding the scope of application of the gaze-adaptive automobile interactive component luminescence adjustment method.

[0124] In this embodiment, when the control module executes step S4, that is, the step of determining the target vehicle interactive component according to the gaze area, the control module may specifically execute the following steps:

[0125] S401. Determine candidate automobile interactive components according to the gaze area;

[0126] S402: When the maintenance existence time of the gaze area is greater than the time threshold, the candidate automobile interactive component is determined as the target automobile interactive component.

[0127] In step S401, the control module may detect, in a unified coordinate system, which vehicle interaction components have position coordinates within the gaze area obtained in step S3, and determine those vehicle interaction components within the gaze area as candidate vehicle interaction components.

[0128] Since steps S1-S5 are executed in a dynamic loop, the maintenance time of the gaze area can be calculated. Specifically, the maintenance time of the gaze area refers to the length of time that the coordinates of the gaze area remain unchanged (or change less than a threshold). If it is detected that the maintenance time of the gaze area is less than the time threshold, then step S401 is re-executed; if it is detected that the maintenance time of the gaze area is greater than the time threshold, then step S402 is executed, and the candidate automobile interaction component obtained in step S401 is determined as the target automobile interaction component for executing step S5 for light adjustment.

[0129] In this embodiment, by executing steps S401-S402, the vehicle interactive components that are located in the gaze area of ​​the vehicle occupants and are gazed at for a long time by the vehicle occupants can be detected and determined as target vehicle interactive components, so that the light emission can be further adjusted, which can effectively respond to the gaze actions of the vehicle occupants and reduce the possibility of false triggering.

[0130] In this embodiment, when the control module executes step S5, that is, the step of adjusting the light emission of the target vehicle interactive component, the control module may specifically execute the following steps:

[0131] S501. When the target vehicle interactive component is a self-luminous component, the target vehicle interactive component is controlled to adjust the brightness;

[0132] S502. When the target vehicle interactive component is a non-self-luminous component, determine a lighting component according to the target vehicle interactive component, and control the lighting component to adjust brightness.

[0133] If the target vehicle interactive component determined by executing step S4 is a self-luminous component such as a central control screen or an ambient light, the control module sends a brightness adjustment instruction to the target vehicle interactive component, thereby controlling the target vehicle interactive component itself to adjust its brightness; if the target vehicle interactive component determined by executing step S4 is a non-self-luminous component such as a knob or a handle, the control module sends a brightness adjustment instruction to the lighting component that matches the non-target vehicle interactive component, thereby controlling the lighting component to adjust its brightness, and the effect of the brightness adjustment is projected onto the target vehicle interactive component.

[0134] Specifically, when executing steps S501-S502 and controlling the target automobile interactive component or lighting component to adjust the brightness, the brightness can be uniformly increased. For example, when adjusting the brightness, the target automobile interactive component or lighting component can be controlled to increase the brightness by 30% based on the original brightness. If the brightness exceeds the upper limit after the 30% increase, the brightness can be adjusted to the upper limit. Since steps S1-S5 are dynamically executed multiple times, the brightness adjustment performed by the last execution of steps S1-S5 can be canceled when the gaze area detected by the last execution of steps S1-S5 changes relative to the gaze area detected by the last execution of steps S1-S5, such as reducing the brightness of the target automobile interactive component or lighting component, so that the luminous brightness of the target automobile interactive component or lighting component is restored to the original level. In this way, when steps S1-S5 are dynamically executed multiple times, it can be achieved that when the people on the vehicle are looking at the target vehicle interactive components, the target vehicle interactive components themselves are either illuminated and have a higher luminous brightness, and when the people on the vehicle are not looking at the target vehicle interactive components, the luminous brightness of the target vehicle interactive components is reduced to the original level or no light is emitted. This can intelligently detect and respond to the gaze operations of the people on the vehicle, reduce the types and number of operations that the people on the vehicle need to perform, provide good lighting conditions for the people on the vehicle to use the target vehicle interactive components, improve the convenience of using the target vehicle interactive components, and ensure traffic safety.

[0135] A computer program for executing the gaze-adaptive luminescence adjustment method for automobile interactive components in the present embodiment can be written and written into a computer device or a storage medium. When the computer program is read out and executed, the gaze-adaptive luminescence adjustment method for automobile interactive components in the present embodiment is executed, thereby achieving the same technical effect as the gaze-adaptive luminescence adjustment method for automobile interactive components in the embodiment.

[0136] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0137] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0138] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0139] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0140] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0141] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0142] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A gaze-adaptive luminescence adjustment method for automobile interactive components, It is characterized in that The gaze-adaptive automobile interactive component luminescence adjustment method comprises: Obtain images of people in the vehicle; Extracting features from the image of the person in the vehicle to obtain sight line features; determining a gaze area according to the sight line characteristics; determining a target automobile interactive component according to the gaze area; The target automobile interactive component is subjected to luminescence adjustment.

2. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 1, It is characterized in that The extracting features of the image of the person on the vehicle to obtain sight features includes: Running the first feature extraction network and the second feature extraction network; Inputting the image of the person on the vehicle into the first feature extraction network for processing, and obtaining background-related features output by the first feature extraction network; Inputting the image of the person on the vehicle into the second feature extraction network for processing, and obtaining the sight-related features output by the first feature extraction network; The sight line feature is determined according to the background related feature and the sight line related feature.

3. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 2, It is characterized in that The determining the sight line feature according to the background related feature and the sight line related feature includes: Eliminating the sight-line related features from the image of the person on the vehicle to obtain an image without sight-line related features; Obtaining the background feature by intersecting the background-related feature and the line-of-sight-removed feature image; The background feature is eliminated from the image of the person in the vehicle to obtain the sight line feature.

4. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 2, It is characterized in that The gaze-adaptive automobile interactive component luminescence adjustment method further includes: Acquire a first training image and a second training image; the first training image corresponds to a marked sight feature, and the second training image corresponds to a marked background feature; Alternatingly performing multiple rounds of first adversarial training processes and second adversarial training processes on the first feature extraction network and the second feature extraction network; In the first adversarial training process, the first training image is input into the first feature extraction network for processing, a first feature output by the first feature extraction network is obtained, the first feature is eliminated from the first training image to obtain an image without the first feature, the image without the first feature is input into the second feature extraction network for processing, a first loss function value is determined according to an output result of the second feature extraction network and the marked line of sight feature, and parameters of the first feature extraction network and / or the second feature extraction network are adjusted according to the first loss function value; In the second adversarial training process, the second training image is input into the second feature extraction network for processing, the second feature output by the second feature extraction network is obtained, the second feature is eliminated from the second training image, and an image without the second feature is obtained. The image without the second feature is input into the first feature extraction network for processing, and a second loss function value is determined based on the output result of the first feature extraction network and the marked background feature, and parameters of the first feature extraction network and / or the second feature extraction network are adjusted based on the second loss function value.

5. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 1, It is characterized in that The determining the gaze area according to the sight line feature comprises: Running the third feature extraction network and the fourth feature extraction network; Inputting the sight line feature into the third feature extraction network for processing, and obtaining a left eye sight line vector output by the third feature extraction network; Inputting the sight line feature into the fourth feature extraction network for processing, and obtaining the right eye sight line vector output by the third feature extraction network; An intersection area is determined according to the left eye sight line vector and the right eye sight line vector, and the intersection area is used as the gaze area.

6. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 1, It is characterized in that The determining the gaze area according to the sight line feature comprises: Run the fifth feature extraction network; Inputting the sight line feature into the fifth feature extraction network for processing, and obtaining a monocular sight line vector output by the fifth feature extraction network; The spatial area through which the monocular sight line vector passes is taken as the gaze area.

7. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 1, It is characterized in that The step of determining a target automobile interactive component according to the gaze area comprises: Determining candidate vehicle interaction components according to the gaze area; the candidate vehicle interaction components are vehicle interaction components located in the gaze area; When the maintaining existence time of the gaze area is greater than a time threshold, the candidate automobile interaction component is determined as the target automobile interaction component.

8. The gaze-adaptive luminescence adjustment method for automobile interactive components according to claim 1, It is characterized in that The step of adjusting the light emission of the target automobile interactive component comprises: When the target vehicle interactive component is a self-luminous component, controlling the target vehicle interactive component to adjust brightness; When the target vehicle interactive component is a non-self-luminous component, a lighting component is determined according to the target vehicle interactive component, and the lighting component is controlled to adjust the brightness; wherein the target vehicle interactive component is located within the lighting range of the lighting component.

9. A computer device, It is characterized in that It comprises a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the gaze-adaptive automobile interactive component luminescence adjustment method as described in any one of claims 1-8.

10. A computer-readable storage medium having a program executable by a processor stored therein, It is characterized in that The program executable by the processor is used to execute the gaze-adaptive automobile interactive component lighting adjustment method described in any one of claims 1-8 when executed by the processor.

Citation Information

Patent Citations

  • Car lamp rotating with sight line of driver and control method

    CN106828285A

  • Human-vehicle interaction method and device based on human eye sight and storage medium

    CN112114671A

  • Method, device and equipment for adjusting lamplight in vehicle in night scene and storage medium

    CN118338494A

  • Visual line guidance sheet, visual line guidance system, vehicle lighting apparatus, and vehicle

    US20190291640A1