Sunshade operation triggering method and device

Abstract

The application provides a sunshade operation triggering method and device, relates to the technical field of artificial intelligence, and can accurately capture the sunshade intention of a user and automatically trigger a sunshade board to perform sunshade. The method comprises the following steps: tracking the position of the hand of the user, and determining a target image containing a hand image; based on the target image, detecting whether the hand posture of the user meets a triggering condition; the triggering condition comprises that multiple fingers are in a close state and / or the hand of the user is in a hovering state; in the case where it is determined that the hand posture meets the triggering condition, a sunshade operation is triggered.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and device for triggering sunshade operation. Background Technology

[0002] In daily life, people may encounter situations where sunlight is glaring and scorching, creating a need for sunshade. For example, while driving, strong sunlight often impairs visibility, and people wish for sunshades to automatically deploy in suitable locations. The automatic deployment of sunshades first requires accurately identifying the user's sunshade needs. However, current solutions struggle to accurately recognize these intentions, resulting in a high false alarm rate. How to more accurately capture the user's sunshade needs and trigger the sunshade device to perform the shading operation has become one of the technical problems that those skilled in the art need to solve. Summary of the Invention

[0003] This application provides a method and device for triggering sunshade operation, which can accurately capture the user's sunshade intention and automatically trigger the sunshade device to achieve intelligent sunshade.

[0004] Firstly, this application provides a method for triggering a sunshade operation, applied to an electronic device. The method includes: tracking the user's hand position and determining a target image containing the hand image; based on the target image, detecting whether the user's hand posture meets triggering conditions; the triggering conditions include at least one of multiple fingers being in a closed state, the user's hand being in a suspended state, and fingers being in a straight state; and triggering a sunshade operation if the hand posture meets the triggering conditions. This method can effectively distinguish whether the user's hand gesture is a sunshade gesture, thereby accurately capturing the user's sunshade needs.

[0005] In one possible implementation, detecting whether the user's hand gesture meets the triggering conditions includes: detecting whether multiple fingers of the user are in a closed position; and / or, detecting whether the user's fingers are in a straight position. In practical applications, the probability of a user making a gesture with multiple fingers closed or fingers straight without intending to shade from the sun is very small. At least by detecting these two discriminant factors, it is possible to accurately reflect whether the user has the intention to shade from the sun in many scenarios.

[0006] In one possible implementation, detecting whether a user's multiple fingers are in a closed-off state includes: detecting the angle between two adjacent fingers; and determining whether the user's multiple fingers are in a closed-off state based on the angle. In practical applications, different users may make slightly different hand gestures when shading their hands. In most cases, users do not intentionally spread their fingers when shading their hands but tend to keep them closed. Therefore, detecting the angle between adjacent fingers of a user's hand can reflect whether the user's hand is closed, thus providing a reference for recognizing the intention to shade the hands.

[0007] In one possible implementation, detecting the angle between two adjacent fingers includes: detecting multiple key points on the user's hand, the multiple key points corresponding to multiple bone nodes on the hand; connecting two adjacent key points among the multiple key points corresponding to the same finger to determine the line segments corresponding to each finger; and determining the angle between the line segments corresponding to two adjacent fingers as the angle between the two adjacent fingers.

[0008] In one possible implementation, detecting whether a user's multiple fingers are in a closed-hand state includes: performing edge detection on the target image to determine the edges of the user's hand and determining a first length corresponding to the edges; and determining whether the user's multiple fingers are in a closed-hand state based at least on the first length. Another significant difference between a hand gesture that tends to close while shading the sun and a gesture with fingers spread lies in the change in the length of the hand's edges. By determining and comparing the edge lengths, it is possible to identify whether the user's hand is in a closed-hand state from another perspective.

[0009] In one possible implementation, before determining whether a user's multiple fingers are in a closed state based at least on a first length, the method further includes: determining a first reference length corresponding to the edge of the user's hand when the user's multiple fingers are in a closed state; determining whether a user's multiple fingers are in a closed state based at least on the first length includes: comparing the first length and the first reference length, and determining whether the user's multiple fingers are in a closed state based on the comparison result.

[0010] In one possible implementation, before determining whether a user's multiple fingers are in a closed state based at least on a first length, the method further includes: determining a second reference length corresponding to the edge of the user's hand when the user's multiple fingers are in an open state; determining whether the user's multiple fingers are in a closed state based at least on the first length, including: comparing the first length with the second reference length; comparing the first length with the first reference length if the first length is less than the second reference length; and determining whether the user's multiple fingers are in a closed state if the comparison result determined based on the first length and the first reference length falls within a predetermined first threshold range.

[0011] In one possible implementation, determining the first length corresponding to the edge includes: taking two adjacent fingers as a group, and determining the third length corresponding to the edge formed by two adjacent fingers in each group; determining whether the user's multiple fingers are in a closed state based at least on the first length includes: identifying whether the corresponding two adjacent fingers are in a closed state based at least on the third length corresponding to each group, so as to determine whether the user's multiple fingers are in a closed state.

[0012] In one possible implementation, determining the first length corresponding to the edge includes: determining the second length corresponding to the edge of the entire hand; determining whether multiple fingers of the user are in a closed state based at least on the first length includes: determining whether multiple fingers of the user are in a closed state based at least on the second length.

[0013] In one possible implementation, detecting whether a user's finger is straight includes: determining the finger's vertebrae from the fingertip to the base as a first to a fourth node, where the first node is the fingertip node and the fourth node is the base node; determining a first straight line based on the third and fourth nodes; determining a second straight line based on the first and second nodes; and / or determining a third straight line based on the second and third nodes; determining that the finger is straight if the angle between the first straight line and the second and / or third straight lines falls within a predetermined second threshold range. Finger straightness is another characteristic of most users when making a sunshade gesture. Recognizing whether the finger is straight can improve the accuracy of sunshade intention recognition and prevent misjudgments.

[0014] In one possible implementation, detecting whether a user's fingers are in a straightened state further includes: determining the bone node corresponding to the palm as the fifth node; determining the first plane where the palm is located based on at least two fourth nodes and the fifth node corresponding to at least two fingers; and determining that the fingers are in a straightened state if the angle between at least one of the first, second, and third straight lines and the first plane falls within a predetermined third threshold range.

[0015] In one possible implementation, detecting whether the user's hand gesture meets the triggering conditions includes detecting whether the user's hand is in a hovering state. Hand hovering is another characteristic of most users when making a sunshade gesture. Waving the hand is generally not for sunshade, so recognizing whether the hand is hovering can further improve the accuracy of sunshade intention recognition.

[0016] In one possible implementation, detecting whether a user's hand is in a hovering state includes: determining that multiple fingers of the user are in a closed state, detecting multiple key points of the user's hand, the multiple key points corresponding to multiple bone nodes of the hand; connecting two adjacent key points according to the hand bone connection information to determine at least one line segment; detecting the displacement of at least one line segment within a predetermined time period; and determining whether the user's hand is in a hovering state based on the displacement.

[0017] In one possible implementation, detecting the displacement of at least one line segment within a predetermined time period includes: determining multiple frames of images within the predetermined time period; detecting a first line segment in each frame of the multiple frames, and determining a first pixel and a second pixel; wherein the first line segment is one of at least one line segment; the first pixel and the second pixel are the two endpoints of the first line segment; obtaining the first coordinate value of the first pixel and the second coordinate value of the second pixel in each frame of images; calculating a first average of multiple first coordinate values, and using the pixel with the first average as the coordinate value as a first reference pixel; and calculating a second average of multiple second coordinate values, and using the pixel with the second average as the coordinate value as a second reference pixel; determining a reference line segment using the first reference pixel and the second reference pixel as the two endpoints; calculating the distance between the first line segment and the reference line segment in each frame of images, and using the obtained multiple distances as the displacement of at least one line segment within the predetermined time period; and determining whether the user's hand is in a hovering state based on the displacement, including: determining that the user's hand is in a hovering state if all multiple distances fall within a predetermined fourth threshold range.

[0018] In one possible implementation, the triggering condition further includes: the projection area formed by the user's hand contains skin; detecting whether the user's hand gesture meets the triggering condition also includes: detecting whether the projection area formed by the user's hand contains skin. When users shade themselves from the sun, the purpose is mostly to prevent light from shining into their eyes or skin. Since the eyes are surrounded by skin, whether for the purpose of protecting the eyes or blocking the skin, the projection area of ​​the shading gesture will most likely contain skin. Therefore, based on this discrimination factor, it can further assist in capturing the user's intention to shade themselves from the sun.

[0019] In one possible implementation, detecting whether the projection area formed by the user's hand contains skin includes: determining a first plane where the palm is located based on the user's current hand posture; determining the intersection point of the normal of the first plane and the user's body surface; detecting whether a projection is generated within a preset distance range around the intersection point and whether the projection area contains skin.

[0020] In one possible implementation, the triggering condition further includes: the comparison result determined based on the area of ​​the projected region formed by the user's hand and the area of ​​the user's hand falls within a predetermined fifth threshold range; wherein, the area of ​​the user's hand is determined based on the front view of the user's current hand posture; detecting whether the user's hand posture meets the triggering condition further includes: detecting the projected region formed by the user's hand and determining the area of ​​the projected region; determining the front view of the user's current hand posture and determining the area of ​​the user's hand based on the front view; comparing the area of ​​the projected region and the area of ​​the hand to determine whether the comparison result falls within the fifth threshold range.

[0021] In one possible implementation, the triggering conditions further include: the brightness difference between the illuminated and shadowed areas in the target image exceeds a predetermined sixth threshold; detecting whether the user's hand posture meets the triggering conditions further includes: identifying the illuminated and shadowed areas in the target image; and determining whether the brightness difference between the illuminated and shadowed areas exceeds the sixth threshold. The brightness difference between the illuminated and shadowed areas can reflect the light intensity in the target environment to some extent. Generally, users are more likely to have a need for sunshade when the light intensity is high.

[0022] In one possible implementation, triggering the shading operation includes: determining the shading location.

[0023] In one possible implementation, the method is applied to a target environment, including a light-transmitting device; determining the shading position includes: detecting the edge of a projection area formed by the user's hand, and the edge of the user's hand; determining at least three first coordinate points from the edge pixels of the projection area to determine the position of the projection area; determining at least three second coordinate points from the edge pixels of the hand to determine the position of the hand; connecting the corresponding first and second coordinate points to obtain at least three spatial vectors, the spatial vectors pointing from the projection area to the hand position; extending the at least three spatial vectors along the pointing direction, the corresponding at least three extension lines intersecting the light-transmitting device to obtain at least one intersection point; determining a corresponding spatial polyhedron based on the at least one intersection point and the at least three second coordinate points; any cross-section of the spatial polyhedron is determined as the shading position; the cross-section includes a horizontal cross-section or a cross-section parallel to a first plane where the user's palm is located; or, based on at least one intersection point, determining a target area on the light-transmitting device, and determining the target area as the shading position. Determining the shading position based on the user's hand position and the position of the projection area is more in line with user needs.

[0024] In one possible implementation, the method is applied to a target environment, which includes a light-transmitting device; determining the shading position includes: determining a first plane where the palm is located based on the user's current hand posture; determining a second plane parallel to the first plane in the normal direction of the first plane, the second plane being located between the light-transmitting device and the user's hand position; and determining the second plane as the shading position.

[0025] In one possible implementation, the method is based on a sunshade device, which includes a sun visor. After determining the sunshade position, the method further includes controlling the sun visor to move to the sunshade position. Determining the sunshade position based on real-time conditions and then moving the sun visor to that position, rather than pre-setting multiple fixed preset positions, allows for more targeted satisfaction of different users' varying needs for sunshade positions.

[0026] In one possible implementation, the sun visor includes tinted glass for adjusting the shading intensity by controlling the color depth of the tinted glass. After the sun visor is moved to the shading position, the method further includes: determining the duration for which the user's hand remains suspended; and controlling the color depth of the tinted glass of the sun visor based on the duration. Controlling the color depth of the tinted glass can adjust the shading intensity and also adjust the size of the shading area within the boundary range of the sun visor.

[0027] In one possible implementation, the sun visor includes tinted glass for adjusting the shading area by controlling the size of the tinted area of ​​the tinted glass. After the sun visor is moved to the shading position, the method further includes: if the user's hand position is detected to have moved compared to before the shading operation was triggered, increasing the tinted area to increase the shading area. If the user continues to hold their hand still, it indicates that the shading intensity is insufficient, and therefore the glass tint can be further deepened to meet the user's needs.

[0028] In one possible implementation, the light-transmitting device is a windshield; the sunshade is located on the windshield, which is tinted glass, used to adjust the sunshade intensity by controlling the color depth of the tinted glass; after determining the target area as the sunshade location, the method further includes: increasing the color depth of the target area on the windshield. Adjusting the color depth of the windshield is a way to control the light-blocking intensity, eliminating the need for an additional sun visor and reducing the space occupied by sun visors and related accessories in the vehicle's interior.

[0029] In one possible implementation, after triggering the sunshade operation, the method further includes: exiting the sunshade operation if it detects that multiple of the user's fingers are not in a closed state and / or not in a straight state. This method aims to cancel the sunshade operation via any other gesture, meaning the user can command the sunshade to retract via gesture.

[0030] Secondly, this application also provides an electronic device, the electronic device comprising: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of the first aspects.

[0031] Thirdly, the present application also provides an electronic system, which includes an electronic device and a sunshade as described in the second aspect; the electronic device is used to control the sunshade to perform a sunshade operation.

[0032] In one possible implementation, the electronic system further includes at least one camera for acquiring images of the target environment.

[0033] In one possible implementation, the at least one camera includes: at least one RGB camera, and / or, at least one full-resolution depth ToF camera. The depth camera is capable of acquiring depth information in three-dimensional space, and combined with the RGB camera, it helps to construct a 3D model of the target environment.

[0034] In one possible implementation, the target environment is the interior environment of a motor vehicle, and the at least one camera is installed in at least one of the following locations: front reading light, front armrest, driver's steering wheel, passenger screen, central control screen, driver's side door, passenger side door, top of driver's seat, top of passenger seat, B-pillar, A-pillar; rear left door, rear right door, B-pillar, C-pillar, rear armrest, rear left seat, and rear right seat. A sufficient number of cameras will result in richer details of objects captured in the images, leading to higher recognition accuracy.

[0035] Fourthly, the present application also provides a storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of the first aspects.

[0036] Fifthly, the present application also provides a chip system, including: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the method as described in any one of the first aspects.

[0037] In this embodiment, the user's hand posture is used to identify whether the user needs sunshade. Specifically, the trigger conditions are whether the user's hand posture is such that multiple fingers are together and whether the hand is hovering. When the user's hand posture meets the trigger conditions, the sunshade device is automatically deployed to provide automatic sunshade. This identification scheme can accurately capture the user's sunshade intention in most cases and automatically trigger the sunshade operation when the user needs sunshade, reducing the false judgment rate. Attached Figure Description

[0038] Figure 1 This is an example diagram illustrating an application scenario of the sunshade operation triggering method provided in this application embodiment;

[0039] Figure 2 These are multiple scene diagrams in an application scenario example of the sunshade operation triggering method provided in the embodiments of this application;

[0040] Figure 3 This is an example diagram illustrating whether the projection area contains skin, which is a discriminant factor in the shading operation triggering method provided in this application embodiment;

[0041] Figure 4 This is an example diagram showing that the discrimination factor in the shading operation triggering method provided in this application is the projection ratio;

[0042] Figure 5 This is an example diagram illustrating whether the sunshade is triggered based on the projection ratio in the sunshade operation triggering method provided in this application embodiment;

[0043] Figure 6 This is a flowchart illustrating one embodiment of the sunshade operation triggering method provided in this application.

[0044] Figure 7 This is a flowchart illustrating another embodiment of the shading operation triggering method provided in this application.

[0045] Figure 8 This is a flowchart illustrating yet another embodiment of the shading operation triggering method provided in this application.

[0046] Figure 9 This is a flowchart illustrating another embodiment of the shading operation triggering method provided in this application.

[0047] Figure 10 This is an example diagram of the hardware architecture corresponding to the sunshade operation triggering method provided in the embodiments of this application;

[0048] Figure 11 This is a schematic diagram of the method for triggering sunshade operation provided in this application, which determines the included angle based on the detection results of key hand points;

[0049] Figure 12 This is an example diagram showing multiple fingers in a closed state in the sunshade operation triggering method provided in the embodiments of this application;

[0050] Figure 13 This is an example diagram showing the five fingers in an ideally joined state in the sunshade operation triggering method provided in this application embodiment;

[0051] Figure 14 This is a schematic diagram illustrating the detection of whether a finger is straightened in the sunshade operation triggering method provided in this application embodiment;

[0052] Figure 15 This is a schematic diagram illustrating the detection of whether the fingers are straight relative to the palm in the sunshade operation triggering method provided in this application embodiment;

[0053] Figure 16 This is a schematic diagram of the sunshade operation triggering method provided in this application, which determines the sunshade position based on the projection area and the hand position;

[0054] Figure 17 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0056] The sunshade operation triggering method provided in this application embodiment can be applied to various application scenarios. For example, it can automatically provide sunshade for passengers or drivers while vehicles such as motor vehicles, airplanes, and ships are in motion or at a stop; or, in public places such as restaurants and cafes, it can be used as part of smart home control to automatically provide sunshade for users in home environments. Alternatively, this solution can also be applied to scenarios where other vehicles are using high beams while driving at night, where the sun visor can not only block sunlight but also block other light that may interfere with the driver. A typical application scenario is automatic sunshade for the driver or passengers inside a car, and this typical scenario will be used as an example for the following explanation.

[0057] During driving, drivers often encounter situations where the sun's rays are glaring and can scorch their skin. Traditionally, users need to manually open or pull the sun visor to a suitable position. However, due to the size limitations of sun visors, the area they can cover is small and may not reach the areas requiring sun protection. Furthermore, some sun visors are fixed in position and cannot be adjusted, while others are adjustable but with limited flexibility, requiring the driver to operate them for an extended period and impacting driving safety. Therefore, there is a need for sun visors that can automatically deploy at the appropriate location.

[0058] In their research on technologies for automatically deploying sunshades, the inventors discovered that some solutions' triggering mechanisms for automatically deploying sunshades are inaccurate in identifying users' sunshade needs, resulting in a high misjudgment rate. For example, one solution determines whether sunshade deployment is necessary based on factors such as the pitch angle between the sun and the vehicle, and the driver's field of vision. This solution fails to consider the complexity of the driving environment and routes. For instance, in scenarios like driving through tunnels or along roads with tall buildings or shady trees, the pitch angle between the sun and the vehicle remains constant. Based solely on the pitch angle and the driver's field of vision, it might be assumed that the user needs sunshade when they don't. Furthermore, the vehicle's direction of travel constantly changes along the route, making it difficult to accurately determine the pitch angle between the sun and the vehicle. Ignoring the vehicle's actual direction of travel and the influence of the surrounding environment further reduces accuracy. The feasibility of this solution requires further investigation.

[0059] Furthermore, when determining when to provide shade, related technologies often identify the area around the eyes as the target. However, in practice, users don't only need to shield themselves from the sun when it's glaringly bright. They may also need to block sunlight under various lighting conditions to avoid getting tanned. For example, in some weather conditions, although the light intensity may not be high, the ultraviolet radiation may be. While the scattered sunlight may not affect vision, users still need to provide shade to avoid prolonged exposure to ultraviolet rays that could damage their skin.

[0060] Similarly, regarding the location of sun shading, users don't only need to shield their eyes from the sun; their skin may also be a target area requiring sun protection. Current eye-based recognition technologies cannot meet the user's need to cover their skin.

[0061] In addition, different users may have different needs in terms of shading depth and shading area. However, the current solutions in related technologies cannot identify the different needs of different users, and therefore cannot adjust the degree of shading according to different user needs.

[0062] This application provides a method for triggering a shading operation, see below. Figure 1 and Figure 2 As shown, taking a motor vehicle driving scenario as an example, during the driving process, the user refers to... Figure 2 A partial view in the upper left corner shows sunlight and other light entering the vehicle's interior through the windshield 10. Due to glare from the sunlight or for skin protection, please refer to... Figure 1 and Figure 2 In the upper right corner of the view, a user (e.g., a driver) raises their hand in a gesture to shield their eyes from the sun. (See attached image.) Figure 2 A partial view in the lower left corner, based on the sunshade operation triggering method proposed in this application embodiment, identifies the user's gestures and detects whether the user's hand posture meets the triggering conditions for sunshade. If the triggering conditions are met, the sunshade position is further determined based on factors such as the user's gestures, triggering the sunshade to unfold and move to the sunshade position. For example, see [reference]. Figure 2 The partial view in the lower right corner shows the sun visor unfolding and moving to the shading position to block the incoming light, achieving the effect of blocking sunlight. Users no longer need to raise their hands to block the incoming light at the corresponding position.

[0063] For example, when it is detected that the user's five fingers are all together and the hand is in a hovering state, the sunshade operation can be triggered, and the sunshade 11 can be moved to the determined sunshade position to capture the user's sunshade intention and automatically unfold the sunshade based on the user's intention.

[0064] The sunshade operation triggering method proposed in this application can be executed by an electronic device, which can be an in-vehicle electronic device or an electronic device that remotely interacts with a car. Specifically, it can include the following process: tracking the user's hand position and determining a target image containing the hand image; based on the target image, detecting whether the user's hand posture meets the triggering conditions; the triggering conditions include multiple fingers being in a closed state and / or the user's hand being in a hovering state; if the hand posture is determined to meet the triggering conditions, triggering the sunshade operation.

[0065] It should be noted that the triggering conditions can include not only the two discrimination factors of multiple fingers together and hand hovering, but also other discrimination factors. Specifically, the triggering conditions can include at least two of the following five discrimination factors:

[0066] The first indicator: multiple fingers together. When intending to shade from the sun, users usually keep their fingers together to obtain more coverage. Therefore, whether the fingers are spread apart or together can be used as a factor to determine whether shading is triggered.

[0067] The second factor: hand hovering. When using their hands to shield themselves from the sun, users generally don't wave their arms back and forth. Therefore, whether the hand is hovered for a period of time can also be used as a factor in determining the intention to shield oneself from the sun.

[0068] The third item: the fingers are in a straight position. A straight finger position includes at least one finger being straight. For example, in one embodiment, the finger is considered straight only if all fingers are straight; in another embodiment, the user's finger is considered straight if at least one finger is detected to be straight. As one possible implementation, sample statistics can be performed, and the number of fingers used to determine whether a finger is straight can be determined based on the statistical results.

[0069] Fourthly: The user's hand projection area includes skin. In most cases, when a user raises their hand to shield their eyes from the sun, some skin will be obscured. Even in bright sunlight, if the user's intention is to block light from shining into their eyes, the skin around the eyes will still be obscured; in other situations, the user's purpose in shielding their eyes might be to prevent sunburn. Therefore, in the vast majority of cases, the object the user intends to shield from the sun will include skin. If the projection area formed by the user's hand does not include skin, it is very likely that the user's gesture is not intended for sun protection. Therefore, detecting whether the hand projection hits the skin can be used as a criterion for identifying whether the user's action is for sun protection. For example, see... Figure 3As shown, both hand gestures were detected in the projection area. The projection area on the left (from the view-facing angle) contained skin, while the projection area on the right did not. This suggests that the user's hand gesture on the left was likely for sunshade, while the hand gesture on the right was not for sunshade.

[0070] The fifth item: Whether the comparison between the area of ​​the projected region formed by the user's hand and the area of ​​the user's hand falls within a predetermined numerical range (the third threshold range). When a user intends to shade themselves, they generally want to cover as much area as possible, thus controlling the angle and posture of their hand to maximize the projected area. Conversely, when a user does not need to shade themselves, the likelihood of creating a large shading area is lower. Therefore, the ratio of the size of the projected area, or shadow area, caused by the hand posture to the size of the user's hand—that is, the projection ratio—can serve as a criterion to help determine whether the user intends to shade themselves or is simply making a normal movement. For example, see [reference needed]. Figure 4 As shown, three hand gestures yield projection areas of different sizes. The leftmost projection area is larger in proportion to the user's hand size, indicating a higher probability that the user's hand gesture is for sun shading. The rightmost projection area is smaller, suggesting that although the user has raised their hand, it may not be for sun shading. The middle projection area falls somewhere in between. The projection ratio can, to some extent, reflect whether the intention of the gesture is sun shading. For example, see [link to relevant documentation]. Figure 5 As shown, when the projection ratio is large, such as exceeding the upper limit of the predetermined threshold, it is considered that the sunshade needs to be triggered; when the projection ratio is small, such as less than 30% or less than 25%, it is considered that the sunshade does not need to be triggered.

[0071] The sixth criterion: The brightness difference between the illuminated and shadowed areas in the target image exceeds a predetermined fourth threshold. In most cases, shading occurs in scenes with strong sunlight. Therefore, the shading by a hand will result in a strong contrast between the shaded and unshaded areas. Thus, the brightness difference between the illuminated and shadowed areas can be used as a factor to determine whether shading needs to be triggered.

[0072] Generally, to ensure the accuracy of the user's sunshade requirement recognition results, the triggering condition is at least a combination of two factors: multiple fingers together and hand hovering. Optionally, in other embodiments, it can be a combination of two discrimination factors: whether multiple fingers are straight and hand hovering, and at least one of three other discrimination factors. In some embodiments, the triggering condition can be a combination of one of multiple fingers together and hand hovering, and at least one of three other discrimination factors.

[0073] The following are several implementation methods that use different trigger conditions to identify a user's intention to shade their eyes:

[0074] Method 1

[0075] See Figure 6 As shown, Method 1 may include the following process:

[0076] 601, Track the user's hand position and determine the target image containing the hand image.

[0077] 602. Detect whether the user's multiple fingers are in a closed position. If yes, proceed to 603; otherwise, exit.

[0078] 603, Detect whether the user's hand is in a hovering state. If yes, proceed to 604; otherwise, exit.

[0079] 604, triggering sunshade operation.

[0080] As can be seen, in this implementation, the triggering conditions include whether multiple fingers are together and whether the hand is hovering. First, it detects whether multiple fingers are together. If multiple fingers are together, it triggers the detection of whether the hand is hovering. After detecting that the hand is hovering, it triggers the sunshade operation, controlling the sunshade to move to the designated sunshade position. The determination of the sunshade position will be described in detail later.

[0081] Method 2

[0082] See Figure 7 As shown, Method 2 may include the following process:

[0083] 701, Track the user's hand position and determine the target image containing the hand image.

[0084] 702. Detect whether the user's multiple fingers are in a closed position. If yes, proceed to 703; otherwise, exit.

[0085] 703, Detect whether the user's hand is in a hovering state. If yes, proceed to 704; otherwise, exit.

[0086] 704: Detect whether the user's finger is straight; if yes, proceed to 705; otherwise, exit.

[0087] 705, triggering sunshade operation.

[0088] As can be seen, in this implementation, the discrimination factors include three items: whether multiple fingers are together, whether the hand is suspended, and whether the fingers are straight. That is, the triggering condition is: multiple fingers are together, the hand is suspended, and the fingers are straight. If the hand posture is detected to meet the triggering condition, the sunshade operation is triggered.

[0089] It should be noted that the step numbers corresponding to multiple discriminant factors in Method 1 or Method 2 and other subsequent methods are for descriptive purposes only and do not represent a limitation on the execution order of the steps corresponding to multiple discriminant factors. For example, in Method 1, the execution order of steps 602 and 603 can be either 602 first and then 603, or 603 first and then 602. In Method 2, steps 702, 703 and 704 have multiple possible execution orders, and any one of them can be executed first. For example, one possible execution order is: execute 704 first, then 702, and then 703, etc.

[0090] Method 3

[0091] See Figure 8 As shown, method three may include the following process:

[0092] 801, Track the user's hand position and determine the target image containing the hand image.

[0093] 802. Detect whether the user's multiple fingers are in a closed position. If yes, proceed to 803; otherwise, exit.

[0094] 803, Detect whether the user's hand is in a hovering state. If yes, proceed to 804; otherwise, exit.

[0095] 804. Detect whether the projection area formed by the user's hand contains skin; if yes, proceed to 805; otherwise, exit.

[0096] 805, detects the projection area formed by the user's hand;

[0097] 806, Determine the projection area and the user's hand area, and determine the ratio of the projection area to the user's hand area;

[0098] 807. Does the ratio value fall within the predetermined numerical range? If yes, proceed to step 808; otherwise, exit.

[0099] 808, triggering sunshade operation.

[0100] Method 4

[0101] In cases where the projection area formed by the hand cannot be identified, the brightness difference between the illuminated and shaded sides of the vehicle interior environment can be used to help determine whether a sunshade operation has been triggered.

[0102] See Figure 9 As shown, method four may include the following process:

[0103] 901, track the user's hand position and determine the target image containing the hand image.

[0104] 902. Detect whether the user's hand is in a hovering state. If yes, proceed to 903; otherwise, exit.

[0105] 903. Detect whether the user's finger is straight. If yes, proceed to 904; otherwise, exit.

[0106] 904. Detect whether the brightness difference between the illuminated area and the backlit area in the target image exceeds the predetermined fourth threshold; if yes, proceed to 905; otherwise, exit.

[0107] 905, triggering sunshade operation.

[0108] This method is primarily suitable for situations where the light intensity is high, resulting in a significant difference in brightness between the illuminated and shaded sides in images of target environments such as the interior of a vehicle. This method can effectively identify a user's need for sun shading in glaring sunlight, but it is not suitable for cloudy days or other weather conditions where the light intensity is low but the ultraviolet radiation is high.

[0109] The four implementation methods described above are merely examples. Based on the aforementioned six discrimination factors, other implementation methods can be obtained. For instance, by changing the execution order of multiple discrimination factors in method three or four, various other implementation methods can be obtained. For example, in method four, it is possible to first detect whether the user's finger is straight and then detect whether it is in a hovering state; or, in method three, deleting step 804 and not detecting whether the projection area contains skin can yield a corresponding implementation method; or, in method three, retaining step 804 and deleting steps 805 and 806 yields yet another implementation method. Specifically, at least two of the aforementioned six discrimination factors can be selected for detection, and by changing the execution order of at least two discrimination factors, various other implementation methods can be obtained. These are not listed one by one in the embodiments of this application.

[0110] To further understand the shading operation triggering method provided in the embodiments of this application, the relevant hardware architecture is described. See also... Figure 10 As shown, the solution provided in the embodiments of this application can be based on, as Figure 10The hardware architecture shown is implemented as follows. This hardware architecture includes a sun visor 11 and at least one camera. The camera is one of the hardware devices for implementing the technical solutions provided in this application embodiment. The at least one camera may include at least one red-green-blue (RGB) three-channel camera 12, and may also include at least one Time of Flight (ToF) camera 13. The RGB camera can be exposed using a photosensitive element, and the image content captured by the RGB camera can be identified using algorithms. The ToF camera is a method of imaging surrounding content by sending waves and receiving the returned signals. ToF cameras can easily obtain distance information, and their working principle is similar to radar. Some ToF cameras also have infrared detection capabilities, supporting infrared detection and simultaneously obtaining temperature information. It should be noted that... Figure 10 The graphic symbols for RGB cameras and ToF cameras are for illustrative purposes only. The different icons are used to distinguish them as different types of image acquisition devices. In practical applications, RGB cameras and ToF cameras can adopt various shapes or designs, and there are no restrictions on this.

[0111] Sunshades may include tinted glass. Figure 10 The component shown, composed of multiple small diamond-shaped grids, is a color-changing glass, with each grid representing a color-changing unit. The shape of this color-changing unit is merely an example; various shapes can be used in practical applications, and this application does not limit this. Electronic devices can adjust the shading intensity by controlling the color depth of the color-changing glass, and adjust the shading area by controlling the number of grids.

[0112] The sun visor 11, the RGB camera 11, and the ToF camera are all connected to an electronic device, which includes a central control unit for controlling the sun visor 11, the RGB camera 11, and the ToF camera. The electronic device can be connected to the RGB camera 11 or the ToF camera via an electrical connection (e.g., a data cable connection) or a wireless signal connection.

[0113] Taking the automotive scenario as an example, the above method can be applied to electronic devices. These electronic devices can be independent in-vehicle electronic devices or one or more components integrated into the in-vehicle central control platform. Specifically, they can be software modules or hardware components in the vehicle that have computing capabilities and can communicate with the sun visor 11, RGB camera 12, or ToF camera 13. For example, the electronic device can be one or more of the following in-vehicle devices or modules, or one or more components integrated into the following in-vehicle devices or modules: vehicle gateway (GW) module, telematics control unit (TCU), telematics box (T-BOX), mobile data center (MDC), human-machine interaction (HMI) module, electronic control unit (ECU), cockpit domain controller (CDC), or vehicle domain controller (VDC).

[0114] The Vehicle Gateway (GW) module is a core component of the vehicle's electronic and electrical architecture. As the data interaction hub of the vehicle network, it can route network data from various networks, including Controller Area Network (CAN), Local Interconnect Network (LIN), Media Oriented System Transport (MOST), and FlexRay. The Mobile Data Center (MDC) serves as the vehicle's intelligent in-vehicle computing platform; the Human-Machine Interface (HMI) module supports the vehicle's infotainment system; the Telematics Control Unit (TCU) communicates with external systems, backend systems, smartphones, and application network devices; the Vehicle Box (T-BOX) communicates with external systems, backend systems, smartphones, and app network devices; the Electronic Control Unit (ECU) is a dedicated automotive microcomputer controller; the Cockpit Domain Controller (CDC) controls cockpit-related content; and the Vehicle Domain Controller (VDC) controls vehicle-related content.

[0115] In another embodiment, the electronic device may also be a device capable of remotely interacting with the vehicle.

[0116] The following section details how to detect each discrimination factor.

[0117] First, based on the global image of the target environment (e.g., the interior of a vehicle) captured by at least one camera, the user's hand position is tracked to initially determine a local image (i.e., the target image) including the hand image. Based on the target image, the first discrimination factor—whether multiple fingers are together—is detected. For example, the specific detection methods can be as follows: Method 1: Detect the angle between adjacent fingers and determine whether they are together based on the angle; Method 2: Perform hand edge detection and determine whether they are together based on the length of the hand edge.

[0118] One method for detecting the angle between adjacent fingers is based on key hand points. This angle is used to determine whether the two adjacent fingers are touching. Multiple fingers can be divided into multiple groups of adjacent fingers for identification, thereby determining whether all the fingers on the entire hand are touching.

[0119] The detection of the angle between two adjacent fingers can be achieved through the following steps: Based on the target image, hand keypoint detection is performed, identifying multiple keypoints on the user's hand. These keypoints correspond to multiple bone nodes in the hand. Hand detection is an algorithm-based process in image recognition, where keypoints can be compared by learning from hand image samples. For example, a basic recognition of hand keypoints can be obtained by calling the OpenPose open-source library for human pose recognition in the cross-platform computer vision and machine learning software library OpenCV. Among the obtained keypoints, adjacent keypoints corresponding to the same finger are connected by straight lines to determine the line segments corresponding to each finger; the angle between the line segments corresponding to two adjacent fingers is then determined as the angle between the two adjacent fingers.

[0120] For example, see Figure 11 As shown, by detecting key points on the hand, it is determined that... Figure 11 The key points shown are, taking the little finger as an example, key points A, B, C, and D are the fingertip node, the first bone node, the second bone node, and the finger root node of the little finger, respectively. Key point E is the hand bone node, and all five fingers correspond to the same hand bone node E.

[0121] As can be seen, multiple key points can be detected for each finger. Based on the connection characteristics of the human hand bones, adjacent key points of multiple key points corresponding to the same finger are connected. For example, the multiple key points corresponding to the little finger are A, B, C, and D. Connecting two adjacent key points means connecting key point A with key point B, B with C, and C with D, resulting in line segments AB, BC, and CD. In one embodiment, the angle between two adjacent fingers is determined based on the angle between the line segments closest to the base of the finger. For example, among the multiple line segments corresponding to the little finger, CD is the line segment closest to the base of the finger. In the ring finger, the line segment closest to the base of the finger is GF. The angle between the little finger and the ring finger is the angle between line segment CD and line segment GF. Figure 11 The included angle α shown.

[0122] Similarly, the angles between the ring and middle fingers, the middle and index fingers, and the index and thumb can be obtained sequentially. These angles are compared with predetermined angle threshold ranges. If a finger falls within the corresponding threshold range, it is considered that the two fingers are together. In this way, it is determined whether the little finger and ring finger, the ring and middle fingers, the middle and index fingers, and the index and thumb are together, thus detecting whether multiple fingers of the entire hand are together. For example, the angle threshold range can be [-5°, +5°] or (-6°, +6°). Figure 11 If the included angle α shown is greater than 5°, then the little finger and ring finger are determined to be in an open state, that is, not in a closed state.

[0123] It should be noted that "multiple fingers together" can refer to either all fingers being together, or at least two adjacent fingers being together. In other words, adjacent fingers are considered a group, and when at least two groups of fingers are together, the user's hand is considered to have multiple fingers together. For example, ... Figure 12 As shown, Figure 12 In the hand image shown, the index and middle fingers are together, the middle and ring fingers are together, the little finger is not together with the ring finger, and the thumb is not together with the index finger. This situation can also be judged as multiple fingers of the hand being together.

[0124] Additionally, it should be noted that being in a closed state does not mean that the fingers are completely and tightly pressed together. In practical applications, many users will have multiple fingers in a slightly closed state that is close to touching or close together when shading themselves from the sun. Whether or not they are in a closed state can be determined by the methods listed in the embodiments of this application.

[0125] The following describes a method for determining whether multiple fingers are together based on the length of the hand's edge. This method may include the following steps:

[0126] First, edge detection is performed on the target image to determine the edges (also known as boundaries or contours) of the user's hand, specifically by determining the edge pixels of the hand image.

[0127] Edge detection identifies image edges by detecting significant changes within a certain range of pixels. Specifically, it uses various filters to highlight differences in a predetermined manner to achieve edge detection. For example, filters from the field of image recognition, such as Sobel, Prewitt, Roberts, and Canny, can be used.

[0128] Then, the lengths of the edges of the hand when multiple fingers are held together are determined. To prevent confusion, the lengths of the hand edges when multiple fingers are held together are defined as the first reference length. In one embodiment, the first reference length can be determined as follows: key point detection is performed on the hand to obtain... Figure 7 The keypoint detection results shown are from Figure 11 The hand key points shown can be used to determine the ideal edge length of each finger in the hand model based on the length of the line segment corresponding to each finger. For example, Figure 11 The length h1 of the little finger is the sum of the lengths of line segments AB, BC, and CD, i.e., h1 = |AB| + |BC| + |CD|. In one possible implementation, since the difference in finger diameter (width) between different people is relatively small, generally not exceeding 0.5cm, information from multiple users' hands can be collected in advance as samples. The average width of the little finger from these multiple user hand samples can be calculated as the arc-shaped edge length x1 at the fingertip of the current user's little finger. Then, the arc-shaped edge length at the fingertip plus the edge lengths on both sides is the edge length of the finger when it is not closed. The edge length on one side can be equal to the length h1 of the little finger. Therefore, the edge length d1 of the little finger in the open state is d1 = x1 + 2h1. By analogy, the edge lengths of the ring finger, middle finger, index finger, and thumb in the open state can be obtained as d2, d3, d4, and d5, respectively.

[0129] Then, the edge length L of the entire hand with all adjacent fingers completely and tightly pressed together is the sum of the edge lengths of each of the five fingers in their open state, minus the lengths that were counted repeatedly. For example: L = d1 + d2 + d3 + d4 + d5 - 2h1 - 2h2 - 2h4, where h1, h2, and h4 are the lengths of the little finger, ring finger, and index finger, respectively. This provides a reference value for the completely tightly pressed state, which can be used as the minimum value of the first reference length.

[0130] The aforementioned near-closed, slightly joined state between adjacent fingers can also be considered as a closed state. As a possible implementation, multiple user samples can be pre-collected, showing the hand edge length L0 when the hand makes a sun-shading gesture (including the slightly joined state, where there is still a gap between the fingers). The percentage difference between each user sample value L0 and the corresponding L value is calculated, i.e., (L0-L) / L. This determines the error range between the minimum edge length L in a completely tightly joined state and the sample value L0. For example, after removing the maximum and minimum values, the value with the largest percentage difference from the remaining samples is selected as the error range value. For instance, if the largest percentage difference is 10%, then the range of the first reference length is [L, L*(1+10%)].

[0131] Based on the above method, the estimated value range of the edge length of the entire hand in the closed state can be calculated, or the estimated value range of the edge length of two adjacent fingers in the closed state can be calculated. This estimated value is the first reference length. In the embodiments of this application, the first reference length is the numerical range of the estimated value of the hand edge length in the closed state; the second reference length is the estimated value of the hand edge length when multiple fingers are fully spread.

[0132] Then, based on the coordinates of the boundary pixels, a first length corresponding to the hand edge is calculated. At least based on this first length, it is determined whether the user's multiple fingers are in a closed state. Specifically, the first length and a first reference length are compared, and based on the comparison result, it is determined whether the user's multiple fingers are in a closed state. Specifically, if the comparison result determined based on the first length and the first reference length falls within a predetermined first threshold range, it is determined whether the user's multiple fingers are in a closed state.

[0133] It should be noted that the first length refers to the edge length of at least one finger or the entire palm detected after edge detection of the acquired image; it can be understood as the measured length of the user's hand edge. The first reference length, on the other hand, is a reference value for determining the edge length of the hand when it is in a closed position.

[0134] For example, comparing the first length and the first reference length can be done by calculating the ratio of the first length to the first reference length, or by calculating the difference between the first length and the first reference length. There are various possible calculation methods or functions for this comparison. If the difference between the first length and the first reference length exceeds the allowable error range (i.e., the first threshold range), it is determined to be in a non-merging state; if it falls within the allowable error range, it is determined to be in a merged state.

[0135] In some embodiments, a second reference length corresponding to the edge of the user's hand when multiple fingers are in an open state is also predetermined. The significance of the second reference length is to assist in judging the validity of the first length detected by the edge. If the actual detected edge length is not only greater than the first reference length but also greater than the second reference length, it indicates that the detected first length is invalid data and needs to be checked or removed. Specifically, the judgment process involving the second reference length is as follows: compare the first length with the second reference length; if the first length is less than the second reference length, compare the first length with the first reference length; if the comparison result determined based on the first length and the first reference length falls within a predetermined first threshold range, determine whether the user's multiple fingers are in a closed state. If the first length is greater than, equal to, or close to the second reference length, it indicates that the first length is erroneous invalid data, or that multiple fingers of the hand are in an open state.

[0136] The following describes how to compare the first length and the reference length. The comparison can be done by comparing the entire hand together, that is, comparing the first length of the entire hand with the first reference length of the entire hand. Alternatively, it can be done by grouping adjacent fingers together and comparing the first length of each pair of fingers with the corresponding first reference length.

[0137] Specifically, the comparison method using two fingers as a group can include the following steps: Grouping adjacent fingers into two groups, determine the third length corresponding to the edge formed by the adjacent two fingers in each group; compare the third length of each group with the first reference length; based on the comparison results, identify whether the corresponding adjacent two fingers are in a closed state, thereby determining whether the user's multiple fingers are in a closed state. The third length represents the measured edge length of two adjacent fingers.

[0138] For example, if the user's actual hand posture is as follows Figure 11 As shown, after keypoint detection, a structure can be constructed as follows: Figure 13 The ideal hand model shown obtains estimated edge lengths of two adjacent fingers based on the actual length of the user's hand. For example, see [reference needed]. Figure 13As shown, taking the little finger and ring finger as a group, based on key point detection, the length of the two sides of the little finger can be approximated as the length of line segment AD from the fingertip node to the finger root node, and the length of the two sides of the ring finger can be approximated as the length of line segment JF. Therefore, in the closed state, assuming the length of AD is h1, the length of JF is h2, the length of the arc edge at the tip of the little finger is x1, and the length of the arc edge at the tip of the ring finger is x2, then the sum of the edge lengths of the little finger and ring finger in the closed state can be estimated as h1 + x1 + (h2 - h1) + x2 = h2 + x1 + x2, where x1 and x2 can be taken as constant values ​​according to the actual width of the little finger and ring finger of the human hand, and the differences between different users are calculated within the error range.

[0139] Then, based on the actual detection results, the edge lengths of the little finger and ring finger detected are much greater than h2+x1+x2. Therefore, it is determined that the little finger and ring finger are in an open state in the user's current hand posture. By analogy, the ring finger and middle finger are then judged to obtain the recognition results of whether multiple fingers of the hand are together.

[0140] If the user's actual hand posture is as follows Figure 12 As shown, based on the constructed model of the ideal joined state, the estimated edge lengths of the index and middle fingers are obtained. This estimated value is compared with the first length. If it falls within the error range, the index and middle fingers are considered to be joined. The error range can be ±15% or ±10%. For example, if the estimated value of the joined state is 15cm and the actual measured edge length is 17cm, then it is determined to be in the joined state.

[0141] Using the entire hand as the comparison object involves comparing the first length of the entire hand with the first reference length of the entire hand. Specifically, the first reference length of the entire hand is determined firstly. Then, based on the edge detection results and the coordinate values ​​of the determined edge pixels, the actual object corresponding to the hand image, i.e., the actual edge size of the human hand, is calculated. The first reference length of the entire hand is compared with this actual edge size. If it falls within the allowable error range, it means that multiple fingers of the hand are in a closed state. If it exceeds the allowable error range, it is determined that multiple fingers of the hand are not in a closed state.

[0142] It should be noted that when people use their hands to shield themselves from the sun, they generally try to stretch their fingers as much as possible to obtain a larger shielding area, and their fingers can be approximately straight. Therefore, the discrimination method based on edge detection and key point detection mentioned above has certain practicality.

[0143] The following describes how to detect whether a user's hand is in a hovering state. In one embodiment, the following method can be used: Given that multiple fingers of the user are in a closed position, detect multiple key points on the user's hand; based on hand skeletal connection information, connect two adjacent key points to determine at least one line segment; detect the displacement of at least one line segment within a predetermined time period; based on the displacement, determine whether the user's hand is in a hovering state. Since the movement characteristics of the human hand are such that the probability of finger movement without palm movement is higher than the probability of finger stillness without palm movement, in one embodiment, at least one line segment can be a line segment located on the fingers, rather than a line segment corresponding to the palm. For example, ... Figure 11 As shown, the detection target can be based on at least one of the line segments AB, BC, CD, and GF, instead of line segment DE.

[0144] The method for detecting the displacement of at least one line segment within a predetermined time period can be as follows: Determine multiple frames of images within the predetermined time period; detect a first line segment in each frame of the multiple frames, and determine a first pixel and a second pixel; wherein the first line segment is one of at least one line segment; the first pixel and the second pixel are the two endpoints of the first line segment; obtain the first coordinate value of the first pixel and the second coordinate value of the second pixel in each frame of the images; calculate a first average of multiple first coordinate values, and use the pixel with the first average as its coordinate value as a first reference pixel; and calculate a second average of multiple second coordinate values, and use the pixel with the second average as its coordinate value as a second reference pixel; determine a reference line segment using the first reference pixel and the second reference pixel as its two endpoints; calculate the distance between the first line segment and the reference line segment in each frame of the images, and use the obtained distances as the displacement of at least one line segment within the predetermined time period. Then, if all distances fall within a predetermined fourth threshold range, it is determined that the user's hand is in a hovering state.

[0145] For example, taking line segment GF as the first line segment to be detected, and setting the predetermined time to 0.5 seconds, assuming that 1 second includes 24 frames of images, then the 0.5-second range includes 12 frames of images. The pixels at the two endpoints of the first line segment in each of these 12 frames are identified, that is, the positions of the second knuckle and the base of the ring finger in each frame are determined. The corresponding pixels are then averaged across the 12 frames. For example, the coordinates of point G are (Xa, Ya, Za), and the coordinates of point F are (Xb, Yb, Zb). The corresponding coordinates of the first line segment are (Xa, Ya, Za) - (Xb, Yb, Zb). The coordinates of point G in the 12 frames are (Xa1, Ya1, Za1), (Xa2, Ya2, Za2) ... (Xa1, Ya1, Za1), (Xa2, Ya2, Za2), ... (Xa1, Ya1, Za1), (Xa2, Ya2, Za2), ... (Xa1, Ya1, Za1), (Xa2, Ya2, Za2), ... (Xa1, Ya1, Za1), (Xa2, Ya2, Za2), ... (Xa1, Ya1, Za1), (Xa2, Ya2, Za2), ... (Xa2, Ya1, Za1), ... Given points F (Xb1,Yb1,Zb1), (Xb2,Yb2,Zb2), ..., (Xb12,Yb12,Zb12), calculate the average of (Xa1,Ya1,Za1), (Xa2,Ya2,Za2), ..., (Xb12,Ya12,Zb12), and the average of (Xb1,Yb1,Zb1), (Xb2,Yb2,Zb2), ..., (Xb12,Yb12,Zb12). This yields the first reference pixel coordinates as (Xe-a,Ye-a,Ze-a) and the second reference pixel coordinates as (Xe-b,Ye-b,Ze-b). Finally, obtain the reference line segment (Xe-a,Ye-a,Ze-a). )-(Xa1,Ya1,Za1)-(Xb1,Yb1,Zb1) and the reference line segment are calculated sequentially, as are the distances between line segment (Xa2,Ya2,Za2)-(Xb2,Yb2,Zb2) and the reference line segment, and so on. If the 12 distance values ​​obtained fall within the pre-set fourth threshold range, the hand is considered to be in a hovering state.

[0146] The fourth threshold range can be a range of distance values, the maximum of which can be 1 / 4 to 1 / 10 of the length of the middle finger (e.g., the length from key point E to the fingertip key point). For example, since it is difficult for a person's hand to remain absolutely still while driving, the fourth threshold range is set to ±3cm. This means that any slight movement of the user's hand within a 3cm radius around the reference line segment can be considered as hovering, equivalent to the slight movement of the first line segment in the 12-frame image within the cylindrical space around the reference line segment. In one feasible approach, the specific value of the fourth threshold can be obtained statistically based on multiple actual user samples. Other threshold ranges or error ranges involved in the embodiments of this application can also be obtained statistically based on multiple actual user samples.

[0147] It should be noted that the first line segment can be a combination of multiple line segments, for example, such as... Figure 7 As shown, the first line segment can also be line segment AD, or line segment BD, etc., and is not limited to the line segment determined by two adjacent key points.

[0148] The following explains how to detect whether a user's fingers are straight. Based on the results of hand key point detection, each finger can be viewed as three line segments composed of four points. See [link to relevant documentation]. Figure 14 As shown, exemplarily, in a finger, along the direction from the fingertip to the base of the finger, the bone nodes are sequentially defined as the first node a (fingertip node), the second node b, the third node c, and the fourth node d (finger base node). In space, two points determine a straight line. The connection between the third node c and the fourth node d defines a straight line. For clarity, the straight line (or line segment) cd defined by the third node c and the fourth node d is defined as the first straight line; the straight line ab defined by the first node and the second node b is defined as the second straight line; and the straight line bc defined by the second node b and the third node c is defined as the third straight line. For example, as... Figure 10 As shown, the second line forms an angle β with the first line. It should be noted that the thumb only includes three nodes and two line segments.

[0149] Generally, when users need sun protection, they tend to straighten their fingers rather than deliberately bending them, especially the portion between the fingertip and the third node. Therefore, in one embodiment, one of the two lines can be selected to determine the angle formed with the first line. That is, the angle between the first and second lines, or between the first and third lines, is detected to determine whether the finger is straight. For example, the angle between the first line cd and the second line ab is calculated; if this angle falls within a second threshold range, the finger is considered straight. For instance, the second threshold range corresponding to the second line could be [-15°, +15°], and the finger is considered straight when the angle between the second line (or third line) and the first line is less than or equal to 15°. Alternatively, the second threshold range corresponding to the third line could be [-7°, +7°], and the finger is considered straight when the angle between the third line and the first line is less than or equal to 5°. In another embodiment, the first angle formed between the second line and the first line, and the second angle formed between the third line and the first line can be detected simultaneously. When the first angle is less than 15° and the second angle is less than 7°, the finger is considered to be in a straight state.

[0150] Furthermore, in another embodiment, the fingers being in a straightened state also includes the fingers being straightened relative to the palm; that is, the entire hand being in a straightened state can also be considered a state of finger straightening. In this embodiment, the palm can be used to further identify whether the fingers are straightened relative to the palm. See also Figure 15 As shown, the palm can be considered as another separate node e, defined as the fifth node. In space, three points determine a plane, which can be the root nodes of two adjacent fingers, that is, two fourth nodes d( Figure 15 The image shows only one finger root node) and the fifth node e to determine the plane where the palm is located. This plane is defined as the first plane. Using the first plane as a reference plane, the angle between at least one of the first, second, and third straight lines and the first plane falls within a third threshold range to determine whether the fingers are straight relative to the palm. For example, Figure 15 The diagram shows the angle α2 formed by the first straight line cd and the first plane, and the angle α1 formed by the second straight line ab and the first plane. If, based on the aforementioned embodiment, it has been detected that all fingers are straight, and the first, second, and third straight lines are within a certain range of the first plane (i.e., the third threshold range), then for example, if the angle α1 between the second straight line and the first plane is 10°, the angle α2 between the third straight line and the first plane is 5°, then it can be considered that the fingers are in a straight state relative to the palm.

[0151] The following describes how to identify whether the projection area formed by the user's hand contains skin. First, based on the user's current hand posture, determine the first plane where the palm is located; determine the intersection point of the normal of the first plane and the user's body surface; detect whether a projection occurs within a preset distance range around the intersection point and whether the projection area contains skin.

[0152] By acquiring images from multiple angles, 3D modeling of the cabin's spatial environment can be performed. For example, KinectFusion, Kintinuous, ElasticFusion, InfiniTAM, DynamicFusion, and BundleFusion can be used for 3D modeling. This allows us to obtain the intersection point of a normal perpendicular to the plane of the hand with other objects, specifically the intersection of the normal with the user's body surface. After determining this intersection point, we further detect whether a shadow is generated and whether skin is included within a preset distance around the intersection point. In other words, we check whether a projection area and a skin area can be detected around the intersection point, and whether the projection area and the skin area overlap or intersect; that is, a portion of the pixels around the intersection point belong to both the skin area and the projection area.

[0153] Specifically, the process first detects shadows around the intersection points to determine the shadow area; then, skin detection is performed around the intersection points to determine the skin area. There are several ways to achieve shadow detection, such as using OpenCV, or employing shadow detection based on the HIS (hue, saturation, intensity) or RGB color spaces, or using spatial context features.

[0154] Skin detection can be achieved through at least one of the following methods:

[0155] Identification by temperature: Similar to a thermometer, an infrared camera can collect temperature information of objects within its field of view. The temperature of exposed human skin is significantly higher than that of other objects in the scene, thus enabling skin identification.

[0156] Image recognition: Similar to edge detection, the extent of skin can be obtained in image recognition by using the correct filters or operators. For example, some known algorithmic models include Bayesian Classifier (BC), Linear Discriminant Analysis (LDA), Binary Logistic Regression (BLR), and Adaptive Neuron Fuzzy Inference System (ANFIS).

[0157] wait;

[0158] Identification by spectrum: Infrared cameras capture images by collecting infrared reflections. Human skin differs from other materials in reflectivity, so human skin can be identified based on the difference in reflectivity.

[0159] The following explains how to detect whether a user's hand posture meets the sixth criterion. Specifically, it involves detecting the projection area formed by the user's hand and determining the area of ​​the projection area; determining the front view of the user's current hand posture and, based on the front view, determining the area of ​​the user's hand; comparing the area of ​​the projection area and the area of ​​the hand, and determining whether the comparison result falls within the fifth threshold range.

[0160] The above describes how to detect the projection area. After detecting the projection area formed by the user's hand, its geometric area is calculated. Then, based on the 3D model of the hand, a front view image of the user's hand is obtained. Based on the front view image, hand edge detection is performed to obtain the hand area.

[0161] When a user needs shade, they will control their gestures to obtain the largest possible shading area, i.e., the largest possible projection area. However, since different users have different hand sizes, the size of their projection areas also varies. Simply comparing the size of the projection area does not take into account these user differences, resulting in low accuracy. Therefore, this application proposes comparing the area of ​​each user's hand with the projection area.

[0162] It should be noted that the comparison result determined based on the area of ​​the projected region formed by the user's hand and the area of ​​the user's hand can, in one embodiment, be the ratio of the area of ​​the projected region formed by the user's hand to the area of ​​the user's hand; in another embodiment, it can be the difference between the area of ​​the projected region (i.e., the shadow area) formed by the user's hand and the area of ​​the user's hand. The setting of the fifth threshold range depends on the data type of the comparison result.

[0163] Due to factors such as the camera's shooting angle and the complexity of the actual application environment, there may be situations where the shadow area of ​​the hand cannot be accurately identified. In this case, the sixth discrimination factor can be used to replace the discrimination factor that needs to identify the shadow area of ​​the hand. Alternatively, if the shadow area can be identified, the fifth discrimination factor can be used for detection.

[0164] Specifically, the system identifies the illuminated and shadowed areas in the target image and detects whether the brightness difference between the illuminated and shadowed areas exceeds a predetermined sixth threshold. The brightness difference between the illuminated and shadowed surfaces in an in-vehicle image reflects the current high light intensity and strong light. In most cases, users' need for sunshade often occurs in situations with high light intensity. Therefore, comparing the dark and illuminated surfaces in an image is meaningful.

[0165] Furthermore, in some cases, the relative position of the user's hand and skin can also help determine whether a sunshade device needs to be triggered. For example, the system can model the user's hand using images to determine the approximate direction of the hand, which is the direction along the z-axis with the palm as the plane. If the intersection of this direction and the body is at the skin, and the angle is less than a certain range—that is, the intersection falls within the coordinate range (xa, ya)(xb, yb) enclosed by a circular or rectangular area—then it is considered that the hand is facing the skin, indicating that the user is likely making a sunshade gesture.

[0166] It should be noted that, to ensure the accuracy of triggering operations and reduce the user experience degradation caused by false triggers, the system can ask the user whether to turn on the sunshade if the user's gesture meets the triggering conditions. For example, it can play a pre-set voice prompt, "Do you need a sunshade?", and only trigger the operation after receiving a positive response from the user. Voice-related operations can be implemented through the in-vehicle voice assistant.

[0167] Based on the above description, the method for triggering the sunshade operation is clarified. After determining that the sunshade operation needs to be triggered, the sunshade position must first be determined, and then the sunshade is moved to the sunshade position. Alternatively, if the windshield or other light-transmitting devices are tinted glass, the sunshade position can be determined on the windshield, and the color depth of the tinted glass at the sunshade position can be controlled to achieve the purpose of sunshade.

[0168] Specifically, the location of the shading can be determined in the following ways:

[0169] The system detects the edge of the projection area formed by the user's hand, as well as the edge of the user's hand; from the edge pixels of the projection area, it determines at least three first coordinate points for determining the position of the projection area; from the edge pixels of the hand, it determines at least three second coordinate points for determining the position of the hand; it connects the corresponding first and second coordinate points to obtain at least three spatial vectors, the spatial vectors pointing from the projection area to the hand position; along the pointing direction of the spatial vectors, it extends at least three spatial vectors, and the corresponding at least three extension lines intersect with the light-transmitting device to obtain at least one intersection point.

[0170] In one embodiment, sun shading is achieved via a sun visor. A corresponding spatial polyhedron is determined based on at least one intersection point and at least three second coordinate points; any cross-section of the spatial polyhedron is determined as the shading position; the cross-section includes a horizontal cross-section or a cross-section parallel to a first plane where the user's palm is located.

[0171] In another embodiment, sun shading can also be achieved using a light-transmitting device such as a windshield. After determining at least one intersection point where at least three extended lines intersect with the light-transmitting device (e.g., the windshield), a target area is determined on the light-transmitting device such as the windshield based on at least one intersection point. For example, the target area on the windshield can be determined based on at least one intersection point in the following manner:

[0172] Method 1: When there is only one intersection point, the target area is defined by taking that intersection point as the center or the center of a circle. For example, if the intersection point is the center and a preset radius of 10cm is set, then the circular area with a radius of 10cm centered on the intersection point is the target area. Another example is to use the intersection point as the center of symmetry to define a rectangular (e.g., square or rectangle) area as the target area.

[0173] Method 2: When at least one intersection point is two intersection points, use the line connecting the two intersection points as the diameter or as the diagonal to construct a corresponding circular or rectangular region as the target region.

[0174] Method 3: When there are at least three or more intersection points, connect each pair of intersection points with lines to obtain multiple boundary lines. The area enclosed by the largest number of boundary lines is taken as the target area. For example, with three intersection points, connecting each pair of intersection points creates a triangle; with four intersection points, connecting each pair of intersection points creates a quadrilateral; with five intersection points, a pentagon is formed; and for multiple intersection points, the polygon formed by connecting each pair of intersection points is taken as the target area.

[0175] Alternatively, if there are at least three or more intersection points, the center point of multiple intersection points can be determined. For example, the average of the spatial coordinates of multiple intersection points can be calculated, and the average value can be used as the coordinates of the target center point. Then, based on the method in Method 1, the target area can be obtained around the target center point.

[0176] In a target environment, light typically enters through a light-transmitting device. For example, in a car driving scenario, the light-transmitting device could be a windshield, such as the front windshield, side windshields, or rear side windows. Light can enter through the front windshield or side windshields.

[0177] For example, in one embodiment, the hand position can be determined by four coordinate points, and the area of ​​the projection region can also be determined by four coordinate points. For example, see [link to relevant documentation]. Figure 16 As shown, by comparing and analyzing the in-vehicle images, the position of the user's hand and the position of the projection area can be obtained. S1 represents the edge range of the hand determined based on the frontal view of the hand, with four pixels (i.e., coordinate points) at the edge, a1, a2, a3, and a4. The four pixels in the projection area are b1, b2, b3, and b4. Connecting b1, a1, b2, a2, b3, a3, b4, and a4 sequentially yields four spatial vectors, pointing from the projection area side to the hand position side. To ensure clarity, the spatial vectors are represented by dashed lines, and their directions are not shown in the figure. Extending the spatial vectors away from the projection area or towards the windshield, each spatial vector intersects the windshield, resulting in at least one intersection point. Using the plane or point formed by these at least one intersection point as the end face or endpoint, and the hand position as the base, a spatial polyhedron is formed. The cross-section of this spatial polyhedron determines the sunshade position.

[0178] For example, assume that the coordinates of a1, a2, a3 and a4 are (q1, w1, e1), (q2, w2, e2), (q3, w3, e3), and (q4, w4, e4) respectively, and the coordinates of the four pixel points b1, b2, b3 and b4 of the hand projection are (j1, k1, l1), (j2, k2, l2), (j3, k3, l3), and (j4, k4, l4) respectively. Linking these two regions yields the spatial vectors (j1q1,k1w1,l1e1)(j2q2,k2w2,l2e2)(j3q3,k3w3,l3e3)(j4q4,k4w4,l4e4), and a reverse extension line is constructed. This reverse extension line intersects the windshield at (b1n1m1)(b2n2m2)(b3n3m3)(b4n4m4). In the implementation of sun shading through the windshield, a quadrilateral can be determined based on (b1n1m1)(b2n2m2)(b3n3m3)(b4n4m4), with (b1n1m1)(b2n2m2)(b3n3m3)(b4n4m4) as its endpoints.

[0179] In the implementation of sun shading via a sunshade, any cross-section of the spatial hexahedron (j1,k1,l1)(j2,k2,l2)(j3,k3,l3)(j4,k4,l4)(b1,n1,m1)(b2,n2,m2)(b3,n3,m3)(b4,n4,m4) is the area where sunlight should be blocked; that is, any cross-section of the spatial hexahedron can be used as a sunshade location.

[0180] In some practical situations, if a single camera is used, it may be unable to capture the corresponding shadow area. In this case, it can be assumed that a shadow is constructed facing the camera, that is, a three-dimensional planar region (b1n1m1)(b2n2m2)(b3n3m3)(b4n4m4) is constructed on the windshield, with the midpoint being the position of the camera (c,v,h), to obtain a spatial hexahedron (j1,k1,l1)(j2,k2,l2)(j3,k3,l3)(j4,k4,l4)(b1,n1,m1)(b2,n2,m2)(b3,n3,m3)(b4,n4,m4).

[0181] In one embodiment, the cross-section of a spatial polyhedron, such as the aforementioned spatial hexahedron, can be a horizontal cross-section. A horizontal cross-section can reduce obstruction of the view of occupants inside the vehicle. In another embodiment, the cross-section of the spatial polyhedron is a second plane parallel to the plane of the palm.

[0182] The above-mentioned method for determining the shading location relies on the accurate identification of the projection area. However, as mentioned earlier, there may be situations where the projection area cannot be identified. Therefore, this application also proposes a method for determining the shading location, as follows:

[0183] Based on the user's current hand posture, a first plane is determined where the palm is located. A second plane, parallel to the first plane, is determined along the normal direction of the first plane, situated between the light-transmitting device and the user's hand position. This second plane is then designated as the sunshade position. In other words, after determining the hand position, there's no need to extend the projection area backwards. Instead, the sunshade position is determined directly between the light-transmitting device (e.g., the windshield) and the hand position, along the normal direction of the first plane where the palm is located. Essentially, a virtual second plane is constructed, parallel to the first plane and located between the hand position and the windshield. In one embodiment, this second plane is positioned close to the windshield.

[0184] The above description only uses the front seats in the car as an example. Correspondingly, for the rear seats in the car, the sun visor can be set between the rear left or rear right windshield and the user's hand position.

[0185] In other embodiments, the sunshade position can also be determined based on user gestures. For example, it can be further recognized whether the user has made a gesture pointing to a certain position, and the sunshade can be moved to the position specified by the user.

[0186] Once the shading position is determined, in one embodiment, the shading panel is controlled to move to the shading position to provide shade.

[0187] In another embodiment, when the windshield is tinted glass, the glass color at the sunshade location is controlled to darken to replace the sunshade panel for sun shading.

[0188] This application also provides a method for adjusting the shading intensity after triggering a shading operation. As mentioned above, in one embodiment, the sunshade includes tinted glass for adjusting the shading intensity by controlling the color depth of the tinted glass; and for adjusting the shading area by controlling the size of the color-changing area of ​​the tinted glass.

[0189] After the sunshade is triggered, the duration of the user's hovering position can be detected. Based on the length of time the user's hand remains hovering, the color depth of the tinted glass in the sunshade can be controlled. The longer the user stays, the deeper the color of the tinted glass becomes, further reducing light transmittance and increasing the light-blocking rate. Furthermore, in another embodiment, the control of shading intensity can also be achieved based on the stacking of multiple layers of tinted glass. The longer the user stays, the more layers of tinted glass can be used to control the light-blocking rate.

[0190] If the system detects that the user's hand is still hovering and that the user's hand position has moved compared to before the sunshade operation was triggered, it means that the user has more areas that need to be shaded. In this case, the color-changing area is increased to increase the sunshade area. Specifically, this can be achieved by increasing the number of color-changing units in the color-changing glass of the sunshade.

[0191] This application also provides an electronic device, including: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any of the above embodiments.

[0192] This application also provides an electronic system comprising an electronic device and a sun visor as described in any of the above embodiments; the electronic device is used to control the sun visor to perform a sun shading operation. See also... Figure 10 As shown, the electronic system also includes at least one camera for acquiring images of the target environment. The at least one camera includes: at least one RGB camera, and / or, at least one full-resolution depth-to-time (ToF) camera.

[0193] Current smart cockpits typically feature a full-vehicle camera system positioned in the front center of the vehicle. These cameras are generally image-capturing cameras, providing a good view of the interior in most scenarios. They capture real-time images of the in-vehicle environment, which are then transmitted to the onboard chip for analysis, including the user's hand position and posture. Ideally, multi-angle cameras can better capture in-vehicle image information, avoiding the problem of a single camera's view being obstructed in special circumstances. For example, combining infrared and depth cameras can better and more directly acquire 3D information, reducing the computational burden on the onboard chip.

[0194] For example, when the target environment requiring sun shading is the interior of a motor vehicle, at least one camera is installed in at least one of the following locations: front reading lights, front armrest, driver's steering wheel, passenger screen, central control screen, driver's side door, passenger side door, top of driver's seat, top of passenger seat, B-pillar, A-pillar; rear left door, rear right door, B-pillar, C-pillar, rear armrest, rear left seat, rear right seat. Ideally, cameras should be installed in multiple locations inside the vehicle to obtain information from multiple angles. Hardware support for cameras can reduce the computational load on electronic devices; the more accurate the recognition results are, the better the number and type of cameras are configured. Increasing the number of cameras reduces computational load but increases hardware costs; a trade-off must be struck between hardware cost and software computational load.

[0195] After determining the location where shading needs to be triggered using the above method and constructing the shading position, the system uses photochromic glass, a sunshade with mechanical devices, or other shading devices to achieve the shading effect at the target location. When using photochromic glass for shading, after triggering the shading, the user can also increase the shading area and depth by hovering or moving the device.

[0196] In summary, based on the sunshade operation triggering method provided in this application embodiment, in application scenarios such as vehicle driving, the user's shading intention is determined by a comprehensive judgment of hand posture from multiple dimensions, automatically identifying whether the user has a sunshade need, allowing the user to complete the shading of light at a specific location by using gestures to instruct the system, and effectively solving the problems of related technologies being unable to identify the user's shading needs for non-eye areas, as well as the inability to adjust the shading area and shading depth and the lack of interaction, providing users with a more intelligent sunshade solution.

[0197] A feasible product hardware architecture of the electronic device provided in this application embodiment is shown below. Figure 17 As shown, the hardware architecture of the electronic device 17 may include:

[0198] At least one data transmission interface 171 is used to connect to the sun visor and at least one camera.

[0199] Processor 172 includes one or more processing units. For example, processor 172 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), etc. Different processing units may be independent devices or integrated into one or more processors. Processor 172 may include one or more interfaces, which may be inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, general-purpose input / output (GPIO) interfaces, and / or universal serial bus (USB) interfaces, etc. When the interface of processor 12 can achieve data communication with the sun visor or camera, an external data transmission interface 171 may not be required.

[0200] The first memory 173 is used to store the computer instructions for implementing image processing and triggering logic control, as well as image data acquired by the RGB camera or the ToF camera. The first memory 173 can be an external memory independent of the processor 172, or it can be located within the processor 172. For example, in some embodiments, the memory built into the processor 172 can be a cache memory, used to store instructions or data that the processor 172 has just used or is repeatedly using. If the processor 172 needs to use the instruction or data again, it can directly retrieve it from the cache memory. This avoids repeated access and reduces the waiting time of the processor 172.

[0201] The memory can be read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types of dynamic storage devices that can store information and instructions. It can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it can be any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0202] The first power management module 174 receives input from the battery and / or charging management module and supplies power to the processor 172 and the first memory 173. The first power management module 174 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the first power management module 174 may also be located within the processor 172. In other embodiments, the first power management module 174 and the charging management module may be located in the same device. Optionally, the first power management module 174 may be a submodule of the vehicle power management module or an independent power management module adapted to the vehicle power management module.

[0203] It is understood that the structures illustrated in the accompanying drawings of the embodiments of this application do not constitute a specific limitation on the electronic device 17. In other embodiments of this application, the electronic device 17 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may also include a display screen and / or a speaker, as well as audio circuitry. The speaker is connected to the processor via the audio circuitry for interacting with the user via voice to determine whether a sunshade operation is triggered.

[0204] This application also provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any of the above embodiments.

[0205] This application also provides a chip system, including: a communication interface for inputting and / or outputting information; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the methods described in any of the above embodiments.

[0206] It should be understood that in the various embodiments of this application, "first," "second," etc., are only used to refer to different objects and do not imply any other limitation on the objects referred to.

[0207] It should be understood that the term "unit" in the embodiments of this application can be implemented in software and / or hardware, and is not specifically limited thereto. For example, "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.

[0208] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0209] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0210] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0211] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0212] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0213] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for triggering a shading operation, characterized in that, Applied to electronic devices, the method includes: Track the user's hand position to identify target images containing hand images; Based on the target image, it is detected whether the user's hand posture meets the triggering conditions; the triggering conditions include multiple fingers being in a closed state and the user's hand being in a suspended state; If the hand posture meets the triggering conditions, the sunshade operation is triggered.

2. The method as described in claim 1, characterized in that, The triggering condition also includes the finger being in an extended position.

3. The method as described in claim 2, characterized in that, The detection of whether the user's hand gesture meets the triggering conditions includes: Detect the angle between two adjacent fingers; Based on the included angle, it is determined whether the user's multiple fingers are in a closed state.

4. The method as described in claim 3, characterized in that, The detection of the included angle between two adjacent fingers includes: Detect multiple key points on the user's hand, the multiple key points corresponding to multiple bone nodes of the hand; Among multiple key points corresponding to the same finger, connect two adjacent key points to determine the line segments corresponding to each finger; The included angle between the line segments corresponding to two adjacent fingers is determined as the included angle between the two adjacent fingers.

5. The method as described in claim 2, characterized in that, The detection of whether the user's hand gesture meets the triggering conditions includes: Edge detection is performed on the target image to determine the edge of the user's hand and the first length corresponding to the edge; Based at least on the first length, it is determined whether the user's multiple fingers are in a closed state.

6. The method as described in claim 5, characterized in that, Before determining whether the user's multiple fingers are in a closed state based at least on the first length, the method further includes: Determine a first reference length corresponding to the edge of the user's hand when the user's multiple fingers are in a closed state; The step of determining whether the user's multiple fingers are in a closed state, based at least on the first length, includes: By comparing the first length with the first reference length, and based on the comparison result, it is determined whether the user's multiple fingers are in a closed state.

7. The method as described in claim 6, characterized in that, Before determining whether the user's multiple fingers are in a closed state based at least on the first length, the method further includes: Determine a second reference length corresponding to the edge of the user's hand when the user's multiple fingers are in an open state; The step of determining whether the user's multiple fingers are in a closed state, based at least on the first length, includes: Compare the first length with the second reference length; If the first length is less than the second reference length, compare the first length with the first reference length; If the comparison result determined based on the first length and the first reference length falls within a predetermined first threshold range, it is determined whether the user's multiple fingers are in a closed state.

8. The method as described in claim 5, characterized in that, Determining the first length corresponding to the edge includes: Determine the third length corresponding to the edge formed by two adjacent fingers in each group; The step of determining whether the user's multiple fingers are in a closed state, based at least on the first length, includes: Based at least on the third length corresponding to each group, it is identified whether the corresponding two adjacent fingers are in a closed state, so as to determine whether the user's multiple fingers are in a closed state.

9. The method as described in claim 5, characterized in that, Determining the first length corresponding to the edge includes: Determine the second length corresponding to the edge of the entire hand; The step of determining whether the user's multiple fingers are in a closed state, based at least on the first length, includes: Based at least on the second length, it is determined whether multiple fingers of the user are in a closed position.

10. The method as described in claim 2, characterized in that, The detection of whether the user's hand gesture meets the triggering conditions includes: Starting from the fingertip and moving towards the base of the finger, identify the various bone nodes of the finger as the first node to the fourth node, where the first node is the fingertip node and the fourth node is the finger base node; The first straight line is determined based on the third and fourth nodes; A second straight line is determined based on the first and second nodes; and / or, a third straight line is determined based on the second and third nodes; If the angle between the first straight line and the second and / or the third straight line falls within a predetermined second threshold range, the finger is determined to be in an extended state.

11. The method as described in claim 10, characterized in that, The step of detecting whether the user's finger is in a straight position also includes: The bone node corresponding to the center of the palm is identified as the fifth node; Based on at least two fourth nodes corresponding to at least two fingers and the fifth node, the first plane where the palm is located is determined; If the angle between at least one of the first, second, and third straight lines and the first plane falls within a predetermined third threshold range, the finger is determined to be in an extended state.

12. The method as described in claim 1, characterized in that, The detection of whether the user's hand gesture meets the triggering conditions includes: Detect whether the user's hand is in a hovering state.

13. The method as described in claim 12, characterized in that, The detection of whether the user's hand is in a hovering state includes: When it is determined that multiple fingers of the user are in a closed state, multiple key points of the user's hand are detected, and the multiple key points correspond to multiple bone nodes of the hand; Based on the hand bone connection information, connect two adjacent key points to determine at least one line segment; Detect the displacement of at least one line segment within a predetermined time period; Based on the displacement, it is determined whether the user's hand is in a hovering state.

14. The method as described in claim 13, characterized in that, The detection of the displacement of the at least one line segment within a predetermined time period includes: Determine multiple frames of images within a predetermined duration; Detect a first line segment in each frame of the multi-frame image, and determine a first pixel and a second pixel; wherein, the first line segment is one of the at least one line segment; the first pixel and the second pixel are the two endpoints of the first line segment; Obtain the first coordinate value of the first pixel and the second coordinate value of the second pixel in each frame image; Calculate the first mean of multiple first coordinate values, and use the pixel with the first mean as the coordinate value as the first reference pixel; and calculate the second mean of multiple second coordinate values, and use the pixel with the second mean as the coordinate value as the second reference pixel; use the first reference pixel and the second reference pixel as two endpoints to determine a reference line segment; Calculate the distance between the first line segment and the reference line segment in each frame image, and use the obtained multiple distances as the displacement of the at least one line segment within a predetermined time period; Determining whether the user's hand is in a hovering state based on the displacement includes: If all of the distances fall within a predetermined fourth threshold range, it is determined that the user's hand is in a hovering state.

15. The method as described in claim 1, characterized in that, The triggering conditions also include: The projection area formed by the user's hand includes skin; The method of detecting whether the user's hand gesture meets the triggering conditions also includes: Detect whether the projected area formed by the user's hand contains skin.

16. The method as described in claim 15, characterized in that, The detection of whether the projected area formed by the user's hand contains skin includes: Based on the user's current hand posture, determine the first plane where the palm is located; Determine the intersection point of the normal to the first plane and the user's body surface; Detect whether a projection is generated within a preset distance range around the intersection point and whether the projection area contains skin.

17. The method as described in claim 1, characterized in that, The triggering conditions also include: The comparison result determined based on the area of ​​the projected region formed by the user's hand and the area of ​​the user's hand falls within a predetermined fifth threshold range; wherein, the area of ​​the user's hand is determined based on the front view of the user's current hand posture; The method of detecting whether the user's hand gesture meets the triggering conditions also includes: Detect the projection area formed by the user's hand and determine the area of ​​the projection area; determine the front view of the user's current hand posture, and determine the area of ​​the user's hand based on the front view; Compare the area of ​​the projection area and the area of ​​the hand to determine whether the comparison result falls within the fifth threshold range.

18. The method as described in claim 1, characterized in that, The triggering conditions also include: The brightness difference between the illuminated area and the backlit area in the target image exceeds a predetermined sixth threshold. The method of detecting whether the user's hand gesture meets the triggering conditions also includes: Identify the illuminated and backlit areas in the target image; Determine whether the brightness difference between the light-receiving area and the backlight area exceeds the sixth threshold.

19. The method as described in claim 1, characterized in that, The triggering of the shading operation includes: Determine the location of the shading.

20. The method as described in claim 19, characterized in that, The method is applied to a target environment, which includes a light-transmitting device. Determining the shading location includes: Detect the edge of the projection area formed by the user's hand, and the edge of the user's hand; From the edge pixels of the projection area, at least three first coordinate points are determined to determine the position of the projection area; From the edge pixels of the hand, at least three second coordinate points are determined to determine the position of the hand; Connect the corresponding first coordinate point and second coordinate point to obtain at least three spatial vectors, wherein the spatial vectors point from the projection area to the hand position; Along the pointing direction of the spatial vector, extend the at least three spatial vectors, and the corresponding at least three extended lines intersect with the light-transmitting device to obtain at least one intersection point; Based on the at least one intersection point and the at least three second coordinate points, a corresponding spatial polyhedron is determined, and any cross-section of the spatial polyhedron is determined as the shading position. The cross-section includes a horizontal cross-section or a cross-section parallel to the first plane where the user's palm is located; or, based on the at least one intersection point, a target area is determined on the light-transmitting device, and the target area is determined as the shading position.

21. The method as described in claim 19, characterized in that, The method is applied to a target environment, which includes a light-transmitting device. Determining the shading location includes: Based on the user's current hand posture, determine the first plane where the palm is located; In the normal direction of the first plane, a second plane parallel to the first plane is determined, and the second plane is located between the light-transmitting device and the user's hand position; The second plane is defined as the shading position.

22. The method according to any one of claims 19-21, characterized in that, The method is implemented based on a sunshade device, which includes a sunshade panel. After determining the location of the shading, the method further includes: Control the sunshade to move to the sunshade position.

23. The method as described in claim 22, characterized in that, The sunshade includes tinted glass for adjusting the shading intensity by controlling the color depth of the tinted glass; After controlling the sunshade to move to the sunshade position, the method further includes: Determine the duration for which the user's hand remains suspended; based on the duration, control the color depth of the tinted glass of the sun visor.

24. The method as described in claim 22, characterized in that, The sunshade includes tinted glass for adjusting the shading area by controlling the size of the tinted area of ​​the tinted glass. After controlling the sunshade to move to the sunshade position, the method further includes: If the user's hand position is detected to have moved compared to before the sunshade operation was triggered, the color-changing area is increased to increase the sunshade area.

25. The method as described in claim 20, characterized in that, The light-transmitting device is a windshield; the sunshade position is located on the windshield, and the windshield is tinted glass, used to adjust the sunshade intensity by controlling the color depth of the tinted glass; After determining the target area as the shading location, the method further includes: Increase the color depth of the target area on the windshield.

26. The method according to any one of claims 1-21, characterized in that, After triggering the shading operation, the method further includes: If the system detects that multiple fingers of the user are not in a closed position, the sunshade operation will be terminated.

27. An electronic device, characterized in that, The electronic device includes: One or more processors; memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-26.

28. An electronic system, characterized in that, The electronic system includes the electronic device and sunshade as described in claim 27; The electronic device is used to control the sunshade to perform sunshade operations.

29. The electronic system as claimed in claim 28, characterized in that, The electronic system also includes at least one camera for capturing images of the target environment.

30. The electronic system as claimed in claim 29, characterized in that, The at least one camera includes: At least one RGB camera, and / or at least one full-resolution ToF camera.

31. The electronic system as claimed in claim 29, characterized in that, The target environment is the interior environment of a motor vehicle, and the at least one camera is installed in at least one of the following locations: Front reading lights, front armrest box, driver's steering wheel, passenger screen, center console screen, driver's side door, passenger side door, top of driver's seat, top of passenger seat, B-pillar, A-pillar; Rear left door, rear right door, B pillar, C pillar, rear armrest box, rear left seat, rear right seat.

32. A storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-26.

33. A chip system, characterized in that, include: A communication interface for inputting and / or outputting data; A processor for executing a computer-executable program, causing a device having the chip system mounted to perform the method as described in any one of claims 1-26.

Images