Vehicle window control method, device, electronic device and storage medium

Abstract

The present disclosure relates to a vehicle window control method, device, electronic device, and storage medium. The method comprises: acquiring a video stream containing a target object within a vehicle; determining first line of sight information of the target object based on a first image in the video stream; detecting second line of sight information of the target object based on a second image in the video stream, if the first line of sight information of the target object meets a preset condition; determining a target window of the target object's attention based on the second line of sight information; and controlling the target window. Embodiments of the present disclosure enable contactless window control, allowing passengers to adjust windows in different positions without changing seats, thereby improving the convenience of window control.

Description

Technical Field

[0001] The present disclosure relates to the field of automobile control technology, and in particular to a method, device, electronic device, and storage medium for controlling a vehicle window. Background Art

[0002] With the continuous advancement of technology, users are increasingly choosing vehicles that offer convenient and comfortable transportation. However, current window control methods are not intelligent and often require occupants to manually adjust the windows, hindering ease of use. This is especially true when passengers are elderly or children, requiring other occupants to assist in adjusting the windows, posing a safety hazard. Summary of the Invention

[0003] The present disclosure proposes a technical solution for controlling vehicle windows.

[0004] According to one aspect of the present disclosure, a method for controlling a vehicle window is provided, comprising: acquiring a video stream containing a target object in a vehicle; determining first line of sight information of the target object based on a first image in the video stream; detecting second line of sight information of the target object based on a second image in the video stream when the first line of sight information of the target object meets a preset condition, the second image being located after the first image in timing; determining a target window that the target object is paying attention to based on the second line of sight information; and controlling the target window.

[0005] In a possible embodiment, determining the target window that the target object is paying attention to based on the second line of sight information includes: determining the gaze area of ​​the target object according to the second line of sight information; and when the gaze area of ​​the target object is a window area, determining the window that the target object is paying attention to as the target window that the target object is paying attention to.

[0006] In one possible embodiment, the detecting the second line of sight information of the target object based on the second image in the video stream includes: detecting the line of sight deflection angle of the target object based on the second image in the video stream as the second line of sight information; or detecting the head posture deflection angle of the target object based on the second image in the video stream as the second line of sight information.

[0007] In one possible embodiment, the detecting the second line of sight information of the target object based on the second image in the video stream includes: detecting the line of sight movement information of the target object based on the second image in the video stream, the detection result of the line of sight movement information including a line of sight deflection angle and a corresponding first confidence level; when the first confidence level is greater than a confidence threshold, using the line of sight deflection angle as the second line of sight information.

[0008] In one possible embodiment, the detecting the second line of sight information of the target object based on the second image in the video stream further includes: when the first confidence is not greater than a confidence threshold, detecting the head posture change information of the target object based on the second image in the video stream, the head posture change information including a head posture deflection angle and a corresponding second confidence; when the second confidence is greater than the confidence threshold, using the head posture deflection angle as the second line of sight information.

[0009] In a possible embodiment, determining the gaze area of ​​the target object based on the second line of sight information includes: determining the gaze area of ​​the target object based on the line of sight deflection angle or the head posture deflection angle, and the correspondence between the preset deflection angle interval and each window in the vehicle.

[0010] In a possible embodiment, when the first line of sight information of the target object meets a preset condition, detecting the second line of sight information of the target object based on the second image in the video stream includes: when the first line of sight information of the target object meets the preset condition, issuing a voice inquiry message for asking the target object whether to adjust the car window; in response to obtaining confirmation information fed back by the target object based on the voice inquiry message, detecting the second line of sight information of the target object based on the second image in the video stream.

[0011] In one possible embodiment, obtaining a video stream containing a target object in a vehicle includes: obtaining a video stream containing the target object from at least one perspective captured by at least one camera installed in the vehicle; when obtaining video streams containing the target object from multiple perspectives in the vehicle captured by multiple cameras installed in the vehicle, determining the target window that the target object is paying attention to includes: detecting the second line of sight information of the target object based on the second image of each video stream in the multiple video streams, and obtaining multiple pieces of second line of sight information corresponding to each of the multiple perspectives; determining the window that the target object is paying attention to based on the multiple pieces of second line of sight information, respectively, to obtain multiple results, and fusing the multiple results to determine the target window that the target object is paying attention to.

[0012] In a possible embodiment, before controlling the target window, the method further includes: detecting the gesture action of the target object and the direction and / or amplitude of the gesture action based on the third image in the video stream; controlling the target window includes: controlling the target window to perform at least one of the following according to the gesture action and the direction and / or amplitude of the gesture action: window lowering, window raising, window opening, and window closing.

[0013] In a possible implementation, before controlling the target vehicle window, the method further includes: generating prompt information for controlling the target vehicle window.

[0014] According to one aspect of the present disclosure, a vehicle window control device is provided, comprising: an image acquisition module for acquiring a video stream containing a target object in a vehicle; a first line of sight information determination module for determining first line of sight information of the target object based on a first image in the video stream; a second line of sight information determination module for detecting second line of sight information of the target object based on a second image in the video stream when the first line of sight information of the target object meets a preset condition, the second image being located after the first image in timing; a target window determination module for determining a target window that the target object is paying attention to based on the second line of sight information; and a target window adjustment module for controlling the target window.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0016] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0017] In an embodiment of the present disclosure, a video stream containing a target object in the vehicle can be obtained, and then the first line of sight information of the target object can be determined based on the first image in the video stream. When the first line of sight information of the target object meets a preset condition, the second line of sight information of the target object can be detected based on the second image in the video stream, and then the target window that the target object is paying attention to can be determined based on the second line of sight information, and finally the target window can be controlled. The window control method provided in the embodiment of the present disclosure can realize contactless window control, and can allow passengers to adjust windows in different positions without changing seats, thereby improving the convenience of window control. In addition, the window control method provided in the embodiment of the present disclosure is highly versatile, and the windows can be controlled to rise and fall by the control method regardless of whether the passengers have the ability to act independently.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0020] Figure 1 A flow chart of a vehicle window control method provided according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A flow chart of a vehicle window control method provided according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A reference schematic diagram of the head posture deflection angle provided according to an embodiment of the present disclosure is shown.

[0023] Figure 4 A reference schematic diagram of gesture actions provided according to an embodiment of the present disclosure is shown.

[0024] Figure 5 A reference schematic diagram of camera setting positions provided according to an embodiment of the present disclosure is shown.

[0025] Figure 6 A block diagram of a vehicle window control device provided according to an embodiment of the present disclosure is shown.

[0026] Figure 7 A block diagram of an electronic device provided according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0028] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0029] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0030] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0031] In the related art, passengers in a vehicle typically raise or lower windows through manual adjustment. This approach presents the following problems: 1. If there are elderly or children among the passengers, the driver typically needs to use a master window control device to help the passengers raise or lower the windows, which can easily distract the driver and increase safety risks. 2. Passengers can usually only adjust the windows next to their seats, and adjusting windows across seats is difficult. For example, a passenger in the front passenger seat cannot adjust the window next to the main driver's seat. In other words, the window control methods in the related art are not only less safe, but also have a low degree of control freedom.

[0032] In view of this, the embodiment of the present disclosure provides a method for controlling a vehicle window, which can obtain a video stream containing a target object in the vehicle, and then determine the first line of sight information of the target object based on the first image in the video stream. When the first line of sight information of the target object meets a preset condition, the second line of sight information of the target object is detected based on the second image in the video stream, and then the target window that the target object is paying attention to is determined based on the second line of sight information, and finally the target window is controlled. The method for controlling a vehicle window provided by the embodiment of the present disclosure can realize contactless vehicle window control, and can allow passengers to adjust the windows in different positions without changing seats, thereby improving the convenience of vehicle window control. In addition, the method for controlling a vehicle window provided by the embodiment of the present disclosure is highly versatile, and the vehicle window can be controlled to be raised or lowered by the control method regardless of whether the passenger has the ability to act independently.

[0033] In one possible implementation, the control method can be executed by an electronic device such as a terminal device, which can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in a memory. For example, the control method can be integrated into an in-vehicle terminal (e.g., a vehicle computer), which can be connected to automobile components such as an instrument panel and a window controller to monitor real-time parameters of the vehicle (e.g., vehicle speed, interior temperature, air conditioning wind direction, etc.). The in-vehicle terminal can be connected to at least one OMS (Occupant Monitoring System) camera and collect video streams through the OMS camera. When it is determined that a window needs to be raised or lowered, the corresponding window can be controlled by the window controller connected thereto.

[0034] See Figure 1 As shown, Figure 1 FIG. 1 is a flow chart showing a method for controlling a vehicle window according to an embodiment of the present disclosure. Figure 1 As shown, the control method includes:

[0035] Step S100, obtaining a video stream containing a target object in the car. The target object may be a target person, for example: the target person may be a person closest to the camera in the video stream (or the person who occupies the largest number of pixels in the video stream), or all people, or people with specific characteristics (for example: age characteristics, posture characteristics, etc.), or people with specific identities identified by face recognition or vehicle position recognition (for example, the driver of the vehicle), and the embodiments of the present disclosure are not limited here. The terminal device may obtain the video stream through a vehicle-mounted camera (such as the above-mentioned OMS camera). The camera may be installed at any position in the vehicle where the target person can be photographed, such as the rearview mirror inside the vehicle, the A-pillar inside the vehicle, the B-pillar inside the vehicle, etc.

[0036] In one possible implementation, step S100 may include obtaining a video stream containing the target object from at least one perspective captured by at least one camera located within the vehicle. For example, the terminal device may also capture multiple video streams using multiple cameras to increase the probability of detecting the occupant. Furthermore, capturing video streams from multiple cameras can also improve the accuracy of identifying the target vehicle window, as described in detail below.

[0037] Step S200, determining the first sight line information of the target object based on the first image in the video stream. Exemplarily, developers can establish a three-dimensional space model based on the interior space, window space, other interior components, etc., for the camera to adjust the external parameters (for example: if there are multiple cameras in the car, then due to their different installation positions, the terminal device can set the calibration value of each camera accordingly, that is, multiple cameras in different positions share the same three-dimensional space model, or share the same world coordinate system, to increase the accuracy of determining the target window). After the camera external parameters are set, the video stream is input into the sight line detection model in the relevant technology to determine the first sight line information corresponding to the first image in the video stream.

[0038] In some embodiments, the first line of sight information can be used to represent the viewing angle and / or head orientation of the user in the three-dimensional space model. For example, it can be represented by the line of sight deflection angle and / or head posture deflection angle in the relevant technology. Exemplarily, the above-mentioned deflection angle may include at least one of the yaw angle, the pitch angle, and the roll angle. In other embodiments, the first line of sight information can be used to identify the line of sight landing point in the three-dimensional space model. For example, the user's line of sight direction information can be obtained by processing the image of the user's eye area, and the user's line of sight can be fitted in the three-dimensional space model to obtain the line of sight landing point position inside the vehicle.

[0039] Combine Figure 3 As shown, Figure 3 FIG2 shows a reference diagram of the head posture deflection angle provided according to an embodiment of the present disclosure. Figure 3 As shown, the yaw angle, pitch angle, and roll angle are used to represent the deflection state of the user's line of sight and head in the above-mentioned three-dimensional spatial model. The yaw angle (or "yaw angle") represents the horizontal orientation of the user's line of sight and head in the above-mentioned three-dimensional spatial model. That is, the terminal device can determine the windows of interest to the target object (such as the driver's seat window, the passenger seat window, and the left and right rear windows) that are approximately on the same horizontal plane by obtaining the yaw angle. The pitch angle (or "pitch angle") represents the vertical orientation state in the above-mentioned three-dimensional spatial model. That is, the terminal device can determine the windows of interest to the target object (such as the sunroof) that are on different horizontal planes by obtaining the head pitch angle. The head roll angle (or "roll angle") represents the vertical orientation state of the head in the above-mentioned three-dimensional spatial model (i.e., perpendicular to the plane formed by the above-mentioned horizontal and vertical directions). The three head posture deflection angles can be arbitrarily combined to improve the representativeness of the first line of sight information, thereby improving the accuracy of determining the target window.

[0040] Step S300, when the first sight line information of the target object meets the preset conditions, the second sight line information of the target object is detected based on the second image in the video stream. The timing of the second image is located after the first image. Exemplarily, the preset conditions may include that the time when the target object pays attention to the camera (that is, the first sight line information is within the preset angle range corresponding to the camera installation position) is greater than the preset time. When the first sight line information meets the preset conditions, the function of controlling the window based on sight is awakened, and then the second sight line information of the target object can be detected to identify the user's specific control intention for the window.

[0041] In combination with actual application scenarios, people in the car can wake up the window control function by looking at the camera for a certain period of time. That is, the above-mentioned first line of sight information is used to wake up or trigger the function of controlling the window based on line of sight, and the above-mentioned second line of sight information is used to determine the window to be controlled and control it, and the parameters of the two can be the same. In one example, step S300 may include: detecting the line of sight deflection angle of the target object based on the second image in the video stream as the second line of sight information, or detecting the head posture deflection angle of the target object based on the second image in the video stream as the second line of sight information. For example: in the above-mentioned three-dimensional space model, there is at least one component area (such as: camera area, which may correspond to a deflection angle interval for comparison with the first line of sight information). The terminal device can determine the target object's gaze area in the three-dimensional space model based on the facial image of the target object in the first image captured by the camera through the line of sight detection model in the relevant technology, for example: calculating the line of sight landing point in the three-dimensional space inside the car through the line of sight or head posture deflection angle, thereby determining the target object's gaze area. If the gaze area overlaps with the camera area in the three-dimensional spatial model (for example, the line of sight or head posture deflection angle is within the deflection angle range corresponding to the camera area), the target person is determined to be paying attention to the camera, and the terminal device then determines that the target person intends to activate the window control function. In one example, a time threshold can also be set, that is, if the target person continues to pay attention to a camera for a period of time that reaches the time threshold, the target person is determined to have activated the window control function, which is not limited in the present embodiment.

[0042] In a possible implementation, step S300 may include: when the first line of sight information of the target object meets a preset condition, issuing a voice inquiry message for inquiring the target object whether to adjust the window. In response to obtaining confirmation information fed back by the target object based on the voice inquiry information, detecting the second line of sight information of the target object based on the second image in the video stream. In an embodiment of the present disclosure, after the first line of sight information corresponding to the target person meets the above-mentioned preset condition, it can be determined that the target person intends to control the window through sight, and the terminal device can further issue a voice inquiry message to further confirm whether the target person has enabled the function of controlling the window, so as to reduce the probability of the target person's misoperation or the corresponding function being mistriggered.

[0043] In a possible embodiment, step S300 may include: detecting the line of sight movement information of the target object based on the second image in the video stream, and the detection result of the line of sight movement information includes the line of sight deflection angle and the corresponding first confidence level. The line of sight movement information represents the change of the line of sight. When the first confidence level is greater than the confidence level threshold, the line of sight deflection angle is used as the second line of sight information. Exemplarily, the above-mentioned first confidence level is used to represent the credibility or reference value of the line of sight deflection angle (for example, the shooting clarity of the target object, the shooting ratio of the target object in the video stream, etc.). The calculation method of the first confidence level is not limited in the embodiment of the present disclosure. For example, the first confidence level may be positively correlated with the degree of deviation between the line of sight deviation angle and the middle angle value in the deviation angle interval. Alternatively, the line of sight movement information of the target object in the second image can be detected by a neural network model, and the line of sight movement information and the corresponding first confidence level can be output by the neural network. The embodiment of the present disclosure can achieve the detection accuracy of the second line of sight information by determining the first confidence level, thereby improving the accuracy of the subsequent determination of the target window.

[0044] In one possible implementation, if a first confidence level exists, step S300 may include: if the first confidence level is not greater than a confidence threshold, detecting head posture change information of the target object based on the second image in the video stream, the head posture change information including a head posture deflection angle and a corresponding second confidence level. The head posture change information indicates a change in head posture. If the second confidence level is greater than the confidence threshold, the head posture deflection angle is used as the second line of sight information. Exemplarily, the second confidence level is used to indicate a reference value for the head posture deflection angle. The present disclosure does not limit the calculation method of the second confidence level. For example, the second confidence level may be positively correlated with the degree of deviation between the head posture deflection angle and the middle angle value in the deflection angle interval. Alternatively, the head posture change information of the target object in the second image may be detected using a neural network model, with the neural network outputting the head posture change information and the corresponding second confidence level. The present disclosure embodiment can comprehensively consider the detection accuracy of the line of sight deflection angle and the head posture deflection angle by determining the second confidence level to achieve the detection accuracy of the second line of sight information, thereby improving the accuracy of subsequent determination of the target vehicle window.

[0045] Continue reading Figure 1 , step S400, determining the target window that the target object is paying attention to based on the second line of sight information. Exemplarily, the target window can be determined by the gaze area, gaze time, etc. corresponding to the second line of sight information, and the embodiments of the present disclosure are not limited thereto. Exemplarily, after the terminal device determines the target window that the target object is paying attention to, a response prompt (such as a voice prompt, etc.) can be generated to prompt the target object which window is the target window, so as to reduce the possibility of misoperation of the target window.

[0046] See Figure 2 As shown, Figure 2 A flow chart of a method for controlling a vehicle window according to an embodiment of the present disclosure is shown. Figure 2As shown, in one possible implementation, step S400 may include: step S410, determining the target object's gaze area based on the second gaze information. In one example, this step may include determining the target object's gaze area based on the gaze deflection angle or the head posture deflection angle, and the correspondence between preset deflection angle intervals and each window in the vehicle. For example, developers may pre-set deflection angle intervals corresponding to each window area based on the location of the window. Then, if the terminal device determines that the gaze deflection angle or head posture deflection angle in the second gaze information does not fall within a deflection angle interval, the target object's gaze area is determined to be a non-window area, and a voice prompt may be generated to remind the user to correct the gaze angle. Alternatively, a three-dimensional model may be created for the vehicle interior space, including the windows. After obtaining the second gaze information, the target object's gaze area may be determined by fitting the landing point of the target object's gaze on the three-dimensional space model after performing a gaze deflection action or a head deflection action. This is not limited in the present disclosed embodiments.

[0047] Step S420: When the gaze area of the target object is the window area, determine the window that the target object gazes at as the target window that the target object is concerned about. When the terminal device determines that the gaze deflection angle and the head pose deflection angle in the second line-of-sight information are within a deflection angle interval, determine the above-mentioned gaze area as the window area, and use the window corresponding to this deflection angle interval as the target window. For example: In the above three-dimensional space model, there is at least one window area. The terminal device can use the line-of-sight detection model in related technologies to determine the area of interest of the target object in this three-dimensional space model (for example: represented by the line of sight and the head pose deflection angle) based on the facial image of the target object in the second image collected by the camera. When the above area of interest coincides with the window area in the three-dimensional space model (for example: the line of sight and the head pose deflection angle are within the deflection angle interval corresponding to the window area), determine the window corresponding to this window area as the target window. The terminal device can also issue a voice reminder to further confirm whether the target person is concerned about this target window. Exemplarily, the above-mentioned preset deflection angle interval can be arbitrarily set by the developer, and the embodiments of the present disclosure do not limit this here. For example: If the yaw angle (that is, the yaw angle) is negative to the left of the vehicle occupants and positive to the right, it can be set as follows: If -60° < yaw < -30°, it means that the target window that the target object is concerned about is the front left window; if 30° < yaw < 60°, it means that the target window that the target object is concerned about is the front right window; if yaw < -60°, it means that the target window that the target object is concerned about is the rear left window; if yaw > 60°, it means that the target window that the target object is concerned about is the rear right window. Exemplarily, the above-mentioned preset angle interval can also be set according to the size of each window. For example: The size of the preset angle interval of the above head yaw angle is positively correlated with the horizontal length of the window.

[0048] In a possible implementation manner, when obtaining video streams of multiple perspectives inside the vehicle containing the target object collected by multiple cameras arranged inside the vehicle, step S400 may include: Detecting the second line-of-sight information of the target object based on the second images of each video stream in the multiple video streams, and obtaining multiple pieces of second line-of-sight information corresponding to the multiple perspectives respectively. Determine the windows that the target object is concerned about based on the multiple pieces of second line-of-sight information respectively, obtain multiple results, and fuse the multiple results to determine the target window that the target object is concerned about.

[0049] Exemplarily, the above fusion may include multiple methods, for example: each result may correspond to an attention score, and the attention score is used to indicate the possibility that the window corresponding to the result in each video stream is the target window. The attention scores corresponding to each window in multiple results may be added together, and the window with the highest attention score may be used as the target window. For another example: each result may correspond to a weight, which is related to the reference value of the second line of sight information in each video stream (for example: clarity, the proportion of the target object in the video stream, etc.), and then the attention scores corresponding to each window are weighted and summed, and the window with the highest attention score may be used as the target window. The embodiments of the present disclosure are not limited here. The embodiments of the present disclosure may capture multi-perspective video streams of the target object through multiple cameras, thereby making the detected target window more representative and more accurate.

[0050] Continue reading Figure 1 , step S500, controlling the target window.

[0051] For example, the target subject can control the target window through voice or gestures. In one example, before step S500, the control method may further include: detecting the target subject's hand gesture and the direction and / or amplitude of the gesture based on a third image in the video stream. In this case, step S500 may include: controlling the target window to perform at least one of the following based on the gesture and the direction and / or amplitude of the gesture: window lowering, window raising, window opening, or window closing. The third image is timed after the first image. For example, a video stream captured by a camera and including the target subject's hand gestures can be input into a hand detection model in the related art. The hand detection model then outputs an image of the hand area in the video stream, which is then input into a gesture detection model in the related art. The gesture detection model then determines whether the target subject's hand gesture is a preset action. The specific content of the preset action is not limited in the embodiments of this disclosure. The terminal device can display the correspondence between the preset action and the window control on a display screen to facilitate window control by the user. For example, the terminal device may control the raising and lowering of the target window according to the direction of the gesture action, and may determine the adjustment amplitude of the target window according to the amplitude of the gesture action.

[0052] See Figure 4 As shown, Figure 4 A reference diagram of a gesture action provided according to an embodiment of the present disclosure is shown. Figure 4 As shown, the occupants of the vehicle can control the degree of opening and closing of the window through the following gestures: the target object keeps moving his hand upward (i.e. Figure 4 The target object keeps moving his hand horizontally downward (i.e. Figure 4The target object controls the hand to move forward (i.e. Figure 4 The target object controls the hand to move backward (i.e. Figure 4 In gesture 4), the terminal device controls the target window to be completely closed.

[0053] In combination with actual application scenarios, refer to Figure 5 As shown, Figure 5 A reference diagram of the camera installation position according to an embodiment of the present disclosure is shown. For example, a vehicle may include at least one camera, which may be installed at Figure 5 Cameras at positions A and B, as well as the rear seating area (not shown), are connected to the aforementioned terminal devices, which then capture video streams from the cameras. For example, if a rear-seat passenger wishes to open the passenger window on a hot day to increase ventilation, they only need to gaze at the camera for three seconds, and the terminal device will send a voice message instructing them on how to proceed. After the passenger gazes at a window, the terminal device determines that the passenger wishes to raise or lower that window. The passenger can then use specific gestures to conveniently and contactlessly control the opening and closing of the passenger window across multiple locations.

[0054] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0055] In addition, the present disclosure also provides a vehicle window control device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any vehicle window control method provided by the present disclosure. The corresponding technical solutions and descriptions can be found in the corresponding records in the method section and will not be repeated here.

[0056] Figure 6 A block diagram of a vehicle window control device according to an embodiment of the present disclosure is shown. Figure 6As shown, the control device 100 includes: an image acquisition module 110, used to acquire a video stream containing a target object in the vehicle; a first line of sight information determination module 120, used to determine the first line of sight information of the target object based on the first image in the video stream; a second line of sight information determination module 130, used to detect the second line of sight information of the target object based on the second image in the video stream when the first line of sight information of the target object meets a preset condition, and the timing of the second image is located after the first image; a target window determination module 140, used to determine the target window that the target object is concerned about based on the second line of sight information; and a target window adjustment module 150, used to control the target window.

[0057] In a possible embodiment, determining the target window that the target object is paying attention to based on the second line of sight information includes: determining the gaze area of ​​the target object according to the second line of sight information; and when the gaze area of ​​the target object is a window area, determining the window that the target object is paying attention to as the target window that the target object is paying attention to.

[0058] In one possible embodiment, the detecting the second line of sight information of the target object based on the second image in the video stream includes: detecting the line of sight deflection angle of the target object based on the second image in the video stream as the second line of sight information; or detecting the head posture deflection angle of the target object based on the second image in the video stream as the second line of sight information.

[0059] In one possible embodiment, the detecting the second line of sight information of the target object based on the second image in the video stream includes: detecting the line of sight movement information of the target object based on the second image in the video stream, the detection result of the line of sight movement information including a line of sight deflection angle and a corresponding first confidence level; when the first confidence level is greater than a confidence threshold, using the line of sight deflection angle as the second line of sight information.

[0060] In one possible embodiment, the detecting the second line of sight information of the target object based on the second image in the video stream further includes: when the first confidence is not greater than a confidence threshold, detecting the head posture change information of the target object based on the second image in the video stream, the head posture change information including a head posture deflection angle and a corresponding second confidence; when the second confidence is greater than the confidence threshold, using the head posture deflection angle as the second line of sight information.

[0061] In a possible embodiment, determining the gaze area of ​​the target object based on the second line of sight information includes: determining the gaze area of ​​the target object based on the line of sight deflection angle or the head posture deflection angle, and the correspondence between the preset deflection angle interval and each window in the vehicle.

[0062] In a possible embodiment, when the first line of sight information of the target object meets a preset condition, detecting the second line of sight information of the target object based on the second image in the video stream includes: when the first line of sight information of the target object meets the preset condition, issuing a voice inquiry message for asking the target object whether to adjust the car window; in response to obtaining confirmation information fed back by the target object based on the voice inquiry message, detecting the second line of sight information of the target object based on the second image in the video stream.

[0063] In one possible embodiment, obtaining a video stream containing a target object in a vehicle includes: obtaining a video stream containing the target object from at least one perspective captured by at least one camera installed in the vehicle; when obtaining video streams containing the target object from multiple perspectives in the vehicle captured by multiple cameras installed in the vehicle, determining the target window that the target object is paying attention to includes: detecting the second line of sight information of the target object based on the second image of each video stream in the multiple video streams, and obtaining multiple pieces of second line of sight information corresponding to each of the multiple perspectives; determining the window that the target object is paying attention to based on the multiple pieces of second line of sight information, respectively, to obtain multiple results, and fusing the multiple results to determine the target window that the target object is paying attention to.

[0064] In one possible embodiment, before controlling the target window, the control device is also used to detect the gesture action of the target object and the direction and / or amplitude of the gesture action based on the third image in the video stream; the control of the target window includes: controlling the target window to perform at least one of the following according to the gesture action and the direction and / or amplitude of the gesture action: window lowering, window raising, window opening, and window closing.

[0065] In a possible implementation manner, before controlling the target vehicle window, the control device is further configured to generate prompt information for controlling the target vehicle window.

[0066] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0067] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0068] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0069] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0070] The electronic device may be provided as a terminal or a device in other forms.

[0071] Figure 7 The block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal device.

[0072] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0073] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0074] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0075] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0076] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0077] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0078] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0079] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0080] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as wireless network (Wi-Fi), second generation mobile communication technology (2G), third generation mobile communication technology (3G), fourth generation mobile communication technology (4G), long term evolution (LTE) of universal mobile communication technology, fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0081] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0082] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0083] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0084] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0085] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0086] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0087] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0089] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0090] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0091] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0092] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0093] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0094] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0095] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for controlling a vehicle window, comprising: Obtain a video stream containing the target object inside the car; determining first sight line information of the target object based on a first image in the video stream; When the first sight line information of the target object meets a preset condition, detecting the second sight line information of the target object based on a second image in the video stream, where the second image is located after the first image in a temporal sequence; determining a target vehicle window that the target object is focusing on based on the second sight line information; controlling the target vehicle window; The detecting, when the first sight line information of the target object satisfies a preset condition, the second sight line information of the target object based on the second image in the video stream includes: When the first sight line information of the target object meets a preset condition, a voice inquiry message is sent to inquire the target object whether to adjust the window; In response to obtaining confirmation information fed back by the target object based on the voice query information, second sight line information of the target object is detected based on a second image in the video stream.

2. The method according to claim 1, wherein The determining the target window of interest to the target object based on the second sight line information includes: determining a gaze area of ​​the target object according to the second sight line information; In a case where the target object's gaze area is a vehicle window area, the vehicle window that the target object is gazing at is determined as the target vehicle window that the target object is paying attention to.

3. The method according to claim 2, wherein: The detecting second sight line information of the target object based on the second image in the video stream includes: detecting a sight line deflection angle of the target object based on a second image in the video stream as the second sight line information; or The head posture deflection angle of the target object is detected based on the second image in the video stream as the second sight line information.

4. The method according to claim 2, wherein: The detecting second sight line information of the target object based on the second image in the video stream includes: Detecting gaze movement information of the target object based on the second image in the video stream, where a detection result of the gaze movement information includes a gaze deflection angle and a corresponding first confidence level; When the first confidence level is greater than a confidence level threshold, the sight line deflection angle is used as the second sight line information.

5. The method according to claim 4, wherein The detecting the second sight line information of the target object based on the second image in the video stream further includes: When the first confidence level is not greater than a confidence threshold, detecting head posture change information of the target object based on a second image in the video stream, the head posture change information including a head posture deflection angle and a corresponding second confidence level; When the second confidence level is greater than the confidence level threshold, the head posture deflection angle is used as the second sight line information.

6. The method according to claim 3 or 5, wherein: The determining the gaze area of ​​the target object according to the second sight line information includes: The gaze area of ​​the target object is determined according to the sight line deflection angle or the head posture deflection angle, and the correspondence between the preset deflection angle interval and each window in the vehicle.

7. The method according to any one of claims 1 to 5, wherein: The obtaining of a video stream containing a target object in the vehicle includes: Acquire a video stream containing a target object from at least one perspective captured by at least one camera disposed in the vehicle; When video streams containing a target object from multiple perspectives in the vehicle are acquired by multiple cameras disposed in the vehicle, determining a target vehicle window that the target object is focused on includes: detecting second sight line information of the target object based on the second image of each video stream in the plurality of video streams, and obtaining a plurality of second sight line information corresponding to each of the plurality of perspectives; The vehicle windows that the target object is paying attention to are respectively determined based on the multiple pieces of second sight line information, a plurality of results are obtained, and the multiple results are fused to determine the target vehicle window that the target object is paying attention to.

8. The method according to any one of claims 1 to 5, wherein: Before controlling the target vehicle window, the method further includes: detecting a gesture of the target object and a direction and / or magnitude of the gesture based on a third image in the video stream; The controlling of the target window includes: controlling the target window to perform at least one of the following according to the gesture action and the direction and / or amplitude of the gesture action: Window down, window up, window open, window closed.

9. The method according to any one of claims 1 to 5, wherein: Before controlling the target vehicle window, the method further includes: Prompt information for controlling the target vehicle window is generated.

10. A vehicle window control device, comprising: An image acquisition module, used to acquire a video stream containing target objects in the vehicle; A first sight line information determining module, configured to determine first sight line information of the target object based on the first image in the video stream; a second sight line information determining module, configured to detect second sight line information of the target object based on a second image in the video stream if the first sight line information of the target object meets a preset condition, the second image being located after the first image in time sequence; a target vehicle window determining module, configured to determine a target vehicle window that the target object is paying attention to based on the second sight line information; a target window adjustment module, for controlling the target window; The second sight line information determining module is further configured to: When the first sight line information of the target object meets a preset condition, a voice inquiry message is sent to inquire the target object whether to adjust the window; In response to obtaining confirmation information fed back by the target object based on the voice query information, second sight line information of the target object is detected based on a second image in the video stream.

11. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the control method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the control method according to any one of claims 1 to 9 when executed by a processor.

Images