Interaction control method and electronic equipment

Abstract

The invention discloses an interaction control method and electronic equipment. The interaction control method comprises the following steps: monitoring a sight line parameter of a target user; determining a target application window in a screen area of the electronic equipment and a target interaction area used for interacting with the target application window based on the sight line parameter; inputting operation data of the target user acting on the target interaction area into the target application window; and controlling the target application window and / or the electronic equipment to execute a target operation in response to the operation data.

Description

Technical Field

[0001] This application relates to the field of interactive technology, and in particular to an interactive control method and electronic device. Background Technology

[0002] More and more electronic devices are equipped with two or even more displays, such as dual-screen laptops, with displays on both the B-side and C-side to meet diverse user display needs. However, based on user habits, it is more convenient and reasonable to perform interactive actions such as writing and editing on the C-side. Therefore, current laptops only support touch functionality on the C-side to meet user interaction needs, while the B-side does not support touch functionality. This means that when the input field is on the B-side, users need to perform multiple steps to complete writing or editing interactions, such as using a mouse or the C-side touch functionality, resulting in low interaction efficiency and high resource consumption. Summary of the Invention

[0003] The purpose of this application is to provide an interactive control method and an electronic device.

[0004] In a first aspect, embodiments of this application provide an interactive control method, including: Monitor the gaze parameters of the target user; Based on the gaze parameters, the target application window in the screen area of ​​the electronic device and the target interaction area for interacting with the target application window are determined. Input the operation data of the target user on the target interaction area into the target application window; Control the target application window and / or electronic device to perform a target operation in response to the operation data.

[0005] In one possible implementation, the monitoring of the target user's gaze parameters includes: The device configuration of the electronic device is determined, and the device configuration is characterized at least by the range of the included angle between the first body and the second body of the electronic device that are rotatably connected. The target camera is determined based on the device configuration and the configuration parameters of at least two cameras disposed on the electronic device, wherein at least two of the cameras share a digital signal processor; The target camera is used to monitor changes in the target user's gaze parameters.

[0006] In one possible implementation, determining the target camera based on the device configuration and configuration parameters of at least two cameras disposed on the electronic device includes: The framing parameters of the target part of the target user detected by the first camera and the second camera in the current device configuration are obtained. The first camera is set on the first body, and the second camera is set on the second body or at its rotatable connection with the first body. The first camera and / or the second camera are determined to be the target camera based on the comparison between the framing range of the first camera and the second camera in the current device configuration and the framing parameters.

[0007] In one possible implementation, determining the target camera based on the device configuration and configuration parameters of at least two cameras disposed on the electronic device includes: The positional relationship between the first camera and the second camera in the at least two cameras is determined. The positional relationship can change accordingly with the change of the device form. The first camera is disposed on the first body, and the second camera is disposed on the second body or at the rotational connection between the second body and the first body. Based on the positional relationship between the first camera and the second camera and their respective configuration parameters, the first camera and / or the second camera adapted to the current device configuration are determined as the target camera.

[0008] In one possible implementation, the step of monitoring changes in the gaze parameters of the target user using the target camera includes: The system uses a first camera to monitor changes in the line-of-sight parameters of a target user within a first area and a second camera to monitor changes in the line-of-sight parameters of a target user within a second area. The first and second cameras share a single digital signal processor. When the target user's line of sight is within the first area, the digital signal processor processes only the image data captured by the first camera. When the line of sight moves to the second area, the digital signal processor switches to processing the image data captured by the second camera. or, The system utilizes a first camera and a second camera to collaboratively monitor changes in the gaze parameters of the target user within their respective fields of view. The digital signal processor corresponding to the first camera processes only the image data acquired by the first camera, and the digital signal processor corresponding to the second camera processes only the image data acquired by the second camera.

[0009] In one possible implementation, determining the target application window in the screen area of ​​the electronic device and the target interaction area for interacting with the target application window based on the gaze parameters includes: The target application window is determined based on the target user's gaze focus position and / or gaze displacement trajectory on the screen area; and... The target interaction area is determined based on the change data of the target user's gaze parameters on the screen area. The change data of the gaze parameters includes changes in at least one of the following: gaze focus position, gaze displacement trajectory, gaze dwell time at the focus position, and gaze angle. The target interaction area may be the same as or different from the area where the target application window is located, and is related to the device form of the electronic device.

[0010] In one possible implementation, determining the target interaction area based on the change data of the target user's gaze parameters on the screen area includes at least one of the following: When the change data indicates that the target user's gaze has moved from the first screen area where the target application window is located to the second screen area, the first target area in the second screen area is determined as the target interaction area, wherein the first target area is any one of the entire area, blank area or first window area in the second screen area; When the change data indicates that the target user's gaze focus has moved from the first screen area where the target application window is located to the second screen area, the second target area in the second screen area is determined as the target interaction area based on the device configuration of the electronic device and / or the first user operation acting on the second screen area; or, When the change data indicates that the target user's gaze has moved from the first screen area where the target application window is located to the second screen area, the target application window is moved to the third target area determined based on the second user operation performed on the second screen area. The first screen area and the second screen area are located on the first body and the second body of the electronic device, respectively. The first body and the second body are able to rotate relative to each other so that the first screen area and the second screen area have a target included angle range. The first screen area and the second screen area belong to the same physical screen or belong to different physical screens.

[0011] In one possible implementation, inputting the operation data of the target user on the target interaction area into the target application window includes: When the target interaction area and the target application window are located in different screen areas, the operation data is transmitted to the target application window through the target application interface, or transmitted to the target application window through the display control component; or, In response to the target pen device performing a target interaction operation with the target interaction area, the target pen device is controlled to enter an audio acquisition mode so as to transmit the acquired voice data to the target application window through the data transmission channel between the target pen device and the electronic device; And / or, The control of the target application window and / or electronic device to perform a target operation in response to the operation data includes: When there is a correlation between the operation data and the target application window, control the target application window to perform at least one of the following operations in response to the operation data: updating the display content of the target application window, updating the first display parameter of the target application window, or updating the functional services that the target application window can provide; or, If there is no correlation between the operation data and the target application window, control the electronic device to perform at least one of the following operations in response to the operation data: adjusting the second display parameters of the electronic device's display screen, switching the display mode of the electronic device, or adjusting the operation in the application or processing model running by the electronic device.

[0012] Secondly, embodiments of this application also provide an electronic device, including a first body and a second body rotatably connected, a camera assembly and a processor, a first display screen disposed on the first body, and an input device disposed on the second body, wherein the input device includes at least one of a keyboard, a touchpad and a touch display screen; The camera component is used to monitor the gaze parameters of the target user; The processor is configured to receive the gaze parameters and, based on the gaze parameters, determine a target application window in the screen area of ​​the electronic device and a target interaction area for interacting with the target application window; input the operation data of the target user acting on the target interaction area into the target application window; and control the target application window and / or the electronic device to perform a target operation in response to the operation data.

[0013] In one possible implementation, the camera assembly includes a first camera and a second camera, wherein the first camera is disposed on the first body and the second camera is disposed on the second body or at a rotatable connection point between the second body and the first body; The processor is also used to determine the device form of the electronic device and to determine the target camera based on the device form and the configuration parameters of the camera. The device form is characterized at least by the angle range between the first body and the second body. The first camera and the second camera share a digital signal processor. The target camera is used to monitor changes in the target user's gaze parameters. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart of an interactive control method provided in this application is shown; Figure 2 This paper presents a flowchart illustrating the monitoring of gaze parameters of a target user in an interactive control method provided in this application. Figure 3 This application provides a schematic diagram illustrating a target user viewing an electronic device. Figure 4 This application provides a schematic diagram showing the structure of two target cameras that alternately monitor changes in the gaze parameters of a target user, with the two target cameras sharing a single digital signal processor. Figure 5 This application provides a schematic diagram showing the structure of two target cameras simultaneously monitoring changes in the gaze parameters of a target user, with each of the two target cameras corresponding to a unique digital signal processor. Figure 6 A schematic diagram of the structure of an electronic device provided in this application is shown; Figure 7 This application illustrates the application provided by this application. Figure 6 The flowchart of the electronic device shown. Detailed Implementation

[0016] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0017] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0018] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0019] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0020] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0021] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0022] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0023] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0024] To facilitate understanding of this application, the interactive control method provided in this application will be described in detail below.

[0025] The interactive control method provided in this application is applicable to electronic devices with two or more screens. To facilitate the description and understanding of the technical interactive control method, the following will take an electronic device with a first body and a second body configured with a rotatable connection as an example for detailed explanation.

[0026] As an example, Figure 1 A flowchart of the interactive control method provided in the embodiments of this application is shown, wherein the steps of the interactive control method flowchart specifically include S101-S104.

[0027] S101 monitors the gaze parameters of the target user.

[0028] Optionally, the electronic device can be equipped with multiple cameras, such as one or more on the first body, one or more on the second body, etc. Based on this, to avoid the power consumption of the electronic device being affected by multiple cameras simultaneously collecting the user's gaze parameters, thus failing to meet energy-saving requirements, when monitoring the gaze parameters of a target user, a target camera is first identified. Then, this target camera is used to monitor the target user's gaze parameters, achieving both energy saving and accurate monitoring. The gaze parameters include the user's gaze focus position, gaze duration, gaze position changes, and gaze angle changes.

[0029] Here, the target user is the user that the camera of the electronic device can capture. If the camera of the electronic device can capture multiple users, then the user who meets the preset conditions can be identified as the target user. The preset conditions may include at least one of the following: the user whose face is facing the electronic device, the user whose distance from the electronic device is the shortest, the user whose captured area is the largest among the multiple users, etc.

[0030] It is worth noting that the target camera is variable. For example, the target camera is determined in real time based on the target user's posture relative to the electronic device and the electronic device's own posture, so as to achieve the purpose of monitoring the target user's gaze parameters in an efficient and low-power manner.

[0031] Furthermore, while the above description uses an electronic device comprising a first and second body connected by rotation as an example, it should be understood that the interactive control method of this application is also applicable to multi-screen devices such as flexible foldable screen devices and form factor terminals. Correspondingly, for flexible foldable screen devices, when the device is in a flexible unfolded state, the tracking technology collected when acquiring the target user's gaze parameters can be extended to continuous screen areas to monitor the target user's gaze parameters, thereby enabling cross-screen interactive control according to subsequent steps.

[0032] S102, based on the gaze parameters, determine the target application window in the screen area of ​​the electronic device and the target interaction area for interacting with the target application window.

[0033] After obtaining the target user's gaze parameters over a continuous time period, the gaze parameters are analyzed to determine at least the target user's gaze focus position, the duration of the gaze focus at each position, the displacement trajectory corresponding to the movement of the gaze focus, the angle between the gaze and the screen or horizontal plane, and the changes in the gaze angle.

[0034] As an example, after analyzing the gaze parameters, the target application window in the screen area of ​​the electronic device is determined. For instance, the application window corresponding to the target user's gaze focus position is the target application window, and the window corresponding to the application interface covered by the displacement trajectory corresponding to the gaze focus movement is also the target application window. In this embodiment, the screen area to which the target application window belongs can be a screen that does not support touch functionality, such as the B-side screen of a dual-screen laptop, or a screen that is inconvenient to operate, such as a split screen at an edge of a large-screen device. The target application window is a window that requires interactive input, such as a Word window, a WeChat window, a search input box, or an AI application dialog box.

[0035] As another example, after analyzing the gaze parameters, the target interaction area for interacting with the target application window can be determined. Considering that the interaction area differs for each application—for example, the target interaction area could be the area where the input box of the target application window is located, the area pointed to by the stylus, or any other area such as the default input area, extended input area, or full-screen area on the C-side—if the target application window moves during the continuous monitoring period, the target interaction area could also be the area where the target application window is located after moving. In this case, the target interaction area can be accurately determined based on changes in gaze parameters, such as the duration of continuous gaze focus at each position, the displacement trajectory corresponding to the movement of the gaze focus, or changes in the gaze angle.

[0036] As another example, embodiments of this application can also incorporate artificial intelligence (AI) technology, such as using an AI chip installed in an electronic device to assist users in achieving predictive interaction. For instance, by using an AI chip to analyze a target user's historical operations and gaze trajectory within the current time period, the next target interaction area can be predicted, and the target application window corresponding to the predicted target interaction area can be pre-loaded, thereby effectively improving interaction efficiency.

[0037] Furthermore, embodiments of this application can also incorporate Augmented Reality (AR) / Virtual Reality (VR) technologies. For example, after determining the target interaction area, the target interaction area can be extended to a virtual display space to achieve immersive interaction by combining AR / VR technologies, which greatly improves the user experience.

[0038] S103, input the operation data of the target user on the target interaction area into the target application window.

[0039] After identifying the target application window and the target interaction area, real-time data on the user's actions within the target interaction area is collected. This data can include voice input, string input, trajectory input, or graphic symbol input via a stylus or the user's limbs.

[0040] As an example, when inputting operation data of the target user on the target interaction area into the target application window, it can be determined first whether the target interaction area and the target application window are located in the same screen area.

[0041] When the target interaction area and the target application window are located in the same screen area, the operation data can be directly transmitted to the target application window to respond to the operation data and thus complete the interaction.

[0042] When the target interaction area and the target application window are located in different screen areas—that is, the screen on which the operation data is generated and collected is different from the screen to which the target application window belongs—a transfer method is needed to transfer the operation data to the target application window. This application's embodiments illustrate the following three transfer methods, but those skilled in the art should understand that it is not limited to these three methods.

[0043] First transmission method Operational data is passed through the target application interface to the target application window. The target interactive area corresponds to its own process or application, which is used to construct, display, and control this target interactive area. Simultaneously with or after the generation of this target interactive area, a communication connection is established between the application to which the target interactive area belongs and the target application window, based on the data transmission protocol and interface protocol followed.

[0044] When the target interaction area and the target application window are located in different screen areas, the operation data is passed through to the target application window through the data transmission protocol and interface protocol between the application to which the target interaction area belongs and the target application window.

[0045] The second transmission method The operation data is transmitted to the target application window through the display control component. Considering that the operation data can be string input, trajectory input, or graphic symbol input from a stylus or user's limbs, the display control component can collect the control trajectory formed by the touch operation in real time and identify the control trajectory to obtain the corresponding operation data, such as strings, trajectories, or graphic symbols.

[0046] Subsequently, the display control component, through its included touch chip, transmits the operation data to the target application window via the electronic device's operating system (OS).

[0047] The third transmission method In response to the target pen device performing a target interaction operation with the target interaction area, the target pen device is controlled to enter audio acquisition mode, so as to transmit the acquired voice data to the target application window through the data transmission channel between the target pen device and the electronic device. Considering that the operation data of the target interaction area can be voice input applied to the target interaction area, that is, the target interaction operation between the target user and the target interaction area can be triggered by long-pressing the target interaction area with the target pen device or limbs, which can trigger the voice input function.

[0048] Optionally, the target pen device also has an audio acquisition function. Based on this, after triggering the voice input function, the target pen device is controlled to enter the audio acquisition mode, so as to transmit the acquired voice data to the target application window through the data transmission channel between the target pen device and the electronic device. The target pen device and the electronic device can achieve communication connection via Bluetooth, near-field communication, etc., and the corresponding data transmission channel between the target pen device and the electronic device can be a Bluetooth channel, near-field communication channel, or a masked transmission channel.

[0049] S104, control the target application window and / or electronic device to perform the target operation in response to the operation data.

[0050] After the operation data is input into the target application window, the target application window and / or electronic device are controlled to perform the target operation in response to the operation data. For example, the control parameters of the operation data determine whether to control the target application window to perform the target operation in response to the operation data, control the electronic device to perform the target operation in response to the operation data, or control both the target application window and / or the electronic device to perform the target operation in response to the operation data.

[0051] Optionally, when controlling the target application window and / or electronic device to perform a target operation in response to operation data, each control parameter in the operation data is first identified to determine whether there is a correlation between the operation data and the target application window based on the control parameters. Further, if there is a correlation between the operation data and the target application window, the target application window is controlled to perform at least one of the following operations in response to the operation data: updating the display content of the target application window, updating the first display parameter of the target application window, or updating the functional services that the target application window can provide; if there is no correlation between the operation data and the target application window, the electronic device is controlled to perform at least one of the following operations in response to the operation data: adjusting the second display parameter of the electronic device's display screen, switching the display mode of the electronic device, or adjusting the application or processing model running on the electronic device.

[0052] The associated data refers to data related to the target application window, and includes at least: the display parameters of the target application window, the display content parameters included in the target application window, and the functional parameters of the target application window. Control parameters other than the associated data may include updating or adjusting the electronic device's display parameters (such as brightness, refresh rate, theme color, etc.), display mode (split-screen, screen mirroring, monitor mode, etc.) parameters, and start / stop parameters corresponding to the running application or model.

[0053] When a target user uses an electronic device with two or more screens, the camera installed on the electronic device monitors the target user's gaze parameters. Based on these parameters, the interaction needs of the target user can be conveniently met. This is especially effective for handwriting-style interactions, improving interaction efficiency while avoiding excessive consumption of interaction resources. Furthermore, combining voice input can further enhance the convenience and efficiency of handwriting interaction.

[0054] This application uses an electronic device with a first body and a second body configured with a rotatable connection as an example for illustration. Correspondingly, the electronic device is equipped with devices such as Hall sensors, sliding rheostats or gyroscopes to collect angle data between the first body and the second body in real time, and then determine the relative attitude between the first body and the second body in real time.

[0055] Based on this, we can refer to Figure 2 The flowchart shown illustrates a method for monitoring the gaze parameters of a target user, with specific steps including S201-S203.

[0056] S201, Determine the device configuration of the electronic device, which is characterized at least by the range of the included angle between the first body and the second body that are rotatably connected to the electronic device.

[0057] S202, a target camera is determined based on the device form and configuration parameters of at least two cameras installed in the electronic device, wherein at least two of the cameras share a digital signal processor.

[0058] S203 uses the target camera to monitor changes in the target user's gaze parameters.

[0059] For example, the device form of the electronic device is determined based on the angle data between the first and second bodies collected in real time. That is, the device form is characterized at least by the range of the included angle between the first and second bodies connected by the rotation of the electronic device. For example, when the included angle between the first and second bodies is 60 to 145 degrees, the device form of the electronic device is a laptop; when the included angle between the first and second bodies is 180 degrees, the device form of the electronic device is a book; when the included angle between the first and second bodies is 225 to 330 degrees, the device form of the electronic device is a tent; when the included angle between the first and second bodies is 360 degrees, the device form of the electronic device is a tablet, etc.

[0060] After determining the device form factor of the electronic device, a target camera is identified based on the device form factor and the configuration parameters of at least two cameras installed on the electronic device. This target camera is then used to monitor changes in the gaze parameters of the target user. The camera configuration parameters may include the camera's field of view (FOV), its placement within the electronic device, resolution, and sampling rate.

[0061] Here, there can be one or more target cameras. For example, if there is only one target camera, multiple cameras placed in different locations on the electronic device can take turns as target cameras to alternately monitor the target user's gaze parameters, reducing the waste of the camera's operating resources and data processing resources; or if there are multiple target cameras, multiple target cameras can simultaneously monitor the target user's gaze parameters to ensure the integrity and accuracy of the data.

[0062] As an example, at least two cameras in this application embodiment share a single digital signal processor. Correspondingly, when there are two or more target cameras, image processing can be performed using the shared digital signal processor (DSP). Specifically, the raw image data can be processed by relying on the DSP in a time-sharing manner to save costs and power consumption.

[0063] As another example, when there are multiple cameras, a unique DSP can be set for each camera, that is, each DSP only processes the image data captured by the corresponding camera to ensure image processing efficiency.

[0064] As another example, image acquisition signals can also be processed using other processors such as AI chips or embedded controllers built into electronic devices, without being limited to DSPs.

[0065] like Figure 3 The illustration shown is of a target user viewing an electronic device. (Refer to...) Figure 3 In the example shown, the electronic device has two cameras. The first camera, U, is located on the top edge of the first body away from the second body, and the second camera, M, is located on the edge of the second body closer to the first body. Alternatively, the second camera (camera M) can be mounted on a pivot axis that enables the rotational connection between the first and second bodies. During the transition of the target user from a first posture to a second posture, it can be determined that camera U is the target camera when the target user is in the first posture, and camera M is the target camera when the target user is in the second posture; it can also be determined that both camera U and camera M are target cameras during the periods when the target user is in the first posture, in the second posture, and during the transition from the first posture to the second posture.

[0066] It is worth noting that, Figure 3 This is merely an example; the first body may also be equipped with cameras other than camera U, and / or the second body may be equipped with cameras other than camera M, and / or other cameras may be installed on the rotating shaft that enables the rotatable connection between the first and second bodies. This application does not impose specific limitations on these aspects.

[0067] Optionally, embodiments of this application illustrate the following two methods for determining a target camera based on device form factor and configuration parameters of at least two cameras disposed on an electronic device.

[0068] First determination method by Figure 3 The example shown illustrates an electronic device equipped with two cameras: a first camera and a second camera. Based on this, when determining the target camera, the framing parameters of the target user's facial features detected by at least the first and second cameras in the current device configuration are obtained. These framing parameters may include the target user's face angle, gaze angle, and facial region range, among other things.

[0069] After obtaining the above framing parameters, the framing range of the first camera in the current device configuration is compared with the framing parameters detected by the first camera, and the framing range of the second camera in the current device configuration is compared with the framing parameters detected by the second camera, and the comparison results are obtained. Here, the framing range is the spatial area that the camera can capture. Different cameras have different framing ranges, and the same camera also has different framing ranges in different postures.

[0070] Based on the comparison results described above, the first camera and / or the second camera are selected as the target cameras. For example, the comparison results characterize the range of the target user's gaze that the camera can detect within its field of view. Therefore, the camera with the larger range of the target user's gaze is selected as the target camera to ensure that the target user's gaze parameters are collected more completely and accurately. That is, the decision to use the first camera or the second camera for monitoring is based on the field of view of each camera and the current angle of the captured face; the aim is to capture as many frontal images of the face as possible for better gaze tracking.

[0071] It is worth noting that the target camera can be repositioned when the target user's posture changes.

[0072] Second determination method Continue to refer to Figure 3 In the example shown, when determining the target camera, the positional relationship between the first and second cameras of at least two cameras is first determined. Here, the framing range of the first and second cameras can be determined based on the positional relationship.

[0073] Since the first body and the second body are rotatably connected, and the first camera is located on the first body and the second camera is located on the second body or at its rotatable connection with the first body, such as a pivot, the positional relationship between the first camera and the second camera can change accordingly with changes in the device's form.

[0074] For example, if the second camera is mounted on the hinge housing, and the device shape changes, i.e. the hinge rotates, the field of view of the second camera changes accordingly, and the positional relationship between the first camera and the second camera also changes accordingly.

[0075] In addition, when the second camera is set on the second body or the support structure of the second body, the second camera can be configured to change its field of view as the device shape changes, thereby ensuring that the positional relationship between the first camera and the second camera changes accordingly.

[0076] The positional relationship between the first and second cameras determines whether their fields of view overlap. After determining the positional relationship between the first and second cameras (at least two cameras), the first and / or second camera is selected as the target camera based on their positional relationship and respective configuration parameters.

[0077] Considering that different device forms correspond to different usage scenarios, such as scenarios requiring high sampling rates or higher resolution images, a target camera can be selected based on the specific usage scenario. That is, when determining the target camera, the positional relationship and configuration parameters of the first and second cameras are used to determine the first and / or second camera adapted to the current device form.

[0078] In addition, considering that electronic devices have at least two cameras, to avoid unnecessary power consumption, the following operating strategies can be implemented for the cameras: 1. When the target user's gaze continuously falls on a single screen area, only the camera corresponding to that screen area should be activated to monitor the target user's gaze parameters, while other cameras should be turned off. Furthermore, when each camera corresponds to a digital signal processor, only the digital signal processor corresponding to the activated camera should be run, while the digital signal processors corresponding to the deactivated cameras should not be run, thereby reducing power consumption. 2. During the target user's gaze shift, such as from screen area A to screen area B, a time-sharing processing mode can be adopted. That is, when the focus falls on screen area A, the camera corresponding to screen area A should be activated to monitor the target user's gaze parameters. When the focus shifts from screen area A to screen area B, the camera corresponding to screen area A should be turned off while the camera corresponding to screen area B should be activated. This utilizes the camera corresponding to screen area B to monitor the target user's gaze parameters, thereby avoiding high power consumption and resource waste caused by two cameras operating at high load and collecting the same gaze parameters simultaneously.

[0079] It is worth noting that, in order to ensure the accuracy of the collected line-of-sight parameters, environmental data collected by the ambient light sensor can also be obtained in real time. By combining the environmental data and / or the usage scenario, the camera sampling rate and resolution can be dynamically adjusted, thereby achieving a balance between energy saving and performance while ensuring the accuracy of the parameters.

[0080] As an example, when determining that target cameras alternately monitor changes in the gaze parameters of a target user, and at least two target cameras share a single digital signal processor, the first camera can be used to monitor changes in the gaze parameters of the target user within a first area, and the second camera can be used to monitor changes in the gaze parameters of the target user within a second area. The first and second cameras share a single digital signal processor. When the target user's gaze is within the first area, the digital signal processor processes only the image data acquired by the first camera; when the gaze moves to the second area, the digital signal processor switches to processing the image data acquired by the second camera. Figure 4The diagram shows a structure in which two target cameras alternately monitor changes in the gaze parameters of a target user, and the two target cameras share a single digital signal processor.

[0081] In another example, when it is determined that two target cameras are simultaneously monitoring changes in the gaze parameters of a target user, and each target camera has a unique digital signal processor, the first and second cameras are used to collaboratively monitor changes in the gaze parameters of the target user within their respective fields of view. The digital signal processor corresponding to the first camera processes only the image data acquired by the first camera, and the digital signal processor corresponding to the second camera processes only the image data acquired by the second camera. Figure 5 The diagram shows a structure where two target cameras simultaneously monitor changes in the gaze parameters of a target user, and each target camera has a unique digital signal processor.

[0082] As an example, when determining the target application window in a screen area of ​​an electronic device based on gaze parameters, the target application window can be determined based on the target user's gaze focus position and / or gaze displacement trajectory on the screen area. For example, if the gaze focus position falls on an application window for a preset duration, then the application window is determined to be the target application window; if the gaze displacement trajectory represents the selection of an application window, or if the start and end points of the gaze displacement trajectory both fall on the same application window, then the application window is determined to be the target application window; if the gaze focus forms a specific graphic trajectory within an application window, then the application window is determined to be the target application window, and so on.

[0083] Another example is determining the order of application windows in the display hierarchy of the application currently running on the electronic device, identifying the application window at the top of the display hierarchy as the target application window; it is also possible to capture the current display interface of the electronic device, identify and analyze the display interface, and identify the application window that is not covered and is fully displayed as the target application window, etc.

[0084] As an example, when determining the target interaction area for interacting with the target application window based on gaze parameters, the target interaction area can be determined based on the change data of the gaze parameters of the target user on the screen area. The change data of gaze parameters includes changes in at least one of the following: gaze focus position, gaze displacement trajectory, gaze dwell time at the focus position, and gaze angle.

[0085] Specifically, the endpoint position or specific displacement trajectory is determined based on changes in the focus of the line of sight, such as changes in direction, speed, and duration, thereby determining the target interaction area.

[0086] When determining the target interaction area based on the position of the user's gaze focus, the desired interaction area, i.e., the target interaction area, is determined by the change in the position of the user's gaze focus. For example, if the change data indicates that the user's gaze focus has moved from the first screen area where the target application window is located to the second screen area, such as from the B side of a laptop to the C side, then the first target area in the second screen area is determined as the target interaction area.

[0087] The first target area can be any one of the following: the entire area of ​​the second screen area, a blank area, or the first window area. For example, when the current scene is a global writing scene, the entire area of ​​the second screen area can be used as the first target area; if there is a blank area in the second screen area in the current scene, the input box of the target application window can be mapped to the blank area for input; if there is a specified area in the second screen area in the current scene that serves as an input window, input can be directly applied to the corresponding target application window based on the specified area without moving the target application window in the first screen area, etc.

[0088] When determining the target interaction area based on the gaze displacement trajectory, the trajectory change is determined based on the change in the direction of the gaze focus, and then the target interaction area is determined. For example, in a flexible screen device or dual-screen device in the Book form, the gaze displacement trajectory represents the movement of the gaze focus from area A to area B, and the area corresponding to area B is determined as the target interaction area.

[0089] When determining the target interaction area based on the duration of gaze at the focal position, the duration of continuous gaze at each position is recorded, that is, the duration of gaze at each focal position. The focal position with the longest duration of gaze is determined as the endpoint of the gaze displacement trajectory, and then the area corresponding to the endpoint is determined as the target interaction area.

[0090] When determining the target interaction area based on the viewing angle, a fixed angle can be predetermined, and the viewing area corresponding to this fixed angle can be identified as the target interaction area. This fixed angle can be determined according to different user operation habits. In specific implementation, when the viewing angle is detected to change from an inclined angle (based on the angle formed by looking at the target application window) to a fixed angle such as a vertical angle or a specific angle, the viewing area corresponding to this fixed angle is identified as the target interaction area.

[0091] It is worth noting that the target interaction area may be the same as or different from the area where the target application window is located. If the target interaction area is different from the area where the target application window is located, for example, if the target interaction area is located on the screen of the second entity, while the area where the target application window is located is located on the screen of the first entity, then the target application window can be moved. In this case, the target interaction area is the area where the user expects the target application window to be moved; that is, the target application window is moved into the target interaction area.

[0092] To facilitate interactive operations for the target user, the target interaction area can be set to the full-screen area in global writing mode, allowing the target user to perform writing interactions without needing to focus on any specific area, making the operation more convenient. Alternatively, an area can be pre-defined, and a positional mapping relationship can be established between this specified area and the target application window, directly defining the specified area as the target interaction area for interaction with the target application window. Considering that this embodiment also incorporates audio interaction based on the target pen device, the area pointed to by the target pen device when collecting audio information can also be set as the target interaction area, etc.

[0093] Furthermore, the target interaction area can also be related to the device form factor of the electronic device. When the changing data indicates that the target user's gaze has moved from the first screen area where the target application window is located to the second screen area, the second target area in the second screen area is determined as the target interaction area based on the device form factor of the electronic device and / or the first user operation acting on the second screen area.

[0094] For example, if the electronic device is in tent form, a mirrored window corresponding to the target application window in the first screen area can be generated in the second screen area, and the second target area corresponding to the mirrored window can be determined as the target interaction area; if the electronic device is in tablet form, the second target area of ​​the target application window in the second screen area can be directly determined as the target interaction area; if the electronic device is in book form, any second target area in the second screen area can be determined as the target interaction area, as long as it is convenient for the user to interact with the electronic device based on the target interaction area.

[0095] In the case of an electronic device in Book form, where any second target area within the second screen region can be designated as the target interaction area, the presence of a first user operation, such as double-clicking or selection, can be monitored in real time. If such an operation is detected, the target interaction area is determined based on the detected first user operation. For example, if the data indicates that the target user's gaze has moved from the first screen region where the target application window is located to the second screen region, and a double-click operation is detected within the second screen region, then the area corresponding to that double-click operation is determined as the target interaction area.

[0096] Of course, when the electronic device is in any form, the target user can perform the first user operation to determine the target interaction area, and this application embodiment does not limit this.

[0097] When the changing data indicates that the target user's gaze has moved from the first screen area where the target application window is located to the second screen area, the target application window is moved to a third target area determined based on a second user action performed on the second screen area. This third target area is the target interaction area. The second user action can be a pre-defined specific operation, such as a double-click operation performed by the target pen device / user's limb / mouse within the second screen area, or the formation of a preset graphic operation.

[0098] It should be noted that the first screen area and the second screen area are located on the first body and the second body of the electronic device, respectively. Since the first body and the second body are rotatably connected, that is, the first body and the second body can rotate relative to each other, correspondingly, during the relative rotation of the first body and the second body, the first screen area and the second screen area can have a target angle range. The target angle range can be any angle range from 0 to 360°, such as 0 to 180°, 0 to 360°, etc., which is determined according to the connection structure of the electronic device, the usage scenario, etc.

[0099] Furthermore, considering that electronic devices can be equipped with flexible folding screens, meaning that flexible folding screens can also deform, resulting in different postures of different areas of the flexible folding screen relative to the target user, the first screen area and the second screen area can belong to the same physical screen, such as a flexible folding screen, or the first screen area and the second screen area can belong to different physical screens, such as dual-screen devices or multi-screen devices.

[0100] The interaction method described in this application can effectively improve the convenience of users interacting with dual-screen / multi-screen electronic devices, such as writing or modifying, resulting in a better user experience. At the same time, it also improves interaction efficiency and avoids excessive consumption of interaction resources.

[0101] This application also provides an electronic device. Figure 6 A schematic diagram of the electronic device is shown, for reference. Figure 6 The system includes a first body and a second body that are rotatably connected, a camera assembly and a processor, a first display screen disposed on the first body, and an input device disposed on the second body. The input device includes at least one of a keyboard, a touchpad, and a touch screen. It should be noted that the input device is not shown in the diagram. Figure 6 In this application, the electronic device provided is a dual-screen electronic device or a multi-screen electronic device. Figure 6 The illustrated electronic device is one example, namely a dual-screen electronic device. Based on this, the camera assembly of the electronic device includes a first camera and a second camera, which share a digital signal processor. The first camera is disposed on a first body, and the second camera is disposed on a second body or at its rotatable connection to the first body. Figure 6 The second camera shown is located at the rotatable connection between the second body and the first body, which can meet the requirements of the thinner and lighter design of electronic devices to a certain extent.

[0102] The processor is typically located inside the first or second body. Figure 6 The example shown illustrates the processor being configured within the second body. In this embodiment, the processor is further configured to determine the device configuration of the electronic device and, based on the device configuration and camera configuration parameters, determine a target camera to monitor changes in the target user's gaze parameters using the target camera. The device configuration is characterized at least by the angular range between the first and second bodies, and the first and second cameras share a single digital signal processor. In this embodiment, the vertical viewing angle range is set to 0~50°, where a smaller angle results in higher accuracy in identifying the eye's gaze position at the same resolution.

[0103] In specific implementation, it can be referred to Figure 7 The application shown is Figure 6 The flowchart shown illustrates interactive control of the electronic device. The camera component monitors the gaze parameters of the target user. Specifically, a digital signal processor can determine the target camera based on the device configuration and the configuration parameters of the two cameras installed on the electronic device, and then transmit the gaze parameters collected by the target camera to the processor. Alternatively, other control devices can also be used to determine the target camera.

[0104] The processor receives gaze parameters and performs interactive control based on these parameters. It's worth noting that the entity executing this interactive control can be the processor itself, or an interactive control application connected to the processor. Optionally, during interactive control, the target application window in the screen area of ​​the electronic device and the target interaction area for interacting with the target application window are determined based on the gaze parameters; the operation data of the target user acting on the target interaction area is input to the target application window; and the target application window and / or the electronic device are controlled to perform the target operation in response to the operation data.

[0105] When an electronic device is equipped with a corresponding target pen device, the target pen device is connected to the electronic device for communication, so as to transmit the target interactive operation performed by the target user through the target pen device and / or the collected voice data to the processor, so that the processor can perform interactive control based on the target interactive operation and / or voice data.

[0106] It is worth noting that the processor can obtain the current device state of the electronic device and the configuration parameters of each component set in the electronic device in real time, such as the configuration parameters of the camera, and then perform interactive control based on the current device state of the electronic device and the configuration parameters of the camera.

[0107] When interactive control is performed through an interactive control application, the processor needs to transmit gaze parameters, target interactive operations and / or voice data, the current device configuration of the electronic device, and camera configuration parameters to the interactive control application so that the interactive control application can perform interactive control based on the received data.

[0108] Next, combined Figure 3 , 4 6 and 7 describe a specific interaction example: In scenarios where users interact with the dual-screen laptop, such as editing Word documents, using WeChat, or web scraping, the laptop is equipped with two cameras, namely Camera U and Camera M. Camera U and Camera M share a single digital signal processor; the specific structure is described in [reference needed]. Figure 4 Here, it is also pre-set to only allow interactive operations such as writing and drawing on the C-side screen.

[0109] Under the above limitations, after the dual-screen laptop is powered on and running, cameras U and M are activated simultaneously to collect image data corresponding to their respective framing ranges. Based on the image data collected by cameras U and M, and / or the configuration parameters of cameras U and M, and / or the current device configuration of the dual-screen laptop, the target camera is determined to monitor the gaze parameters of the target user.

[0110] While the target user is viewing the WeChat chat interface on the B-side screen and chatting with WeChat friends, the target user is in... Figure 3 The first posture shown indicates that the target user's line of sight falls within the field of view of camera U. At this point, camera U is identified as the target camera, and camera M can be turned off. Subsequently, camera U is used to monitor the target user's gaze parameters to determine the target application window on the electronic device's screen and the target interaction area for interacting with the target application window. Correspondingly, the target application window is the window corresponding to WeChat, and the target interaction area is the input box corresponding to the WeChat window.

[0111] Similarly, refer to Figure 3 When the target user switches from the first posture to the second posture, the target user's line of sight falls within the field of view of camera M. Correspondingly, camera M is switched to the target camera, and camera U is turned off. Then, camera M is used to monitor the target user's line of sight parameters. The line of sight parameters collected by both camera U and camera M are transmitted to a digital signal processor (DSP), which then transmits the line of sight parameters to the processor.

[0112] After receiving the target user's gaze parameters, the processor analyzes these parameters to obtain data on changes in gaze parameters. This change data indicates that the target user's gaze focus has shifted from the B-side screen to the C-side screen, while maintaining the dual-screen laptop's form factor. Figure 3 In the illustrated configuration, if a double-click operation is detected by the target user on the C-side screen, the WeChat window and its corresponding input box are moved from the B-side screen to the C-side screen to generate the area corresponding to the double-click operation. This allows the target user to perform a writing operation based on this area on the C-side screen, enabling them to use the WeChat application for social interaction.

[0113] Of course, when you want to send a voice message, you can press the area used for writing operations and hold it down to trigger voice input, thereby achieving the purpose of voice recording and sending a voice message.

[0114] Of course, the above is just one of many examples, and the electronic devices described above can meet the interactive control needs of various scenarios.

[0115] Furthermore, although exemplary embodiments have been described herein, their scope includes any and all embodiments based on this application that have equivalent elements, modifications, omissions, combinations (e.g., schemes involving intersections of various embodiments), adaptations, or alterations. Elements in the claims will be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the implementation of this application, which will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered illustrative only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.

[0116] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments may be used by those skilled in the art upon reading the above description. Furthermore, in the above detailed description, various features may be grouped together to simplify the application. This should not be construed as an intention that a disclosed feature not claimed is necessary for any claim. Rather, the subject matter of this application may be less than all the features of a particular disclosed embodiment. Thus, the following claims are incorporated herein by reference as examples or embodiments, wherein each claim is an independent, separate embodiment, and these embodiments are contemplated as being possible in various combinations or arrangements. The scope of this application should be determined by reference to the appended claims and the full scope of their equivalents.

[0117] The foregoing has described in detail several embodiments of this application, but this application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications based on the concept of this application, and all such variations and modifications should fall within the scope of protection claimed in this application.

Claims

1. An interactive control method, comprising: Monitor the gaze parameters of the target user; Based on the gaze parameters, the target application window in the screen area of ​​the electronic device and the target interaction area for interacting with the target application window are determined. Input the operation data of the target user on the target interaction area into the target application window; Control the target application window and / or electronic device to perform a target operation in response to the operation data.

2. The interactive control method according to claim 1, wherein monitoring the gaze parameters of the target user includes: The device configuration of the electronic device is determined, and the device configuration is characterized at least by the range of the included angle between the first body and the second body of the electronic device that are rotatably connected. The target camera is determined based on the device configuration and the configuration parameters of at least two cameras disposed on the electronic device, wherein at least two of the cameras share a digital signal processor; The target camera is used to monitor changes in the target user's gaze parameters.

3. The interactive control method according to claim 2, wherein determining the target camera based on the device configuration and configuration parameters of at least two cameras disposed on the electronic device includes: The framing parameters of the target part of the target user detected by the first camera and the second camera in the current device configuration are obtained. The first camera is set on the first body, and the second camera is set on the second body or at its rotatable connection with the first body. The first camera and / or the second camera are determined to be the target camera based on the comparison between the framing range of the first camera and the second camera in the current device configuration and the framing parameters.

4. The interactive control method according to claim 2, wherein determining the target camera based on the device configuration and configuration parameters of at least two cameras disposed on the electronic device includes: The positional relationship between the first camera and the second camera in the at least two cameras is determined. The positional relationship can change accordingly with the change of the device form. The first camera is disposed on the first body, and the second camera is disposed on the second body or at the rotational connection between the second body and the first body. Based on the positional relationship between the first camera and the second camera and their respective configuration parameters, the first camera and / or the second camera adapted to the current device configuration are determined as the target camera.

5. The interactive control method according to claim 3 or 4, wherein monitoring the change data of the target user's gaze parameters using the target camera includes: The system uses a first camera to monitor changes in the line-of-sight parameters of a target user within a first area and a second camera to monitor changes in the line-of-sight parameters of a target user within a second area. The first and second cameras share a single digital signal processor. When the target user's line of sight is within the first area, the digital signal processor processes only the image data captured by the first camera. When the line of sight moves to the second area, the digital signal processor switches to processing the image data captured by the second camera. or, The system utilizes a first camera and a second camera to collaboratively monitor changes in the gaze parameters of the target user within their respective fields of view. The digital signal processor corresponding to the first camera processes only the image data acquired by the first camera, and the digital signal processor corresponding to the second camera processes only the image data acquired by the second camera.

6. The interactive control method according to claim 1, wherein determining the target application window in the screen area of ​​the electronic device and the target interaction area for interacting with the target application window based on the gaze parameter includes: The target application window is determined based on the target user's gaze focus position and / or gaze displacement trajectory on the screen area. as well as, The target interaction area is determined based on the change data of the target user's gaze parameters on the screen area. The change data of the gaze parameters includes changes in at least one of the following: gaze focus position, gaze displacement trajectory, gaze dwell time at the focus position, and gaze angle. The target interaction area may be the same as or different from the area where the target application window is located, and is related to the device form of the electronic device.

7. The interactive control method according to claim 6, wherein determining the target interactive area based on the change data of the target user's gaze parameters on the screen area includes at least one of the following: When the change data indicates that the target user's gaze has moved from the first screen area where the target application window is located to the second screen area, the first target area in the second screen area is determined as the target interaction area, wherein... The first target area is any one of the entire area of ​​the second screen area, the blank area, or the first window area; When the change data indicates that the target user's gaze focus has moved from the first screen area where the target application window is located to the second screen area, the second target area in the second screen area is determined as the target interaction area based on the device form of the electronic device and / or the first user operation on the second screen area. or, When the change data indicates that the target user's gaze has moved from the first screen area where the target application window is located to the second screen area, the target application window is moved to the third target area determined based on the second user operation performed on the second screen area. The first screen area and the second screen area are located on the first body and the second body of the electronic device, respectively. The first body and the second body are able to rotate relative to each other so that the first screen area and the second screen area have a target included angle range. The first screen area and the second screen area belong to the same physical screen or belong to different physical screens.

8. The interactive control method according to claim 1, wherein inputting the operation data of the target user acting on the target interactive area into the target application window includes: When the target interaction area and the target application window are located in different screen areas, the operation data is transmitted to the target application window through the target application interface, or transmitted to the target application window through the display control component; or, In response to the target pen device performing a target interaction operation with the target interaction area, the target pen device is controlled to enter an audio acquisition mode so as to transmit the acquired voice data to the target application window through the data transmission channel between the target pen device and the electronic device; And / or, The control of the target application window or electronic device to perform a target operation in response to the operation data includes: When there is a correlation between the operation data and the target application window, the target application window is controlled to perform at least one of the following operations in response to the operation data: updating the display content of the target application window, updating the first display parameter of the target application window, or updating the functional services that the target application window can provide; or, If there is no correlation between the operation data and the target application window, control the electronic device to perform at least one of the following operations in response to the operation data: adjusting the second display parameters of the electronic device's display screen, switching the display mode of the electronic device, or adjusting the operation in the application or processing model running by the electronic device.

9. An electronic device comprising a first body and a second body rotatably connected, a camera assembly and a processor, a first display screen disposed on the first body, and an input device disposed on the second body, the input device comprising at least one of a keyboard, a touchpad, and a touch display screen; The camera component is used to monitor the gaze parameters of the target user; The processor is configured to determine, based on the gaze parameters, a target application window in the screen area of ​​the electronic device and a target interaction area for interacting with the target application window; input the operation data of the target user acting on the target interaction area into the target application window; and control the target application window and / or the electronic device to perform a target operation in response to the operation data.

10. The electronic device according to claim 9, wherein the camera assembly includes a first camera and a second camera, the first camera being disposed on the first body, and the second camera being disposed on the second body or at a rotatable connection point between the second body and the first body; The processor is also used to determine the device form of the electronic device and to determine the target camera based on the device form and the configuration parameters of the camera. The device form is characterized at least by the angle range between the first body and the second body. The first camera and the second camera share a digital signal processor. The target camera is used to monitor changes in the target user's gaze parameters.

Images