Device control method, electronic device and storage medium

Abstract

The present application provides a device control method, an electronic device and a storage medium, and relates to the technical field of terminal devices. The method includes: when the electronic device is in a preset state, a prompt message is displayed, and the prompt message is used to prompt the user to interact with the electronic device in a target control manner. After the electronic device receives a first operation of the user to control the electronic device in a target control manner, the electronic device is controlled to perform a corresponding function in response to the first operation, wherein the preset state is that the duration of the electronic device being in a static state reaches a first preset duration, and the target control method is different from the touch method. As a result, the electronic device recommends that the user interact with the electronic device in a target control manner, which not only improves the interaction efficiency between the user and the electronic device, but also improves the user experience.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on November 3, 2023, with application number 202311465139.9 and invention name “A multi-modal input method based on context perception”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The embodiments of the present application relate to the technical field of terminal devices, and in particular to a device control method, an electronic device, and a storage medium. Background Art

[0003] With the development of electronic devices (such as mobile phones, tablet computers, etc.), the scenarios in which users use electronic devices are increasing, and the use of electronic devices has also brought many conveniences to people's lives.

[0004] At present, when users use electronic devices, there are some scenarios where it is inconvenient to operate the electronic devices when they interact with the electronic devices, such as cooking, washing, exercising, driving, etc., resulting in low efficiency in the interaction between users and electronic devices and poor user experience. Summary of the invention

[0005] The embodiments of the present application provide a device control method, an electronic device, and a storage medium. By recommending a user to interact with the electronic device in a target control manner, not only the interaction efficiency between the user and the electronic device is improved, but also the user experience is improved.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a device control method is provided, which is applied to an electronic device, and the method includes:

[0008] When the electronic device is in a preset state, prompt information is displayed, the prompt information is used to prompt the user to interact with the electronic device in a target control manner, receive a first operation of the user to control the electronic device in the target control manner, and in response to the first operation, control the electronic device to perform a corresponding function.

[0009] The preset state includes that the duration of time that the electronic device is in a static state reaches a first preset duration.

[0010] The above-mentioned target control method is different from the touch method. For example, the touch method includes click (single click, double click or multi-click) or touch, and the target control method can be an air gesture control method, an eye movement control method or a voice control method.

[0011] It can be understood that when the electronic device is in a preset state, the user cannot interact with the electronic device by manually clicking or touching the display screen. The electronic device prompts the user to interact with the electronic device using a target control method, which is conducive to improving the interaction efficiency between the user and the electronic device and improving the user experience.

[0012] In a possible implementation manner of the first aspect, the preset state also includes one or more of the following: the electronic device currently displays an interface, the distance between the electronic device and the user is greater than a distance threshold, or the electronic device is in a non-horizontal placement state.

[0013] The electronic device being in a non-horizontal placement state means that the angle between the electronic device and the horizontal plane is greater than 0°.

[0014] In another possible implementation manner of the first aspect, the target control mode includes a first control mode or a second control mode, the first control mode is different from the second control mode, and when the electronic device is in a preset state, the prompt information is displayed, including:

[0015] When the electronic device meets a first condition, displaying a first prompt message, the first prompt message is used to prompt the user to interact with the electronic device in a first control manner, the first condition including that the electronic device is in a preset state and the volume value of the current environment is greater than a volume threshold;

[0016] When the electronic device meets the second condition, a second prompt message is displayed, and the second prompt message is used to prompt the electronic device to interact with the electronic device in a second control manner. The second condition includes that the electronic device is in a preset state and the volume value of the current environment is less than the volume threshold.

[0017] Exemplarily, when the electronic device is in a high-noise environment, the electronic device prompts to use a first control method (for example, air gesture control method or eye movement control method) to interact with the device; when the electronic device is in a low-noise environment, the electronic device prompts to use a second control method (for example, voice control method) to interact with the device.

[0018] It can be understood that when an electronic device is in a high-noise environment, if the user uses voice control to interact with the device, there will be a lot of environmental noise, causing the microphone of the electronic device to be greatly disturbed by the environmental sound and unable to accurately capture the trigger instructions of the user's voice instructions, resulting in low accuracy in voice interaction between the user and the mobile phone.

[0019] In another possible implementation of the first aspect, the first condition includes that the electronic device is in a preset state, the foreground application is a preset application, and the volume value of the current environment is greater than a volume threshold; the second condition includes that the electronic device is in a preset state, the foreground application is a preset application, and the volume value of the current environment is less than the volume threshold.

[0020] It can be understood that when an electronic device recommends that a user adopt a target control method to interact with the electronic device, the electronic device can combine the foreground application and the volume value of the current environment to comprehensively determine the recommended target control method, thereby avoiding the problem that the foreground application cannot be controlled using the target control method.

[0021] In another possible implementation manner of the first aspect, the second condition further includes that the current position of the electronic device is not a preset position, and the method further includes:

[0022] When the volume value of the current environment is less than the volume threshold, if the current position is not the preset position, the second prompt information is displayed.

[0023] That is, when the electronic device is in a low-noise environment, it can continue to determine whether the current position is a preset position, so as to recommend a more suitable target control method to the user.

[0024] In another possible implementation manner of the first aspect, the second condition further includes that the current time is within a preset range, and the method further includes:

[0025] If the volume value of the current scene is less than the volume threshold, the current position is a preset position, and the current time is not within the preset range, the second prompt information is displayed.

[0026] For example, assuming that the electronic device is in a low-noise environment, the current location is a preset location (for example, home), and the current time is not within a preset range (the preset range is 10:00 pm to 6:00 am), the electronic device prompts to use voice control to interact with the electronic device.

[0027] In another possible implementation manner of the first aspect, the target control mode includes a third control mode and a fourth control mode, the third control mode is different from the fourth control mode, and when the electronic device is in a preset state, the prompt information is displayed, including:

[0028] When the electronic device meets a third condition, displaying a third prompt message, the third prompt message is used to prompt the user to interact with the electronic device in a third control manner, the third condition including that the electronic device is in a preset condition and the foreground application is an application of the first type;

[0029] When the electronic device meets the fourth condition, a fourth prompt message is displayed, where the fourth prompt message is used to prompt the user to interact with the electronic device in a fourth control manner. The fourth condition includes that the electronic device is in a preset condition and the foreground application is an application of the second type.

[0030] It can be understood that some types of foreground applications are suitable for control by air gestures or eye movement control, such as reading applications, while some types of foreground applications are suitable for control by voice control, such as video applications and audio applications. Therefore, the electronic device can prompt the user to use the corresponding control method to interact with the device according to the type of foreground application to improve the user experience.

[0031] In another possible implementation of the first aspect, the device control method may further include:

[0032] Displaying notification information, the notification information is used to prompt to turn on the intelligent control mode; receiving a second operation from the user indicating to turn on the intelligent control mode; and controlling the intelligent control mode to turn on in response to the second operation;

[0033] Displays prompt information, including:

[0034] After the intelligent control mode is turned on, a prompt message is displayed.

[0035] It can be understood that the electronic device can prompt the user to turn on the intelligent control mode to avoid the problem that the user does not know that the electronic device has the intelligent control mode and cannot use the target control method to interact with the device in time.

[0036] In another possible implementation of the first aspect, the device control method may further include:

[0037] When it is determined that the display time of the prompt information reaches the second preset time, and no operation of the user using the target control method to trigger the foreground application is received, the prompt information stops being displayed.

[0038] It can be understood that when the prompt information is displayed in the electronic device for a second preset time, the electronic device stops displaying the prompt information to avoid the problem that the prompt information is displayed for too long, affecting the user experience.

[0039] In another possible implementation of the first aspect, the first control method is an air gesture control method or an eye movement control method, and the second control method is a voice control method.

[0040] In another possible implementation manner of the first aspect, the preset application is a video application, the first control mode is an air gesture control mode or an eye movement control mode, the second control mode is a voice control mode, and when the electronic device meets the first condition, the first prompt information is displayed, including:

[0041] When the electronic device is in a preset state, the foreground application is a video application and the volume value of the current environment is greater than the volume threshold, a first prompt message is displayed, and the first prompt message is used to prompt the user to interact with the electronic device by using an air gesture control method or an eye movement control method;

[0042] When the electronic device meets the second condition, the second prompt information is displayed, including:

[0043] When the electronic device is in a preset state, the foreground application is a video application and the volume value of the current environment is less than the volume threshold, a second prompt message is displayed, and the second prompt message is used to prompt the user to interact with the electronic device using voice control.

[0044] In another possible implementation manner of the first aspect, the first control mode is an air gesture control mode or an eye movement control mode, the second control mode is a voice control mode, and if the current position is not a preset position, displaying the second prompt information includes:

[0045] When the electronic device is in a preset state, the volume value of the current environment is less than the volume threshold and the current location is not a public place, a second prompt message is displayed, and the second prompt message is used to prompt the user to interact with the electronic device using voice control.

[0046] In a second aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the device control method described in any one of the first aspects above.

[0047] In a third aspect, the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the device control method described in any one of the first aspects is implemented.

[0048] In a fourth aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the device control method described in any one of the first aspects.

[0049] It can be understood that the electronic device described in the second aspect, the computer storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 An example diagram of a scenario where it is inconvenient for a user to operate an electronic device;

[0051] Figure 2 An example diagram of a path for activating a voice control function of an electronic device is shown;

[0052] Figure 3 An example diagram of a gesture control function activation path of an electronic device is shown;

[0053] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0054] Figure 5 A software structure diagram of an electronic device provided in an embodiment of the present application;

[0055] Figure 6 An example of prompting the user to turn on the intelligent control function provided in the embodiment of the present application Figure 1 ;

[0056] Figure 7 An example of prompting the user to turn on the intelligent control function provided in the embodiment of the present application Figure 2 ;

[0057] Figure 8 An example diagram of a switch for turning off the intelligent control function provided in an embodiment of the present application;

[0058] Fig. 9 An example of a recommended intelligent control method provided in the embodiment of the present application Figure 1 ;

[0059] Fig.10 An example of a recommended intelligent control method provided in the embodiment of the present application Figure 2 ;

[0060] Fig.11 A schematic diagram of a flow chart of a device control method provided in an embodiment of the present application;

[0061] Fig.12 This is a rendering of a device control method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solution in the embodiment of the present application will be described below in conjunction with the drawings in the embodiment of the present application. In the description of the embodiment of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0063] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0064] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0065] When users use electronic devices, there may be some scenarios where it is inconvenient to operate the electronic devices, such as: Figure 1 In the scenarios of sports, driving, dining, washing, etc. shown in the figure, when the electronic device plays a video or runs other applications, the user cannot click or touch the screen in time to interact with the electronic device, resulting in a poor user experience.

[0066] It should be noted that Figure 1 The scenarios shown in the figure are only examples, and there may be other scenarios where the user cannot click or touch the screen of the electronic device, which are not introduced here one by one.

[0067] In the related art, electronic devices support users to interact with the electronic devices through voice control or gesture control to achieve the purpose of controlling the electronic devices. However, in some scenarios, when the user needs to control the electronic device through voice or gesture control, the voice control function or gesture control function of the electronic device has not been turned on, so that the user can interact with the electronic device through voice or gesture control only after triggering the electronic device to turn on the voice control function or manual control function, resulting in a poor user experience.

[0068] In another scenario, the voice control function and / or gesture control function of the electronic device is turned on. However, when the user cannot click or touch the screen of the electronic device to interact with the electronic device, the user forgets to use the voice control function and the gesture control function, resulting in the user being unable to interact with the electronic device in a timely manner.

[0069] Figure 2 An example diagram of a voice control function activation path for an electronic device is shown. Figure 2 It can be seen that the electronic device detects the user's trigger operation on the setting icon 210, and displays the setting interface 220 in response to the user's trigger operation. The electronic device detects the user's trigger operation on the "voice control" 230, and displays the voice control interface 240 in response to the user's trigger operation. The electronic device responds to the user's trigger operation on the voice control switch 250, and controls the voice control function to be turned on, so that the user can interact with the electronic device through voice control. However, if the voice control function of the electronic device is in a long-on state, it will cause the power consumption of the electronic device to be relatively large.

[0070] Figure 3 An example diagram of a gesture control function activation path for an electronic device is shown. Figure 3 As shown in (a), the electronic device detects the user's trigger operation on the setting icon 310, and in response to the user's trigger operation, displays Figure 3 The electronic device detects the user's trigger operation on "intelligent perception" 330, and in response to the user's trigger operation, displays Figure 3 The smart sensing control interface 340 shown in (c) of FIG. 3 is a diagram showing the air gesture switches in the smart sensing control interface 340. The electronic device detects the user's triggering operation of the "air sliding screen" 350, and in response to the user's triggering operation, displays Figure 3 The electronic device detects the user's trigger operation on the "air sliding screen" switch 380, and in response to the user's trigger operation, turns on the air sliding screen switch. The electronic device detects the user's trigger operation on the "air pressing" 360, and in response to the user's trigger operation, displays Figure 3 Interface 390 shown in (e). The electronic device detects the user's triggering operation on the air-pressing switch 391, and turns on the air-pressing switch in response to the user's triggering operation.

[0071] It needs to be explained that the above Figure 2 The voice control function activation path shown in the figure is only an example. Figure 3 The path for enabling the gesture control function in the figure is only an example. The path for enabling the voice control function and the gesture control function in the electronic device shall be based on the path actually set in the electronic device. Figure 2 and Figure 3 It can be seen that the activation path of the voice control function and gesture control function of the electronic device is long, and there is a disadvantage of poor user experience.

[0072] To this end, the embodiment of the present application provides a device control method that can be applied to electronic devices. By judging whether the electronic device meets the preset state, the method recommends to the user to use an applicable target control method to interact with the electronic device, which can solve the problem in the prior art that the user cannot interact with the electronic device in a touch mode such as clicking or touching in a scenario where it is inconvenient for the user to use both hands, resulting in the user being unable to interact with the electronic device in a timely manner. In the embodiment of the present application, the user uses the recommended intelligent control mode to interact with the electronic device, which improves the efficiency of the user's interaction with the electronic device, thereby improving the user experience.

[0073] The target control method recommended by the electronic device for the user includes, but is not limited to, voice control, air gesture control, or eye movement control. This target control method is different from the touch control method, that is, the user can interact with the electronic device without clicking or touching the display screen of the electronic device.

[0074] Exemplarily, the device control method provided in the embodiments of the present application can be applied to mobile phones, tablet computers, personal computers (PCs), personal digital assistants (PDAs), smart watches, netbooks, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, vehicle-mounted devices, smart cars and other electronic devices with display screens, and the embodiments of the present application do not impose any restrictions on this.

[0075] like Figure 4 As shown, Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0076] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0079] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0080] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0081] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0082] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0083] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0084] The power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 .

[0085] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0086] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0087] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0088] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0089] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0090] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0091] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0092] In the embodiment of the present application, the camera 193 may be a low-power camera for capturing the user's eye movements or gesture movements in real time, so that the user can interact with the electronic device through eye contact or gestures in the air. Exemplarily, several preset gestures may be pre-set in the electronic device, such as swiping up, swiping down, swiping left, swiping right, pressing or flipping. After the camera 193 captures the preset gesture as swiping up, the electronic device executes the function corresponding to the swiping up.

[0093] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0094] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0095] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0096] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0097] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0098] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0099] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0100] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0101] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0102] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0103] In the embodiment of the present application, when the user turns on the switch of the intelligent control function through voice instructions, the microphone 170C can obtain the sound signal input by the user.

[0104] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0105] The pressure sensor 180A is used to sense pressure signals and convert the pressure signals into electrical signals.

[0106] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0107] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0108] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0109] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0110] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0111] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0112] Figure 5 A software structure diagram of an electronic device provided in an embodiment of the present application.

[0113] It is understandable that the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system may include an application layer (abbreviated as application layer), an application framework layer (abbreviated as framework layer), a system library, and a kernel layer.

[0114] The above-mentioned application layer may include a series of application packages.

[0115] like Figure 5 As shown, the application package may include system applications. System applications refer to applications that are set in the electronic device before leaving the factory. For example, the system applications may include programs such as camera, gallery, calendar, music, short message, memo, and weather.

[0116] The application package may also include third-party applications, which are applications that users install after downloading the installation package from an application store (or application market), such as map applications, food delivery applications, reading applications (such as e-books), social applications, and travel applications.

[0117] The application framework layer provides an application programming interface (API) and a programming framework for the application programs in the application layer. The application framework layer includes some predefined functions.

[0118] like Figure 5 As shown, the application framework layer may include a window manager, a view system, a resource manager, a notification manager, a context awareness module, and an intelligent control module, etc.

[0119] In some embodiments, the context awareness module of the mobile phone can determine whether the mobile phone is in a preset state. When the context awareness module determines that the mobile phone is in the preset state, the context awareness module continues to determine whether the foreground application is a preset application.

[0120] When the context perception module determines that the foreground application is a preset application, the intelligent control module prompts the user to turn on the switch of the intelligent control function. After the mobile phone determines that the switch of the intelligent control function is turned on, the context perception module determines whether the current scene of the mobile phone is a high-noise environment or a low-noise environment.

[0121] In one case, when the situational awareness module determines that the current scene of the mobile phone is a high-noise environment, the intelligent control module can recommend the user to use an air gesture control method or an eye movement control method to interact with the mobile phone.

[0122] In another case, when the context awareness module determines that the current scene of the mobile phone is a low-noise environment and not a public place, the intelligent control module can recommend the user to use the voice control method to interact with the mobile phone. The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0123] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0124] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0125] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a reminder sound is emitted, the phone vibrates, the indicator light flashes, etc.

[0126] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.

[0127] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0128] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0129] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0130] The surface manager is used to manage the display subsystem and provide the fusion of two-dimensional and three-dimensional layers for multiple applications.

[0131] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0132] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0133] A 2D graphics engine is a drawing engine for 2D drawings.

[0134] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver and sensor driver.

[0135] The following takes a mobile phone as an example of an electronic device and describes in detail the device control method provided by the present application in conjunction with the accompanying drawings.

[0136] In an embodiment of the present application, when the mobile phone is in a preset state and determines that the intelligent control function is in an off state, the mobile phone can display a prompt message on the display interface, or prompt the user to turn on the intelligent control function by voice prompt. After the mobile phone detects the user's trigger operation to turn on the intelligent control function, in response to the user's trigger operation, the mobile phone controls the intelligent control function to turn on and enters the intelligent control mode. In the intelligent control mode, the mobile phone can interact with the mobile phone according to the intelligent control method recommended to the user, so that the user can use the appropriate intelligent control method to interact with the mobile phone in time in different scenarios. Among them, the intelligent control method is the target control method, including voice control method, air gesture control method or eye movement control method.

[0137] For example, when the phone is in a low-noise environment, it is recommended that users use voice control to interact with the phone; when the phone is in a high-noise environment, it is recommended that users use air gesture control to interact with the phone.

[0138] In one embodiment, the above-mentioned preset state includes that the duration of the mobile phone being in a static state reaches duration 1 (i.e., the first preset duration, such as 3 seconds, 5 seconds, or 8 seconds, etc.). It can be understood that when the mobile phone is in the preset state, it is inconvenient for the user to interact with the mobile phone by clicking or touching the mobile phone screen, and the mobile phone can recommend to the user to use the control method suitable for the scene to interact with the mobile phone according to the current scene.

[0139] For example, assuming that the phone is playing music in the off-screen state, and the phone is in a static state for a period of time 1, the phone can also light up the screen and display a prompt message to prompt the user to interact with the phone in a target control manner. For example, when the phone is playing music in the off-screen state, the user can switch to playing songs through voice control. Therefore, when the phone is in the off-screen state, the user can interact with the phone through voice control without turning on the screen, which improves the user experience.

[0140] In another embodiment, the preset state may also include one or more of the following: the mobile phone currently displays an interface, the distance between the mobile phone and the user is greater than a distance threshold (e.g., 50 cm, 80 cm, etc.), or the mobile phone is in a non-horizontal placement state (i.e., the angle between the mobile phone and the horizontal plane is greater than 0°). For example, the mobile phone is supported by a bracket and the distance between the mobile phone and the user is greater than the distance threshold.

[0141] For example, when the mobile phone currently displays a certain interface, when the mobile phone is in a static state for a period of time of 1, the distance between the mobile phone and the user is greater than the distance threshold, and the mobile phone displays a prompt message to prompt the user to interact with the mobile phone in a target control manner.

[0142] It can be understood that when the distance between the mobile phone and the user is greater than the distance threshold, the user cannot touch or click the mobile phone screen with both hands. In this case, the mobile phone can prompt the user to interact with the mobile phone in a target control manner, which is conducive to improving the user experience. When the mobile phone is in a non-horizontal position, for example, the mobile phone is supported by a bracket, it is inconvenient for the user to use both hands to trigger the display screen of the mobile phone. The mobile phone can display a prompt message to prompt the user to interact with the mobile phone in a target control manner.

[0143] In addition, the interface currently displayed on the mobile phone may be the interface of the foreground application, or may be any other interface, which is not limited in the embodiments of the present application.

[0144] It should be explained that when the mobile phone in the embodiment of the present application is in a preset state, the mobile phone can be in a horizontal screen state or a vertical screen state, and the solution in the embodiment of the present application does not limit the placement state of the mobile phone screen.

[0145] In one embodiment, the mobile phone can prompt the user to turn on the switch of the intelligent control function by displaying a prompt message, so as to prompt the user to interact with the mobile phone by voice or gesture control. For example, assuming that the mobile phone is playing a video in the stand state, the mobile phone determines that the switch of the intelligent control function is in the off state, and the mobile phone interface displays Figure 6 The prompt box 601 shown in (a) includes prompt information to prompt the user to turn on the switch of the intelligent control function.

[0146] After the phone prompts the user to turn on the smart control function, the user can interact with the phone through voice control to turn on the smart control function. Figure 6 In the process of the prompt box 601 shown in (a), the user triggers the operation of turning on the switch of the intelligent control function through the voice message "Turn on the switch of the intelligent control function", and in response to the user's triggering operation, the mobile phone controls the switch of the intelligent control function to turn on.

[0147] For example, the mobile phone displays Figure 6 The prompt box 602 shown in (b) includes prompt information to prompt the user to turn on the switch of the intelligent control function.

[0148] After the phone prompts the user to turn on the smart control function, the user can interact with the phone through air gesture control to turn on the smart control function. Figure 6 In the process of the prompt box 602 shown in (b), the user triggers the operation of turning on the switch of the intelligent control function through a preset gesture, and in response to the user's triggering operation, the mobile phone controls the switch of the intelligent control function to turn on.

[0149] For example, the mobile phone displays Figure 6 The prompt box 603 shown in (c) includes a prompt message "Do you want to turn on the intelligent control function?" to prompt the user to turn on the switch of the intelligent control function.

[0150] It should be noted that the mobile phone displays Figure 6 After receiving the prompt information shown in , the user can indicate to turn on the switch of the smart control function, so that the mobile phone enters the smart control mode. It can be understood that Figure 6 The method of displaying the prompt information through the prompt box is only an example. The embodiment of the present application does not limit the display method of the prompt information. For example, the mobile phone can also directly display the prompt information in the display interface, or the mobile phone can also display a card including the reminder information in the display interface, etc. In addition, Figure 6The several prompt messages displayed in the example are only used as an exemplary description, and the specific prompt messages displayed in the embodiment of the present application are not limited. For example, the prompt message can also be "turn on the intelligent control function".

[0151] In another embodiment, the mobile phone can prompt the user to turn on the switch of the intelligent control function through voice prompts to enter the intelligent control mode. For example, assuming that the mobile phone is playing a video in the stand state, the mobile phone determines that the switch of the intelligent control function is in the off state, and the mobile phone prompts the user to trigger the switch of the intelligent control function through a voice prompt message "Turn on the intelligent control function". The user triggers the operation of turning on the switch of the intelligent control function through the voice message "Turn on the switch of the intelligent control function", and in response to the user's triggering operation, the mobile phone controls the switch of the intelligent control function to turn on. After the mobile phone enters the intelligent control mode for the first time, it can give the user a prompt description on the display or through voice to inform the user of the function and role of the intelligent control mode. For example, the mobile phone can prompt the user by displaying a text message "After entering this mode, you can operate the device by voice, gesture or eye movement". For another example, see Figure 7 The mobile phone can prompt the user through window 701 that in the intelligent control mode, the user can interact with the device by voice, gesture or eye movement, so as to facilitate the user to understand the functions of the intelligent control mode.

[0152] Above Figure 6 and Figure 7 The position of the prompt information displayed in the figure is only an example. The prompt information can be displayed at any position of the display screen of the mobile phone. The position of displaying the prompt information in the embodiment of the present application is not limited.

[0153] In the embodiment of the present application, after the mobile phone prompts the user to turn on the switch of the intelligent control function by displaying a prompt message or a voice prompt, the mobile phone does not receive the user's operation of triggering the switch to turn on the intelligent control function within a time length of 2 (for example, 8 seconds or 10 seconds, etc.), and the mobile phone no longer prompts the user to turn on the switch of the intelligent control function. Alternatively, the mobile phone does not receive the user's operation of triggering the switch to turn on the intelligent control function within a time length of 2 (for example, 8 seconds or 10 seconds, etc.), and the mobile phone again prompts the user to turn on the switch of the intelligent control function by displaying a prompt message or a voice prompt. If the mobile phone still does not receive the user's operation of triggering the switch to turn on the intelligent control function within a time length of 2, the mobile phone no longer prompts the user to turn on the switch of the intelligent control function.

[0154] For example, assuming Figure 6 During the display time 2 of the prompt box 601 shown in (a), the mobile phone still does not receive the user's operation of triggering the switch to turn on the intelligent control function, and the prompt box 601 is no longer displayed in the interface of the mobile phone.

[0155] During the operation of the mobile phone, although the mobile phone no longer prompts the user to trigger the switch to turn on the intelligent control function, the user indicates the operation to trigger the switch to turn on the intelligent control function, and in response to the user's triggering operation, the mobile phone can still control the switch to turn on the intelligent control function.

[0156] After the mobile phone responds to the user's triggering operation and controls the switch of the intelligent control function to be turned on, the switch of the intelligent control function of the mobile phone is in a long-on state until the user manually turns off the switch of the intelligent control function in the setting interface.

[0157] For example, Figure 8 As shown, the mobile phone detects the user's trigger operation on the setting icon 810, and in response to the user's trigger operation, displays the setting interface 820. Among them, the switch of "smart control" 830 in the setting interface 820 is in the on state. The mobile phone detects the user's trigger operation on "smart control" 830, and in response to the user's trigger operation, displays the smart control interface 840. The mobile phone can control the smart control function to be turned off in response to the user's trigger operation on the main switch 850 of the smart control. Alternatively, the mobile phone can also control the smart control function to be turned off in response to the user's trigger operations on "voice control", "air gesture control" and "eye control" respectively, and the mobile phone exits the smart control mode.

[0158] After the mobile phone exits the smart control mode in response to the user's trigger operation, the mobile phone determines that the user does not need to interact with the mobile phone through the smart control method during this use of the mobile phone, and no longer prompts the user to turn on the smart control function switch. However, during the operation of the mobile phone, if the user instructs the mobile phone to turn on the smart control function switch, the mobile phone responds to the user's instruction operation, controls the smart control function switch to turn on, and enters the smart control mode.

[0159] After the mobile phone enters the intelligent control mode, the mobile phone can recommend the intelligent control method to the user to prompt the user that the user can interact with the mobile phone through the intelligent control method.

[0160] In one embodiment, when the mobile phone meets the first condition, a first prompt message is displayed. The first prompt message is used to prompt the user to use the first control method to interact with the mobile phone. The first condition includes that the mobile phone is in a preset state and the volume value of the current environment is greater than the volume threshold. The first control method can be an air gesture control method or an eye movement control method. In other words, the mobile phone recommends that the user use the corresponding intelligent control method under different noise intensities to interact with the mobile phone according to the noise intensity of the current environment.

[0161] In one scenario, assuming that the mobile phone is in a preset state and the current environment of the mobile phone is a high-noise environment, the mobile phone can recommend the user to use air gestures or eye control to interact with the mobile phone.

[0162] For example, suppose a mobile phone is playing a video in a high noise environment (for example, the user is watching a video in a noisy restaurant), and the mobile phone displays Fig. 9 The prompt box 901 shown in (a) includes prompt information to recommend the user to use the air gesture control method to interact with the mobile phone. This solves the problem that the user's hands are occupied and cannot interact with the mobile phone by clicking or touching the screen.

[0163] for example, Fig. 9 In the mobile phone shown in (a), a video is playing. The user can trigger the air press operation. In response to the user's trigger operation, the mobile phone controls Fig. 9 The video shown in (a) is paused.

[0164] It can be understood that when the mobile phone is in a high-noise environment, if the user uses voice control to interact with the mobile phone, the microphone of the mobile phone may be greatly disturbed by the ambient sound and cannot accurately capture the trigger instruction indicated by the user's voice, resulting in low accuracy of voice interaction between the user and the mobile phone. Therefore, in the embodiment of the present application, when the mobile phone is in a high-noise environment, the mobile phone recommends that the user use air gesture control or eye movement control to interact with the mobile phone to improve the accuracy of the user's control of the mobile phone, which is conducive to improving the user's experience.

[0165] When the mobile phone meets the second condition, the second prompt information is displayed. The second prompt information is used to prompt the user to use the second control method to interact with the mobile phone. The second condition includes that the mobile phone is in a preset state and the volume value of the current environment is less than the volume threshold. The second control method can be a voice control method.

[0166] That is to say, assuming that the mobile phone is in a preset state and the current environment of the mobile phone is a low-noise environment, the mobile phone can recommend the user to use voice control to interact with the mobile phone.

[0167] For example, assuming that a user uses a mobile phone at home to play a video in a stand state, the mobile phone can display Fig.10 The prompt box 1001 shown in (a) includes prompt information to recommend the user to use voice control to interact with the video application running in the foreground. Therefore, by recommending the user to use voice control to interact with the mobile phone, the scenario where the user cannot click or touch the screen with both hands and cannot interact with the mobile phone is avoided. The mobile phone can be accurately controlled to perform corresponding operations through voice instructions, which is conducive to improving the user experience.

[0168] for example, Fig.10 The mobile phone shown in (a) is playing a video. The user can use voice to instruct a trigger operation of "fast forward 3 seconds". In response to the user's trigger operation, the mobile phone controls the video to fast forward 3 seconds.

[0169] For example, when a mobile phone is playing a video, the playback interface can prompt the user to use the voice control function, such as Fig.10 In the prompt box 1002 shown in (b), there is a prompt "adjust the definition to HD". The user triggers the operation by voice instruction "adjust the definition to HD", and in response to the user's operation, the mobile phone adjusts the definition of the video being played to HD.

[0170] It can be understood as, Fig.10 The prompt information displayed in the prompt box 1002 shown in (b) is only an example, and other information may also be displayed, such as fast forward, fast rewind, pause, next episode, previous episode, etc.

[0171] In another embodiment, the mobile phone may also recommend, based on the current location of the mobile phone, that the user use control methods corresponding to different geographical locations to interact with the mobile phone.

[0172] In one scenario, assuming that the mobile phone is in a preset state and the current location of the mobile phone is home, the mobile phone can recommend the user to use voice control to interact with the mobile phone. For example, assuming that the mobile phone is placed on a treadmill at home, the mobile phone can still display Fig.10 The prompt box 1001 shown in (a) includes prompt information to recommend the user to use voice control to interact with the video application running in the foreground.

[0173] In another scenario, assuming that the phone is in a preset state and the phone is currently in a public place, the phone can recommend the user to use air gesture control to interact with the phone. For example, assuming that the phone is placed on a treadmill in a gym, the phone can still display the video when it is playing. Fig. 9 The prompt box 901 shown in (a) includes prompt information to recommend the user to use air gesture control to interact with the video application running in the foreground.

[0174] In yet another embodiment, in order to make the control method recommended by the mobile phone more in line with the user's needs, the mobile phone can recommend a suitable control method to the user in combination with the noise intensity of the current environment and the current geographical location.

[0175] In one scenario, assuming that the mobile phone is in a preset state, the ambient sound of the current scene of the mobile phone is a low-noise environment, and the current geographical location is home, the mobile phone can recommend the user to use voice control to interact with the mobile phone.

[0176] In another scenario, assuming that the mobile phone is in a preset state, the ambient sound of the current scene in which the mobile phone is located is a low-noise environment, and the current geographical location is a public place, the mobile phone can recommend the user to use air gestures or eye control to interact with the mobile phone.

[0177] In yet another embodiment, the mobile phone may also recommend to the user a control method corresponding to the current scene based on the noise intensity of the current scene, the current geographical location, and the current time.

[0178] For example, assuming that the mobile phone is in a preset state, the ambient sound of the scene in which the mobile phone is currently located is a low-noise environment, the current geographical location is a preset location (for example, home), and the current time is within a preset range (for example, the preset range is 22:00 to 06:00, and the current time is 23:00). In order to avoid disturbing the family's rest, the mobile phone can recommend the user to interact with the mobile phone using air gestures or eye movement control.

[0179] In another embodiment, the mobile phone can also recommend control methods corresponding to different application types to the user according to the application type of the foreground application, so that the user can interact with the mobile phone using the recommended control method. The foreground application refers to the state where the application is being opened, operated, or currently displayed by the user.

[0180] In one scenario, when the electronic device meets the third condition, a third prompt message is displayed, and the third prompt message is used to prompt the user to use a third control method to interact with the electronic device. The third condition includes that the electronic device is in a preset condition and the foreground application is a first type of application. Assuming that the mobile phone determines that the foreground application is a first type of application (for example, a reading application), the mobile phone can recommend the user to interact with the mobile phone using air gestures or eye control. After the mobile phone receives the user's operation of indicating the foreground application using air gestures or eye control, it controls the foreground application to execute the corresponding function in response to the user's operation.

[0181] For example, assuming that a mobile phone is playing an e-book in a stand state, the mobile phone recommends that the user interact with the mobile phone using air gestures or eye control. The mobile phone receives the user's left swipe instruction operation, and in response to the user's operation, it can control the e-book to turn pages.

[0182] In another scenario, when the electronic device meets the fourth condition, the fourth prompt information is displayed, and the fourth prompt information is used to prompt the user to use the fourth control method to interact with the electronic device. The fourth condition includes that the electronic device is in a preset condition and the foreground application is a second type of application. Assuming that the mobile phone determines that the foreground application is a second type of application (for example, a video playback application, an audio playback application, etc.), the mobile phone can recommend the user to use voice control to interact with the mobile phone. After the mobile phone receives the user's operation of indicating the foreground application using voice control, it controls the foreground application to execute the corresponding function in response to the user's operation.

[0183] For example, suppose a mobile phone is playing a song in a cradle state, and the mobile phone recommends that the user interact with the mobile phone using voice control. The mobile phone receives a voice message from the user indicating an operation to "play the next song". In response to the user's operation, the music playback application can be controlled to play the next song.

[0184] In another embodiment, the mobile phone can also recommend to the user a control method corresponding to the current scene based on the foreground application, the noise intensity of the current scene, the current geographical location and the current time.

[0185] For example, assuming that the mobile phone is in a preset state, the foreground application is a preset application (for example, a video playback application, an audio playback application, etc.), the ambient sound of the current scene is a low-noise environment, the current geographical location is home and the current time is 12 o'clock in the evening, the mobile phone recommends to the user to use air gesture control or eye movement control to interact with the mobile phone.

[0186] In an embodiment of the present application, after the mobile phone recommends that the user use a certain intelligent control method to interact with the mobile phone, the user may not use the control method recommended by the mobile phone to interact with the mobile phone. For example, when the mobile phone is determined to be in a high-noise environment, the mobile phone recommends that the user use air gesture control or eye movement control to interact with the mobile phone. However, when the user interacts with the mobile phone through voice control, the mobile phone can still perform the corresponding function in response to the user's voice trigger operation. For example, when the mobile phone is in a low-noise environment, but someone is resting, the mobile phone recommends that the user use voice control to interact with the mobile phone. In this case, the user uses air gesture or eye movement control to interact with the mobile phone, and in response to the user's trigger operation, the mobile phone performs the corresponding function. For example, when the mobile phone is in a public place, the mobile phone recommends that the user use air gesture control to interact with the mobile phone, but when the user interacts with the mobile phone through voice control, the mobile phone can also perform the corresponding function in response to the user's voice trigger operation.

[0187] In an embodiment of the present application, the mobile phone prompts the user to interact with the mobile phone by voice or air gesture control by displaying a prompt message. The mobile phone displays the prompt message for a duration of 3 (i.e., the second preset duration, for example, the second preset duration is 5 seconds or 8 seconds, etc.), and does not receive a trigger operation from the user to interact with the mobile phone by voice or air gesture control. The prompt message is no longer displayed in the mobile phone's interface. Alternatively, the mobile phone displays the prompt message for a duration of 3 again, and still does not receive a trigger operation from the user to interact with the mobile phone by voice or air gesture. The mobile phone prompts the user to interact with the mobile phone by voice or air gesture by displaying a prompt message again. If the mobile phone displays the prompt message for a duration of 3 and still does not receive a trigger operation from the user to interact with the mobile phone by voice or air gesture control, the prompt message is no longer displayed in the mobile phone's interface.

[0188] For example, assuming Fig. 9 The display time of the prompt box 901 shown in (a) reaches time 3, and the mobile phone still does not receive the trigger operation of the user interacting with the mobile phone by voice or air gesture control. The prompt box 901 is no longer displayed in the interface of the mobile phone, and the display Fig. 9 Interface 902 shown in (b).

[0189] For example, suppose Fig.10 The display time of the prompt box 1001 shown in (a) reaches time 3, and the mobile phone has not received a trigger operation from the user to interact with the mobile phone using voice or air gesture control. The prompt box 1001 is no longer displayed in the interface of the mobile phone, affecting the user's experience of watching the video.

[0190] When the mobile phone is running a foreground application, although the mobile phone no longer prompts the user to interact with the mobile phone using voice or air gesture control, when the user triggers the foreground application through voice or air gesture control, the mobile phone can still control the foreground application to execute the corresponding function in response to the user's triggering operation.

[0191] It needs to be explained that the mobile phone recommends users to use voice or air gesture control to interact with the mobile phone by displaying prompt information only as a prompt method. The mobile phone can also prompt users to use voice or air gesture control to interact with the mobile phone through voice prompts. This is not limited here.

[0192] In combination with the above embodiments and related drawings, the present application embodiment provides a device control method. The method can be used in Figure 4 and Figure 5 The electronic device (e.g., mobile phone, tablet computer, etc.) shown in FIG. Fig.11 As shown, the method may include S1101 to S1107.

[0193] S1101, determining whether the mobile phone is in a preset state.

[0194] In an embodiment of the present application, after the mobile phone is turned on, the mobile phone can continuously obtain acceleration information collected by the acceleration sensor or angle information collected by the gyroscope, so as to determine whether the mobile phone is in a stationary state for a period of time exceeding the duration 1 according to the acceleration information or the angle information. Alternatively, the mobile phone can also determine whether the mobile phone is in a horizontal state according to the angle information. Alternatively, the mobile phone can also continuously obtain the distance information between the mobile phone and the user collected by the distance sensor, so as to determine whether the distance between the mobile phone and the user is greater than the distance threshold according to the distance information.

[0195] In one case, when the mobile phone determines, based on the acceleration information or the angle information, that the duration for which the mobile phone is in a stationary state exceeds duration 1, it is determined that the mobile phone is in a preset state.

[0196] In another case, when the mobile phone determines that the time the mobile phone is in a stationary state exceeds time 1 based on acceleration information or angle information, and determines that the mobile phone is not in a horizontal state based on angle information and determines that the distance between the mobile phone and the user is greater than a distance threshold based on distance information, it is determined that the mobile phone is in a preset state.

[0197] For example, suppose that a user places a mobile phone on a mobile phone holder while cooking. At this time, when the mobile phone is in a stationary state for more than time 1, there is a certain angle between the mobile phone and the holder, and the distance between the mobile phone and the user is greater than a distance threshold, it is determined that the mobile phone is in a preset state.

[0198] S1102, determining whether the foreground application is a preset application.

[0199] The preset application is a pre-set application suitable for running the mobile phone in a static state. For example, the preset application can be a video playback application, a music playback application, a reading application, an album application, etc.

[0200] As a possible implementation manner, the mobile phone can determine the running foreground application by monitoring ActivityManager to determine whether the foreground application is a preset application.

[0201] Of course, the mobile phone may also use other methods to determine whether the foreground application is a preset application, and the method for determining the foreground application is not limited here.

[0202] S1103, turning on the switch of the intelligent control function.

[0203] In the embodiment of the present application, the mobile phone determines that the current state meets the preset state, and the running foreground application is the preset application. In this case, the mobile phone determines that the switch of the intelligent control function is in the off state, and the mobile phone can prompt the user to control the switch of the intelligent control function to turn on.

[0204] It should be explained that when the mobile phone only meets the condition of S1101, the mobile phone will also prompt the user to turn on the switch of the intelligent control function. Alternatively, the mobile phone meets the conditions of S1101 and S1102 at the same time, and the mobile phone prompts the user to turn on the switch of the intelligent control function. It can be understood that when it is inconvenient for the user to interact with the mobile phone by clicking or touching, and the application currently running on the mobile phone can be controlled by intelligent control methods such as voice or air gestures, the mobile phone can prompt the user to turn on the intelligent control function, so that the user can interact with the mobile phone by voice control or air gesture control, which is conducive to improving the user experience.

[0205] In some embodiments, when the mobile phone is in a preset state and the foreground application running in the mobile phone is a preset application, the mobile phone determines that the switch of the intelligent control function is in the on state. In this case, the mobile phone does not need to prompt the user to turn on the switch of the intelligent control function. In other words, when the intelligent control function of the mobile phone is in the on state, step S1103 is not executed, and step S1104 is directly executed.

[0206] S1104, determining whether the volume value of the current environment is greater than a volume threshold.

[0207] The volume threshold is pre-set and is used to distinguish the volume values ​​of high-noise ambient sound and low-noise ambient sound. For example, the volume threshold can be set to 60db. When the mobile phone determines that the volume value of the current environment is 65db, it is determined that the mobile phone is currently in a high-noise environment; when the mobile phone determines that the volume value of the current environment is 40db, it is determined that the mobile phone is currently in a low-noise environment.

[0208] In an embodiment of the present application, when the mobile phone is in a preset state and the foreground application is a preset application, the mobile phone obtains the sound signal of the ambient sound collected by the microphone and determines whether the volume value of the ambient sound is greater than the volume threshold based on the sound signal.

[0209] In one possible case, when the mobile phone determines that the volume value of the current environment is less than the volume threshold, that is, the mobile phone is in a low-noise environment, the mobile phone executes step S1105.

[0210] In another possible case, when the mobile phone determines that the volume value of the ambient sound is greater than the volume threshold, that is, the mobile phone is in a high-noise environment, the mobile phone executes step S1106.

[0211] S1105, recommending using the first control method to interact with the mobile phone.

[0212] Among them, the first control method can be an air gesture control method or an eye movement control method.

[0213] In one scenario, the mobile phone is in a high-noise environment, and the mobile phone recommends that the user use the first control method to interact with the mobile phone. For example, assuming that the first control method is the air gesture control method, the mobile phone can display Fig. 9 The prompt box 901 shown in (a) recommends the user to use air gestures to interact with the mobile phone.

[0214] S1106, determining whether the mobile phone is in a preset position.

[0215] The preset location may be a location where it is not suitable to interact with the mobile phone by voice control, for example, a library, a cinema, a shopping mall, an office or other public place.

[0216] In an embodiment of the present application, the mobile phone is in a low-noise environment, and the mobile phone can obtain the current location information of the mobile phone through the global positioning system (GPS) to determine whether the mobile phone is in a preset location based on the location information.

[0217] In addition, the mobile phone can also determine the current location information of the mobile phone through base station positioning, wireless network (wireless fidelity, WiFi) positioning, etc., so as to determine whether the mobile phone is in a preset location based on the location information.

[0218] In one case, if the mobile phone determines that the mobile phone is at a preset position according to the location information, the mobile phone recommends the user to use the first control method to interact with the mobile phone, that is, execute step S1105.

[0219] In another case, if the mobile phone determines that the mobile phone is not at the preset location based on the location information, the mobile phone can obtain the current time and recommend to the user a suitable control method for interacting with the mobile phone based on the current time.

[0220] For example, assuming that the mobile phone determines that the current location is home, and the current time obtained by the mobile phone is 5 o'clock in the morning, the mobile phone can recommend the user to use air gestures or eye control to interact with the mobile phone, thereby avoiding the problem of users using voice control to interact with the mobile phone and disturbing their family's rest.

[0221] In another case, if the mobile phone determines that the mobile phone is not at the preset location based on the location information, step S1107 is executed.

[0222] S1107, it is recommended to use the second control method to interact with the mobile phone.

[0223] The second control method may be a voice control method.

[0224] In one scenario, when the mobile phone is in a low-noise environment and at home, the mobile phone may recommend the user to use the second control method to interact with the mobile phone. Fig.10 The prompt box 1001 shown in (a) recommends the user to use voice control to interact with the mobile phone.

[0225] Here, when the mobile phone is in a low-noise environment and recommends the user to use the second control method to interact with the mobile phone, the user may not use the second control method recommended by the mobile phone to interact with the mobile phone. For example, the mobile phone recommends the user to use voice control to interact with the mobile phone, and the user may still use air gestures or eye movement control to interact with the mobile phone.

[0226] In one scenario of an embodiment of the present application, when the mobile phone is in a preset state and the running foreground application is a preset application, the mobile phone can determine the control method used by the user last time in the scenario, or the control method used most times. Then, the mobile phone recommends the user to use the control method used last time, or the control method used most times to interact with the mobile phone. Thus, the mobile phone can recommend the user to use the control method to interact with the mobile phone according to the user's usage habits, which is conducive to improving the user's usage experience.

[0227] It can be understood that the mobile phone prompts the user to use the intelligent control method to interact with the mobile phone, so as to remind the user that when it is inconvenient to use both hands to interact with the mobile phone, the user can use the intelligent control method to interact with the mobile phone in time, thereby improving the interaction efficiency between the user and the mobile phone and improving the user experience.

[0228] In addition, if Fig.12 As shown, the device control method of the present application is improved at the application framework layer of the electronic device, and the intelligent control method is recommended to the user through the context awareness module and the intelligent control module in the multi-mode human-computer interaction framework.

[0229] In the embodiment of the present application, the electronic device can obtain the information collected by the low-power sensing camera in real time, so as to enable the user to interact with the electronic device through eye contact or gestures in the air according to the user's eye movements or gestures collected by the camera, which has the advantage of accurate visual detection. In addition, the electronic device can identify the current scene and accurately identify the volume value of the ambient sound, so as to recommend the user to use the control method suitable for the current environment according to the volume value of the ambient sound, thereby realizing the recommendation of the most suitable interaction method to the user according to the current scene of the device, and achieving the purpose of intelligent scheduling of interaction methods.

[0230] For example, when the electronic device is in the screen-off state, the electronic device can light up the screen of the device in response to the user's breath wake-up operation.

[0231] In addition, electronic devices can also remove noise in the environment and accurately identify control information indicated by user voice in a high-noise environment, thereby improving the accuracy of voice interaction and helping to improve user experience.

[0232] It is understandable that, in order to realize the above functions, the above-mentioned electronic devices, etc. include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present invention.

[0233] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device etc. according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0234] In the case of dividing each functional module according to each function, a possible composition diagram of the electronic device involved in the above embodiment, the electronic device may include: a display unit, a transmission unit and a processing unit, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0235] The embodiment of the present application also provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned related method steps to implement the device control method in the above-mentioned embodiment.

[0236] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the device control method in the above-mentioned embodiment.

[0237] An embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the device control method in the above-mentioned embodiment.

[0238] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the device executes the device control method executed by the electronic device in the above-mentioned method embodiments.

[0239] Among them, the electronic device, computer-readable storage medium, computer program product or device provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0240] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0241] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0243] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A device control method, characterized in that: The method comprises: When the electronic device is in a preset state, a prompt message is displayed, wherein the prompt message is used to prompt the user to interact with the electronic device in a target control mode, wherein the preset state includes the electronic device being in a static state for a first preset time, and the target control mode is different from the touch mode; Receiving a first operation of a user controlling the electronic device in a target control manner; In response to the first operation, controlling the electronic device to perform a corresponding function; Wherein, when the target control mode includes a first control mode or a second control mode, and the first control mode is different from the second control mode; when the electronic device is in a preset state, displaying prompt information includes: When the electronic device meets a first condition, displaying a first prompt message, wherein the first prompt message is used to prompt the user to interact with the electronic device in the first control mode, wherein the first condition includes that the electronic device is in the preset state and the volume value of the current environment is greater than a volume threshold; When the electronic device meets the second condition, a second prompt information is displayed, and the second prompt information is used to prompt the second control method to interact with the electronic device. The second condition includes that the electronic device is in the preset state and the volume value of the current environment is less than the volume threshold.

2. The method according to claim 1, characterized in that: The preset state also includes one or more of the electronic device currently displaying an interface, the distance between the electronic device and the user is greater than a distance threshold, or the electronic device is in a non-horizontally placed state.

3. The method according to claim 1, characterized in that The first condition includes that the electronic device is in the preset state, the foreground application is a preset application, and the volume value of the current environment is greater than the volume threshold; The second condition includes that the electronic device is in the preset state, the foreground application is the preset application, and the volume value of the current environment is less than the volume threshold.

4. The method according to claim 1, characterized in that The second condition also includes that the current position of the electronic device is not a preset position, and the method further includes: When the volume value of the current environment is less than the volume threshold, if the current position is not the preset position, the second prompt information is displayed.

5. The method according to claim 4, characterized in that The first condition also includes that the current time is within a preset range, and the method further includes: When the volume value of the current environment is less than the volume threshold and the current position is a preset position, if the current time is not within the preset range, the second prompt information is displayed.

6. The method according to claim 1, characterized in that When the target control mode includes the third control mode or the fourth control mode, and the third control mode is different from the fourth control mode, when the electronic device is in a preset state, displaying prompt information includes: When the electronic device meets a third condition, displaying a third prompt message, wherein the third prompt message is used to prompt the user to interact with the electronic device in the third control mode, wherein the third condition includes that the electronic device is in the preset state and the foreground application is an application of the first type; When the electronic device meets the fourth condition, a fourth prompt message is displayed, where the fourth prompt message is used to prompt the fourth control method to be used to interact with the electronic device, and the fourth condition includes that the electronic device is in the preset state and the foreground application is an application of the second type.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Display notification information, where the notification information is used to prompt the user to turn on the intelligent control mode; Receive a second operation of the user indicating to start the intelligent control mode; In response to the second operation, controlling the intelligent control mode to start; The displayed prompt information includes: After the intelligent control mode is turned on, the prompt information is displayed.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When it is determined that the display time of the prompt information reaches the second preset time, no operation of the user using the target control method to trigger the foreground application is received; Stop displaying the prompt information.

9. The method according to any one of claims 1 to 6, characterized in that: The first control method is air gesture control or eye movement control, and the second control method is voice control.

10. The method according to claim 3, characterized in that: The preset application is a video application, the first control mode is an air gesture control mode or an eye movement control mode, the second control mode is a voice control mode, and when the electronic device meets the first condition, displaying the first prompt information includes: When the electronic device is in the preset state, the foreground application is the video application and the volume value of the current environment is greater than the volume threshold, the first prompt information is displayed, and the first prompt information is used to prompt the user to interact with the electronic device by using the air gesture control method or the eye movement control method; When the electronic device meets the second condition, displaying the second prompt information includes: When the electronic device is in the preset state, the foreground application is the video application and the volume value of the current environment is less than the volume threshold, the second prompt information is displayed, and the second prompt information is used to prompt the use of the voice control method to interact with the electronic device.

11. The method according to claim 4, characterized in that The first control mode is an air gesture control mode or an eye movement control mode, the second control mode is a voice control mode, and if the current position is not the preset position, displaying the second prompt information includes: When the electronic device is in the preset state, the volume value of the current environment is less than the volume threshold and the current location is not a public place, the second prompt information is displayed, and the second prompt information is used to prompt the user to interact with the electronic device using the voice control method.

12. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images