Electronic device performing lift-up-wake-up and method of operating electronic device

Abstract

An electronic device according to one embodiment comprises: a display; a plurality of sensors including a first acceleration sensor; one or more processors; and a memory for storing instructions, wherein when the instructions are individually or collectively executed by the one or more processors, the electronic device may extract first motion information of a motion section from a first signal obtained from the first acceleration sensor, generate first motion recognition information (posture information) on the basis of inputting the first signal and the first motion information into an artificial intelligence model, and turn on the display on the basis of the generated first motion recognition information.

Description

Electronic device performing pick-up and turning on and method of operating the electronic device

[0001] Embodiments of the present invention relate to an electronic device performing lift-up-wake-up and a method of operating the electronic device.

[0002] The electronic device can perform a lift-up-wake-up function by using a built-in sensor to detect the user's lifting motion of the electronic device and automatically turn on the screen.

[0003] Specifically, an electronic device can monitor its movement or changes in direction in real time using built-in sensors. The electronic device can recognize specific movement patterns (e.g., lifting up) based on signals related to movement or changes in direction acquired using the sensors. The electronic device can then activate its screen based on the recognized specific movement pattern. The electronic device can enhance the user experience and make everyday electronic device use more convenient by allowing the user to quickly check the screen of the electronic device without having to press a button through the lift-to-turn function.

[0004] The above information may be provided as background information to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

[0005] Lift-up-wake-up technology performed on electronic devices has a limited scenario in which it only works when a user holds the electronic device vertically and lifts it while standing. For example, the electronic device can perform the lift-up-wake-up action when the user holds the electronic device vertically and lifts it perpendicular to the ground. The electronic device can recognize that the user has performed the lift-up-wake-up motion by using data acquired through an acceleration sensor when the axis of the acceleration sensor exceeds a certain threshold value relative to a reference axis, and can perform the lift-up-wake-up action.

[0006] However, users can pick up electronic devices in a variety of positions. For example, they may pick up the device lying down or holding it horizontally. Comparing the acceleration sensor's axis to the reference axis in various situations in which the user picks up the device requires a significant amount of computation, which can result in slow motion recognition.

[0007] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0008] According to one embodiment, an electronic device includes a display, a plurality of sensors including a first acceleration sensor, one or more processors, and a memory storing instructions, wherein when the instructions are individually or collectively executed by the one or more processors, the electronic device can: extract first motion information of a motion section from a first signal acquired from the first acceleration sensor, and generate first motion recognition information based on inputting the first signal and the first motion information into an artificial intelligence model (e.g., a machine learning model), and turn on a display based on the generated first motion recognition information.

[0009] A method performed by an electronic device according to one embodiment may include an operation of extracting first motion information of a motion section from a first signal acquired from a first acceleration sensor among a plurality of sensors, an operation of generating first motion recognition information based on inputting the first signal and the first motion information into an artificial intelligence model (e.g., a machine learning model), and an operation of turning on a display based on the generated first motion recognition information.

[0010] According to one embodiment, an electronic device includes a plurality of sensors and one or more processors, wherein the one or more processors obtain a first motion signal of the electronic device through at least one of the plurality of sensors, extract first motion information of the electronic device at a specified time interval through the first motion signal, and the first motion information includes movement direction information of the electronic device and final posture information of the electronic device at an end point of the specified time interval, and can distinguish two or more different motions of the electronic device based on the first motion information.

[0011] A method performed by an electronic device according to one embodiment includes an operation of acquiring a first motion signal of the electronic device through at least one of a plurality of sensors, an operation of extracting first motion information of the electronic device at a specified time interval through the first motion signal, and an operation of distinguishing two or more different motions of the electronic device based on the first motion information, wherein the first motion information may include movement direction information of the electronic device and final posture information of the electronic device at an end point of the specified time interval.

[0012] According to various embodiments disclosed in the present document, an electronic device may be provided that performs a pick-up-and-turn operation not only when a user holds the electronic device vertically and lifts the electronic device while standing, but also when a user holds the electronic device horizontally and lifts the electronic device, and lifts the electronic device while lying down. In addition, the electronic device may extract features corresponding to a pick-up operation from sensor data acquired using a sensor included in the electronic device, train an artificial intelligence model (e.g., a machine learning model) based on the extracted features, and infer in real time a user's pick-up-and-turn operation for the electronic device based on the trained artificial intelligence model.

[0013] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0014] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0015] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device within a network environment according to various embodiments.

[0016] FIG. 2 is a diagram illustrating a lift-up-wake-up operation performed in an electronic device according to a comparative example.

[0017] FIGS. 3A through 3D illustrate scenarios in which a user picks up an electronic device according to various embodiments.

[0018] FIG. 4 is a flowchart schematically illustrating a method for an electronic device to perform a lift-up-wake-up operation based on an artificial intelligence model according to various embodiments.

[0019] FIG. 5 is a block diagram schematically illustrating a pick-up and turn-on operation performed in an electronic device according to various embodiments.

[0020] FIG. 6 is a block diagram specifically explaining a pick-up and turn-on operation performed in an electronic device according to various embodiments.

[0021] FIGS. 7A to 7C illustrate sensor signals that appear when motion occurs in an electronic device according to various embodiments.

[0022] FIG. 8 is a diagram illustrating an electronic device according to various embodiments extracting a feature corresponding to motion from a sensor signal.

[0023] FIGS. 9A and 9B illustrate signals that may be mistaken for signals corresponding to a lifting motion among sensor signals obtainable from an electronic device according to various embodiments.

[0024] FIG. 10 illustrates an electronic device according to various embodiments.

[0025] FIGS. 11 and 12 illustrate an electronic device and another electronic device communicating with at least one of a wired and wireless method according to various embodiments.

[0026] FIG. 13 is a drawing for explaining the structure and operation of a processor included in an electronic device according to various embodiments.

[0027] FIG. 14 is a block diagram specifically explaining a process of training and converting an artificial intelligence model for recognizing a user's electronic device picking-up motion by an electronic device according to various embodiments.

[0028] FIG. 15 is a block diagram illustrating a process of training an artificial intelligence model in an embedded environment by an electronic device according to various embodiments.

[0029] FIG. 16 is a block diagram illustrating a process of performing a pick-up and turn-on function of an electronic device according to various embodiments.

[0030] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0035] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0036] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0037] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0038] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0039] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0040] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0041] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0042] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0043] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0044] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0046] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0047] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0048] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0049] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0050] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0051] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0052] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0053] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0054] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0055] FIG. 2 is a diagram illustrating a lift-up-wake-up operation performed in an electronic device according to a comparative example.

[0056] An electronic device (200) according to a comparative embodiment can perform a lift-up-wake-up operation. For example, a user (230) can lift the electronic device (200) to check the time through the electronic device (200). As another example, the user (230) can lift the electronic device (200) toward the user's (230) face to check an event that has occurred in the electronic device (200), such as an application push alarm, a text message, or an instant messaging message.

[0057] An electronic device (200) according to a comparative example can recognize a motion (210) of picking up the electronic device (200) performed by a user (230). For example, the electronic device (200) can include a three-axis acceleration sensor. The electronic device (200) can recognize the motion (210) of picking up the electronic device (200) using data from the three-axis acceleration sensor. For example, the electronic device (200) can recognize the motion (210) of picking up the electronic device (200) based on the angle formed by each axis (220, 221, 222) of the three-axis acceleration sensor with respect to the ground (201). For example, each of the axes (220, 221, 222) of the three-axis acceleration sensor may include a horizontal axis (220) (e.g., x-axis), a vertical axis (221) (e.g., y-axis), and a rearward axis (222) (e.g., z-axis) of the electronic device (200). For example, the electronic device (200) may extract an angle formed by each of the axes (220, 221, 222) with the ground (201). The electronic device (200) may recognize that the operation (210) of picking up the three-axis acceleration sensor data corresponds to a case where at least one of the extracted angles exceeds a specified threshold. For example, the electronic device (200) can extract the angle formed by each of the axes (220, 221, 222) of the electronic device (200) based on the case where the electronic device (200) is horizontal to the ground (201) (e.g., the rear-facing axis (222) is perpendicular to the ground (201)). The angle formed by each of the axes (220, 221, 222) of the electronic device (200) to the ground (201) can correspond to the degree of inclination of the electronic device (200) from the case where the electronic device (200) is horizontal to the ground (201). In addition, the electronic device (200) can recognize the magnitude (e.g., the amount of change) of the gravitational acceleration in each direction corresponding to each of the axes (220, 221, 222). The electronic device (200) can measure the vector sum of the magnitude of the acceleration of gravity measured on each axis (220, 221, 222) based on a state horizontal to the ground (201) (e.g., 1 G for the rear-facing axis (222)).When a lifting operation (210) is performed on the electronic device (200) from a state horizontal to the ground (201), the electronic device (200) can measure the amount of change in the gravitational acceleration recognized in each of the axes (220, 221, 222). The electronic device (200) can detect the lifting operation (210) of the electronic device (200) based on modeling the amount of change in the gravitational acceleration recognized in each of the axes (220, 221, 222). Accordingly, the electronic device (200) can detect the lifting operation (210) based on the angle formed between the ground (201) and each of the axes (220, 221, 222) and the amount of change in the gravitational acceleration measured in the direction of each of the axes (220, 221, 222). When the electronic device (200) recognizes a picking-up motion (210), it can perform a picking-up-and-turning-on motion. In other words, even if the electronic device (200) does not receive an input from the user (230) to press a button to turn on the screen of the electronic device (200), it can recognize the picking-up motion (210) and automatically turn on the screen of the electronic device (200).

[0058] However, the electronic device (200) according to the comparative example recognizes the picking up motion (210) based on real-time monitoring of the angle between each axis (220, 221, 222) of the acceleration sensor and the ground (201). In other words, since the electronic device (200) calculates the angle between each axis (220, 221, 222) of the acceleration sensor and the ground (201) in real-time, a large amount of computation may occur to recognize the picking up motion (210). In addition, the electronic device (200) according to the comparative example only recognizes the picking up motion (210) in an environment where the user holds the electronic device (200) vertically and stands perpendicular to the ground (201), and cannot recognize the picking up motion (210) in various other environments (e.g., an environment where the user is lying down, an environment where the user holds the electronic device (200) horizontally). In order for the electronic device (200) according to the comparative embodiment to recognize the picking-up motion (210) in various environments, it is necessary to set a designated threshold angle between the ground (201) and each axis (220, 221, 222) for each environment. Adding a designated threshold angle for the electronic device (200) to recognize the picking-up motion (210) in various environments may increase the complexity of the complex exception handling structure and source. In addition, the electronic device (200) according to the comparative embodiment may misrecognize the picking-up motion (210) as a picking-up motion (210) even when the user does not perform the picking-up motion (210), such as when the user is walking, sitting on a chair, or exercising. Therefore, a technology for performing the picking-up-and-turning motion of the electronic device in environments where the user picks up the electronic device in various postures is required. Hereinafter, situations where the picking-up-and-turning motion of the electronic device is required will be described as examples.

[0059] FIGS. 3A through 3D illustrate scenarios in which a user picks up an electronic device according to various embodiments.

[0060] FIG. 3A illustrates a first situation (300) in which a user picks up an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment while lying down in a vertical direction (e.g., in the Z-axis direction) relative to the ground. In the first situation (300), the user can move the electronic device from a first point (301) to a second point (302) while lying down in a direction toward the ceiling or sky (e.g., in the Z-axis direction).

[0061] FIG. 3B illustrates a second situation (310) in which a user picks up an electronic device (e.g., electronic device (101) of FIG. 1) according to an embodiment while lying on the user's right side (e.g., in the negative direction of the x-axis). In the second situation (310), the user can move the electronic device from a third point (311) to a fourth point (312) while lying on the user's right side (e.g., in the negative direction of the x-axis).

[0062] FIG. 3c illustrates a third situation (320) in which a user picks up an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment while lying on the user's left side (e.g., in the positive direction of the x-axis). In the third situation (320), the user can move the electronic device from a fifth point (321) to a sixth point (322) while lying on the user's left side (e.g., in the positive direction of the x-axis).

[0063] In other words, as illustrated in FIGS. 3A to 3C , a user can lie down and pick up the electronic device according to an embodiment in various postures. Therefore, the electronic device according to an embodiment needs to recognize the user's picking-up motion in the first situation (300), the second situation (310), and the third situation (320) and perform a picking-up-and-turning-on motion.

[0064] FIG. 3D illustrates examples of situations where a user picks up an electronic device in various orientation states.

[0065] A user may pick up an electronic device according to one embodiment in a first orientation state (351). For example, the first orientation state (351) may indicate a state in which the user holds the electronic device vertically. For example, the electronic device may turn on the display in portrait mode in the first orientation state (351). In a fourth situation (350), the user may rotate the electronic device from the first orientation state (351) to a second orientation state (352). For example, the second orientation state (352) may indicate a state in which the user holds the electronic device horizontally. In the fourth situation (350), the electronic device may convert the screen displayed on the display from portrait mode to landscape mode based on the rotation of the electronic device. For another example, the user may directly pick up the electronic device in the second orientation state (352) in a fifth situation (360).

[0066] In other words, the electronic device needs to immediately turn on the screen corresponding to the second orientation state (352) not only when the user picks up the electronic device in the first orientation state (351), but also when the user picks up the electronic device directly in the second orientation state (352).

[0067] Below, the electronic device specifically describes how to turn on a screen corresponding to each of the user's actions of picking up the electronic device while standing, picking up the electronic device while lying down, picking up the electronic device vertically, and picking up the electronic device horizontally, and how to do so.

[0068] FIG. 4 is a flowchart schematically illustrating a method for an electronic device to perform a lift-up-wake-up operation based on an artificial intelligence model (e.g., a machine learning model) according to various embodiments.

[0069] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0070] According to one embodiment, operations (410) to (430) may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0071] In operation (410), an electronic device (e.g., electronic device (101) of FIG. 1) can extract first motion information of a motion section from a first signal obtained from a first acceleration sensor among a plurality of sensors.

[0072] An electronic device according to various embodiments may be a device that turns on a display based on recognizing a motion of a user picking up the electronic device in various postures. For example, when a user picks up the electronic device horizontally while lying down, the electronic device may recognize the user's picking up motion and display screen information in landscape mode on the display. The electronic device may obtain a sensor signal corresponding to the user's picking up the electronic device based on a plurality of sensors. For example, the electronic device may obtain an acceleration signal based on an acceleration sensor. For example, when the electronic device is stationary on a table with its display facing upward, the acceleration sensor may obtain a sensor signal corresponding to a gravitational acceleration of '-1G' toward the rear of the electronic device. Based on the user picking up the electronic device while it is stationary on the table, the acceleration sensor of the electronic device may detect a change in the acceleration signal. Thereafter, when the user ends the picking up motion (e.g., when the electronic device is stationary in front of the user's face for a predetermined period of time), the acceleration sensor may obtain an acceleration signal that is maintained within a specific acceleration value range corresponding to the user's posture. Although the acceleration sensor has been used as an example for explanation, the types of sensors included in the electronic device are not limited thereto, and furthermore, the sensor signal corresponding to the user's motion of picking up the electronic device is not limited to the acceleration signal. The sensor signal that the electronic device acquires based on the user's motion of picking up the electronic device may be referred to as a first signal or a first motion signal. The electronic device may extract first motion information from a motion section of the first signal. The motion section may refer to a section in the first signal that forms a certain pattern in response to the user's motion when the user performs a motion on the electronic device.A motion section of a first signal (or a first motion signal) may be referred to as a designated time section of a sensor signal. For example, if the first signal (or the first motion signal) is maintained at a first signal value (e.g., 0) and then changes according to a user's picking-up motion and then is maintained at a second signal value (e.g., 3), the motion section (or designated time section) may represent a time period from when the first signal starts to change from the first signal value to when it converges to the second signal value. A specific description of the motion section is described in detail with reference to FIGS. 7A, 7B, 7C, and 8 below. The first motion information may represent information including characteristics of a signal generated according to a user's picking-up motion of an electronic device. For example, the first motion information may include peak-related information acquired in the motion section. For example, the first motion information may include kurtosis for a peak value in the motion section, a slope between peak values, and a difference in signal values ​​at a start point and an end point of the motion section. The extraction of the sensor signal waveform and the first motion information obtained according to the user's electronic device picking-up action in the electronic device is specifically described below in FIGS. 7a, 7b, 7c and 8.

[0073] In operation (420), the electronic device according to various embodiments may generate first motion recognition information based on inputting a first signal and first motion information into an artificial intelligence model (e.g., a machine learning model).

[0074] An electronic device according to various embodiments may input first motion information, including characteristics of a signal generated according to a first signal and a user's action of picking up the electronic device, into an artificial intelligence model. The artificial intelligence model may infer a signal pattern based on the user's action of picking up the electronic device, based on the first signal and the first motion information. For example, if a user vertically picks up an electronic device that is stationary on a table, the first signal may have a certain first pattern. For another example, if a user horizontally picks up an electronic device that is stationary on a table, the first signal may have a second pattern that is different from the first pattern. The electronic device may generate first motion recognition information corresponding to the user's action of picking up the electronic device, based on inputting the first signal and the first motion information into the artificial intelligence model. The first motion recognition information may indicate information corresponding to which direction the user lifted the electronic device (e.g., vertically or horizontally relative to the ground) and which shape (e.g., horizontally or vertically) the user lifted the electronic device. The first motion recognition information may include information on the movement direction of the electronic device. For example, the first motion recognition information may include movement direction information regarding whether the electronic device moves vertically or horizontally parallel to the ground when the user performs a motion of picking up the electronic device. In addition, the first motion recognition information may include a rate of a vertical component and a horizontal component with respect to the direction in which the electronic device is lifted relative to the ground. The first motion recognition information may include attitude information of the electronic device at the end point of the motion section (or a specified time section) of the first signal (or the first motion signal). For example, the attitude information of the electronic device at the end point of the motion section may indicate final orientation information of the electronic device picked up by the user.For example, the posture information may include information indicating the current pose of the electronic device when the user picks up the electronic device and looks at the screen of the electronic device. In other words, the first motion recognition information may indicate a probability value corresponding to each of the movement direction, the rate of the movement direction, and the orientation of the electronic device when the electronic device moves. The electronic device may determine that the target posture condition corresponds to a probability value corresponding to the first motion recognition information if the probability value corresponding to the first motion recognition information is greater than or equal to a specified threshold value.

[0075] In operation (430), the electronic device according to various embodiments can turn on the display based on the generated first motion recognition information.

[0076] An electronic device according to various embodiments may display screen information of a screen orientation determined based on the generated first motion recognition information on the display. For example, the electronic device may determine the device orientation of the electronic device based on the first motion recognition information. The electronic device may display a screen corresponding to the device orientation on the display. For example, when the electronic device is in a portrait orientation, the electronic device may display screen information corresponding to a screen orientation in portrait mode on the display. For another example, when the electronic device is in a landscape orientation, the electronic device may display screen information corresponding to a screen orientation in landscape mode on the display.

[0077] For example, an electronic device can distinguish two or more different motions based on first motion information. The electronic device can determine a target motion corresponding to the first motion information among the two or more different motions based on the first motion information. The electronic device can display screen information corresponding to the determined target motion on a display. The two or more different motions can be determined based on the direction in which the user lifts the electronic device relative to the ground and the orientation of the electronic device that the user lifts.

[0078] For example, the two or more different motions may include at least two of the first to eighth motions. For example, the two or more different motions may be classified into a motion in which the user picks up the electronic device while standing and a motion in which the user picks up the electronic device while lying down. Furthermore, the two or more different motions may be classified into a motion in which the user holds the electronic device horizontally and picks it up and a motion in which the user picks it up vertically. Each of the two or more different motions may generate a specific signal pattern. For example, if the user picks up the electronic device vertically while standing, the electronic device may obtain an acceleration signal having a specific pattern. For another example, if the user picks up the electronic device horizontally while standing, the electronic device may obtain an acceleration signal having another specific pattern. The two or more different motions may be classified into the following eight motions. However, the types of motions are not limited thereto. Each motion will be described in detail below.

[0079] For example, the first motion may indicate a motion in which the final posture of the electronic device is vertical, in which the rate of the vertical component and the horizontal component with respect to the ground of the vector corresponding to the direction in which the electronic device is lifted is greater than or equal to a first threshold. For reference, the electronic device may calculate a vector from the first point to the second point in the direction of the first point when the electronic device moves from the first point to the second point. The magnitude of the vector may correspond to a displacement difference from the first point to the second point. The electronic device may divide the vector into vertical components and horizontal components with respect to the ground. The electronic device may calculate the ratio of the vertical components and the horizontal components of the divided vector. Accordingly, the electronic device may determine, based on the ratio of the vertical components and the horizontal components, whether the electronic device has moved more in the horizontal direction than in the vertical direction with respect to the ground. For example, the first motion may correspond to a motion in which a user stands upright and picks up the electronic device. When the user picks up the electronic device while standing up, the vertical distance may be longer than the horizontal distance. For example, the electronic device may determine whether the electronic device is lifted vertically or horizontally relative to the ground based on a ratio of the vertical component to the horizontal component of the displacement by which the electronic device is lifted relative to the ground.

[0080] For example, the second motion may represent a motion in which the ratio of the vertical component to the horizontal component of the direction in which the electronic device is lifted relative to the ground is less than a first threshold, such that the final posture (e.g., orientation of the electronic device) of the electronic device is horizontal. For example, the second motion may correspond to a motion in which the user stands and lifts the electronic device horizontally.

[0081] For example, the third motion may represent a motion in which the ratio of the vertical component and the horizontal component of the direction in which the electronic device is lifted relative to the ground is less than a first threshold, in which the user moves the electronic device from the right side of the user to the front of the user, and the final posture of the electronic device (e.g., the orientation of the electronic device) is vertical. For example, the third motion may represent a motion in which the user holds the electronic device vertically, looks to the right of the user, lies down, and picks up the electronic device (e.g., the second situation (310) of FIG. 3B).

[0082] For example, a fourth motion may represent a motion in which the user lifts the electronic device in the same direction as the third motion (e.g., from the user's right side to the user's front side), but the final posture of the electronic device (e.g., the orientation of the electronic device) is horizontal.

[0083] For example, the fifth motion may represent a motion in which the ratio of the vertical component and the horizontal component of the direction in which the electronic device is lifted relative to the ground is less than a first threshold, in which the user moves the electronic device from the left side of the user to the front side of the user, and the final posture of the electronic device (e.g., the orientation of the electronic device) is vertical. For example, the fifth motion may represent a motion in which the user holds the electronic device vertically, looks to the left side of the user, lies down, and picks up the electronic device (e.g., the third situation (320) of FIG. 3c).

[0084] For example, the sixth motion may represent a motion in which the user lifts the electronic device in the same direction as the fifth motion, but the final posture of the electronic device is horizontal.

[0085] For example, the seventh motion may represent a motion in which the ratio of the vertical component and the horizontal component of the direction in which the electronic device is lifted relative to the ground is less than a first threshold, in which the user moves the electronic device from one point in front of the user to another point in front of the user, and the final posture of the electronic device is vertical. For example, the seventh motion may represent a motion in which the user holds the electronic device vertically and lifts it from a position in front of the user's chest to a position in front of the user's face while lying down (e.g., the first situation (300) of FIG. 3A).

[0086] For example, the eighth motion may represent a motion in which the electronic device is lifted from the ground in the same direction as the seventh motion, but the final posture of the electronic device is in a horizontal direction.

[0087] Accordingly, the electronic device can, for example, determine the first motion as the target motion if the first motion information is information corresponding to the first motion, and perform a pick-up-and-turn operation to turn on the display in portrait mode.

[0088] FIG. 5 is a block diagram schematically illustrating a pick-up and turn-on operation performed in an electronic device according to various embodiments.

[0089] According to various embodiments, an electronic device (500) (e.g., electronic device (101) of FIG. 1) may perform a data collection operation (510) to collect a signal detected by a sensor in response to a user performing an operation of picking up and turning on the electronic device (500) in various postures (e.g., standing posture, lying down while looking at the ceiling, lying down while looking at the right side, lying down while looking at the left side). For example, a user may pick up the electronic device (500) while lying on the right side and turn on the display. For another example, a user may pick up the electronic device (500) while lying on the left side and turn on the display. For another example, a user may pick up the electronic device (500) while lying down while looking at the ceiling and turn on the display. For another example, a user may pick up the electronic device (500) vertically or horizontally. The electronic device (500) can acquire a specific pattern of sensor signals through a plurality of sensors in response to the posture in which the user picks up and turns on the electronic device (500) and the orientation of the electronic device (500) when the user picks up the electronic device (500). The specific pattern of sensor signals acquired when the user picks up the electronic device (500) is described in detail below with reference to FIGS. 7A, 7B, 7C, 8, 9A, and 9B.

[0090] An electronic device (500) according to various embodiments may provide sensor data acquired through a data collection operation (510) as training data for an artificial intelligence model (e.g., a machine learning model). Based on the sensor data, the electronic device (500) may perform a model update operation (520) that updates the weights and / or biases of the artificial intelligence model.

[0091] An electronic device (500) according to various embodiments may perform a model miniaturization operation (530) to reduce the weight of an updated artificial intelligence model based on a model update operation (520). For example, the electronic device (500) may perform a model miniaturization operation (530) for an updated artificial intelligence model based on an artificial intelligence model efficiency toolkit (AIMET). For example, the electronic device (500) may perform a model miniaturization operation (530) to quantize an artificial intelligence model including weights in a 32-bit floating point format into an artificial intelligence model including weights in an 8-bit floating point format.

[0092] An electronic device (500) according to various embodiments may perform a model conversion operation (540) on a miniaturized artificial intelligence model based on a model miniaturization operation (530) so that the miniaturized artificial intelligence model can be used in an embedded environment. For example, the electronic device (500) may perform the model conversion operation (540) based on an embedded artificial intelligence (eAI) framework. For example, the electronic device (500) may perform a model conversion operation (540) for integerizing weights of a miniaturized artificial intelligence model by taking into account the hardware characteristics of a processor included in the electronic device (e.g., the processor (120) of FIG. 1).

[0093] An electronic device (500) according to various embodiments may perform a sensor driver implementation operation (550) to enable interaction between a converted artificial intelligence model obtained based on a model conversion operation (540) and a plurality of sensors (e.g., a sensor module (176) of FIG. 1) included in the electronic device (500). For example, the electronic device (500) may generate a software-based SSC (sensors core) device driver based on the sensor driver implementation operation (550). Based on the generated SSC device driver, the electronic device (500) may extract sensor data obtained through a plurality of sensors (e.g., a sensor module (176) of FIG. 1) included in the electronic device (500) at a specific sampling cycle and input the extracted sensor data into the converted artificial intelligence model.

[0094] An electronic device (500) according to various embodiments may perform a verification operation (560) on the operation of a converted artificial intelligence model and an implemented sensor driver. For example, in the verification operation (560), the electronic device (500) may check whether the performance of the converted artificial intelligence model and / or the implemented sensor driver satisfies a specified performance. If the performance of the converted artificial intelligence model and / or the performance of the sensor driver does not meet the specified performance in the verification operation (560), the electronic device (500) may perform the sensor driver implementation operation (550) again. In other words, the electronic device (500) may repeat the verification operation (560) until the performance of the converted artificial intelligence model and / or the implemented sensor driver satisfies the specified performance.

[0095] For reference, the model update operation (520), model miniaturization operation (530), model conversion operation (540), sensor driver implementation operation (550), and verification operation (560) performed by the electronic device (500) may be included in the operation of training the artificial intelligence model by the electronic device (500). However, although FIG. 5 illustrates training the artificial intelligence model within the electronic device (500), the present invention is not limited thereto. For example, the artificial intelligence model may be trained to infer a motion corresponding to a user's motion of picking up the electronic device (500) from another electronic device outside the electronic device (500) based on sensor data acquired through the data collection operation (510) of the electronic device (500).

[0096] FIG. 6 is a block diagram specifically explaining a pick-up and turn-on operation performed in an electronic device according to various embodiments.

[0097] An electronic device (600) according to various embodiments (e.g., the electronic device (101) of FIG. 1 or the electronic device (500) of FIG. 5) may obtain sensor data (e.g., an acceleration signal, a gyro signal) through a sensor when the electronic device (600) is lifted by a user. The electronic device (600) may perform a training operation (601) to train an artificial intelligence model in an embedded environment of the electronic device (600) so that the artificial intelligence model infers the direction and state in which the electronic device (600) moved based on the obtained sensor data. The electronic device (600) may input sensor data obtained at a specified sampling cycle into the artificial intelligence model learned through the training operation (601) and perform a motion inference operation (602) of the electronic device (600). In FIG. 6, it is illustrated that the electronic device (600) performs a training operation (601) on an artificial intelligence model and then performs a motion inference operation (602) of the electronic device (600) based on the trained artificial intelligence model, but it is not limited thereto.

[0098] In the training operation (601), the electronic device (600) can perform a data collection operation (510), a model update operation (520), a model miniaturization operation (530), a model conversion operation (540), and an SSC (sensors core) driver implementation operation (550). The operations (510, 520, 530, 540, 550) listed above are identical to the operations described in FIG. 5, and therefore, redundant descriptions are omitted.

[0099] In the feature extraction operation (620), the electronic device (600) can extract feature information corresponding to the motion of the electronic device (600) from the sensor data collected through the sensor in the data collection operation (510). For example, the electronic device (600) can extract motion information of a signal section that changes based on the movement of the electronic device (600) from the collected acceleration sensor signal. The feature information (e.g., motion information) corresponding to the motion of the electronic device (600) extracted by the electronic device (600) from the sensor data is described in detail below in FIG. 8.

[0100] In the motion inference operation (602), the electronic device (600) may perform a data collection operation (635) of a specified sampling cycle based on an artificial intelligence model. For example, the electronic device (600) may collect sensor data at approximately 50 Hz.

[0101] In the input data integer conversion operation (636), the electronic device (600) can convert data collected at a specified sampling period into integers. The electronic device (600) can input the converted integer input data into an artificial intelligence model. At this time, the artificial intelligence model can include a model generated based on the model miniaturization operation (530) and / or the model conversion operation (540) included in the training operation (601). In other words, the artificial intelligence model can include a model that can be operated based on integer data. Therefore, when the electronic device (600) infers the motion of the electronic device (600) based on the artificial intelligence model, the electronic device (600) can perform the integer conversion operation (636) on data collected at a specified sampling period in order to reduce the amount of operation and increase the operation speed.

[0102] The electronic device (600) can perform a motion recognition operation (637) for inferring the motion of the electronic device (600) based on inputting integer-converted data into an artificial intelligence model. The electronic device (600) can perform a turn-on operation (580) for the display of the electronic device (600) based on recognizing the motion of the electronic device (600). However, if the electronic device (600) does not recognize the motion of the electronic device (600), the electronic device (600) can perform an operation (635) for collecting data at a specified sampling cycle again, and repeat this until the motion of the electronic device (600) is recognized.

[0103] In the motion section time length calculation operation (638), the electronic device (600) can calculate the time length of the motion section from the start point of the motion (e.g., lifting motion) for the electronic device (600) to the end point of the motion. For example, the electronic device (600) can collect the time lengths of the motion section in which the motion for display turn-on occurs from the sensor data. The electronic device (600) can perform the sampling period adjustment operation (590) for the sensor based on the collected time lengths of the motion section. For example, if a user who lifts the electronic device (600) at a fast speed lifts the electronic device (600), the time length of the motion section can be relatively shorter than a designated sampling period. The electronic device (600) can set the sampling period of the sensor to be relatively shorter than the designated sampling period in response to a user who lifts the electronic device (600) at a fast speed. Accordingly, the electronic device (600) can improve the accuracy of motion recognition based on setting the sampling period of the sensor to be relatively shorter than the specified sampling period.

[0104] Conversely, if a user who lifts the electronic device (600) slowly lifts the electronic device (600), the time length of the motion segment may be relatively longer than the specified sampling period. The electronic device (600) may set the sampling period of the sensor to be relatively longer than the specified sampling period in response to a user who lifts the electronic device (600) slowly. If the sampling period of the sensor is set to be relatively longer, the current used by the sensor may be reduced. Accordingly, the electronic device (600) may perform a personalized motion recognition operation (637) for the user of the electronic device (600) based on the sampling period adjustment operation (590), and may perform a pick-up-and-turn operation.

[0105] In the data transmission operation (641) to the big data server, the electronic device (600) may transmit data acquired while picking up and turning on the electronic device (600) to the big data server. For example, the electronic device (600) may transmit characteristic information of the motion section (e.g., kurtosis of peak values, slope between peaks, difference between the start and end points of the motion section, and time length of the motion section) to the big data server. Thereafter, the electronic device (600) may provide the data transmitted to the big data server as training data for artificial intelligence model training.

[0106] FIGS. 7A to 7C illustrate sensor signals that appear when motion occurs in an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (500) of FIG. 5, or the electronic device (600) of FIG. 6) according to various embodiments.

[0107] FIG. 7A illustrates sensor signals that appear when a user holds an electronic device vertically and performs a motion of picking up the electronic device vertically from the ground. The graph (700) is a graph representing a first signal (e.g., a 3-axis acceleration signal) acquired by a first acceleration sensor (e.g., a 3-axis acceleration sensor) of an electronic device when a user holds an electronic device vertically and performs a motion of picking up the electronic device vertically from the ground. The graph (700) includes a first-axis acceleration signal (702) (e.g., an x-axis acceleration signal), a second-axis acceleration signal (703) (e.g., a y-axis acceleration signal), and a third-axis acceleration signal (704) (e.g., a z-axis acceleration signal) included in the first signal. In the graph (700), the first signals (e.g., the first-axis acceleration signal (702), the second-axis acceleration signal (703), and the third-axis acceleration signal (704)) may have a signal change in a motion section (701) in which motion occurs in the electronic device. In other words, when a user holds the electronic device vertically and performs a motion of lifting the electronic device vertically from the ground, the sensor signal appearing in the motion section (701) may have a signal pattern shown in the graph (700).

[0108] FIG. 7B illustrates sensor signals that appear when a user holds an electronic device horizontally and performs a motion of picking up the electronic device horizontally from the ground. For example, graph (705) is a graph corresponding to a sensor signal that appears when a user lies down, holds the electronic device horizontally, and performs a motion of picking up the electronic device. Graph (705) is a graph representing a first signal (e.g., a 3-axis acceleration signal) acquired by a first acceleration sensor (e.g., a 3-axis acceleration sensor) of an electronic device when a user holds the electronic device horizontally and performs a motion of picking up the electronic device horizontally from the ground. In graph (705), the first signal may include a first-axis acceleration signal (706), a second-axis acceleration signal (707), and a third-axis acceleration signal (708). Referring to graph (705), the first signal (e.g., the first-axis acceleration signal (706), the second-axis acceleration signal (707), and the third-axis acceleration signal (708)) may have a signal pattern corresponding to a motion in which the user holds the electronic device horizontally and lifts the electronic device horizontally from the ground in the motion section (710).

[0109] FIG. 7C illustrates a sensor signal that appears when a user holds an electronic device vertically and performs a motion of picking up the electronic device horizontally on the ground. Graph (715) is a graph representing a first signal (e.g., a 3-axis acceleration signal) acquired by a first acceleration sensor (e.g., a 3-axis acceleration sensor) of an electronic device when a user holds an electronic device vertically and performs a motion of picking up the electronic device horizontally on the ground. In graph (715), the first signal may include a first-axis acceleration signal (716), a second-axis acceleration signal (717), and a third-axis acceleration signal (718). The first signal (e.g., the first-axis acceleration signal (716), the second-axis acceleration signal (717), and the third-axis acceleration signal (718)) may have a signal pattern shown in graph (715) in a motion section (720) when a user holds an electronic device vertically and performs a motion of picking up the electronic device horizontally on the ground.

[0110] In other words, referring to the graphs (700, 705, 715) illustrated in FIGS. 7A to 7C, the sensor signals acquired through the sensors of the electronic device may have signal patterns that are distinguished in the motion sections (701, 710, 720) for each direction in which the user picks up the electronic device (e.g., vertically or horizontally to the ground) and each orientation in which the user holds the electronic device (e.g., vertically or horizontally).

[0111] FIG. 8 is a diagram illustrating an electronic device according to various embodiments extracting a feature corresponding to motion from a sensor signal.

[0112] An electronic device according to various embodiments (e.g., the electronic device (101) of FIG. 1, the electronic device (500) of FIG. 5, or the electronic device (600) of FIG. 6) may extract a feature corresponding to motion from a sensor signal. A graph (800) is a graph of a first signal (e.g., an acceleration signal) acquired through a first acceleration sensor (e.g., a 3-axis acceleration sensor) of the electronic device in response to the motion of the electronic device. The first signal illustrated in the graph (800) may include a first-axis acceleration signal (860) (e.g., an x-axis acceleration signal), a second-axis acceleration signal (870) (e.g., a y-axis acceleration signal), and a third-axis acceleration signal (880) (e.g., a z-axis acceleration signal). For reference, in a 3-axis acceleration sensor, the second axis is perpendicular to the first axis, and the third axis is perpendicular to both the first axis and the second axis. Below, we focus on how an electronic device extracts features corresponding to motion (e.g., motion information) from a third-axis acceleration signal (880).

[0113] First, the electronic device according to various embodiments may determine a motion section (803) from a third-axis acceleration signal (880). For example, the electronic device may determine the motion section (803) based on monitoring the third-axis acceleration signal (880) along the time axis with a window of a specified size. While the electronic device is maintained at a certain position or in a certain state, the intensity of the third-axis acceleration signal (880) may appear within a specified intensity range. For example, when the electronic device is placed still on a table, the signal intensity of the third-axis acceleration signal (880) may appear within a specified first reference intensity range (805). Then, when the user performs an action of holding the electronic device vertically and lifting it up to the front of the user in a direction perpendicular to the ground, the intensity of the third-axis acceleration signal (880) may change in response to the user's action. When the user's action is completed, the intensity of the third-axis acceleration signal (880) may converge to a certain signal intensity value. For example, the intensity of the third-axis acceleration signal (880) may converge within a designated second reference intensity range (806). Accordingly, the electronic device may determine a point in time before a designated time from the time at which the third-axis acceleration signal (880) deviating from the designated first reference intensity range (805) is detected as the start point of the motion section (803). In addition, the electronic device may determine a point in time after a designated time from the time at which the intensity of the third-axis acceleration signal (880) is detected as converging within the designated second reference intensity range (806) as the end point of the motion section (803).

[0114] In other words, the intensity of the sensor signal acquired by the electronic device may have a designated reference intensity range corresponding to each state of the electronic device (e.g., a state where the electronic device is placed on a table, a state where the user is lying down and looking at the ceiling and holding it vertically, a state where the user is lying on the right side and holding it horizontally). Accordingly, the electronic device can determine the start and end of motion for the electronic device based on the comparison result between the intensity of the sensor signal and the designated reference intensity range.

[0115] An electronic device according to one embodiment may extract first motion information of a motion section (803) from a third-axis acceleration signal (880). For example, the electronic device may sample a signal at a specified period within the motion section (803) from the third-axis acceleration signal (880). In other words, the electronic device may sample a predetermined number of third-axis acceleration signals (880) within the motion section (803). The electronic device may determine a first peak value (801) and a second peak value (802) from among the sampled signals. For example, the first peak value (801) may correspond to a maximum value from among the sampled signals, and the second peak value (802) may correspond to a minimum value from among the sampled signals. The electronic device can calculate the kurtosis (810) for the first peak value (801) and the kurtosis (820) for the second peak value (802). For reference, the kurtosis (810) and the kurtosis (820) can represent characteristics corresponding to the speed of motion occurrence at the start and end points of the motion for the electronic device. Mathematical expression 1 below represents a calculation formula for the electronic device to calculate the kurtosis (810, 820).

[0116] [Mathematical Formula 1]

[0117]

[0118] The above mathematical formula 1 is merely an example to aid understanding, and embodiments of the present disclosure may not be limited thereto. For example, the above mathematical formula 1 may be modified, applied, or expanded in various ways.

[0119] In mathematical expression 1 represents the intensity value of the third-axis acceleration signal (880), represents the average value of the third-axis acceleration signal (880) intensity, represents the standard deviation value of the third-axis acceleration signal (880) intensity. The electronic device, Based on this, the intensity value of the third-axis acceleration signal (880) can be standardized, and the intensity value of the standardized third-axis acceleration signal (880) can be raised to the fourth power. The electronic device can calculate an expected value of the result of raising the intensity value of the standardized third-axis acceleration signal (880) to the fourth power. The calculated expected value can correspond to the kurtosis (e.g., kurtosis (810) or kurtosis (820)) of the peak value (e.g., the first peak value (801) or the second peak value (802)).

[0120] For example, the electronic device can calculate the slope (830) between the first peak value (801) and the second peak value (802) as motion information.

[0121] For example, the electronic device can extract the difference (850) (e.g., |z2-z1|) between the first reference value (840) (e.g., z2) of the third-axis acceleration signal (880) corresponding to the start time of the motion section (803) and the second reference value (841) (e.g., z1) of the third-axis acceleration signal (880) corresponding to the end time of the motion section (803) as the first motion information. For reference, the difference (850) can represent a feature corresponding to a change in the attitude (e.g., a change in position, a change in rotation) of the terminal at the start and end of the motion for the electronic device.

[0122] Although the third-axis acceleration signal (880) is mainly described in FIG. 8, the electronic device can extract motion information from the motion section (803) of the first-axis acceleration signal (860) and the second-axis acceleration signal (870) in a manner substantially identical to that of extracting motion information from the third-axis acceleration signal (880).

[0123] FIGS. 9A and 9B illustrate signals that may be mistaken for signals corresponding to a lifting motion among sensor signals obtainable from an electronic device according to various embodiments.

[0124] Electronic devices according to various embodiments (e.g., electronic device (101) of FIG. 1, electronic device (500) of FIG. 5, or electronic device (600) of FIG. 6) can capture various signals through sensors in response to movements of the electronic device. Hereinafter, signals caused by other operations than a picking-up operation of the electronic device, which are detected by sensors of the electronic device, will be described.

[0125] The graph (900) of FIG. 9A represents signals detected by a sensor of an electronic device based on the user's walking. For example, the graph (900) includes signals detected by a three-axis acceleration sensor of the electronic device. For example, the graph (900) includes a first-axis acceleration signal (901), a second-axis acceleration signal (902), and a third-axis acceleration signal (903) detected by the first-axis to third-axis acceleration sensors of the electronic device in response to the user's walking. For example, when a user holds the electronic device in his / her hand (or wears the electronic device if it is a wearable device) and walks while swinging his / her arms back and forth, the third-axis acceleration sensor of the electronic device may detect signals having a pattern similar to the first-axis acceleration signal (901), the second-axis acceleration signal (902), and the third-axis acceleration signal (903) of the graph (900).

[0126] The graph (910) of FIG. 9B represents signals detected by a sensor of an electronic device based on a user sitting on a chair. For example, the graph (910) includes a first-axis acceleration signal (911) detected by a first-axis acceleration sensor among the three-axis acceleration sensors of the electronic device, a second-axis acceleration signal (912) detected by a second-axis acceleration sensor, and a third-axis acceleration signal (913) detected by a third-axis acceleration sensor. For example, when a user of the electronic device is standing still and then sits on a chair, the third-axis acceleration sensor of the electronic device may detect signals having a pattern similar to the first-axis acceleration signal (911), the second-axis acceleration signal (912), and the third-axis acceleration signal (913) of the graph (910). For example, a signal included before a section (950) in the graph (910) may correspond to a signal acquired through a sensor while the user is standing still. Additionally, the signal included in the section (950) in the graph (910) may correspond to an acceleration signal acquired from a three-axis acceleration sensor of the electronic device while the user performs a motion of holding the electronic device and sitting on a chair.

[0127] The signals shown on the graph (900) of FIG. 9A (e.g., the first-axis acceleration signal (901), the second-axis acceleration signal (902), and the third-axis acceleration signal (903)) and the signals shown on the graph (910) of FIG. 9B (e.g., the first-axis acceleration signal (911), the second-axis acceleration signal (912), and the third-axis acceleration signal (913)) are signals acquired by actions other than the user's action of picking up the electronic device. When the electronic device acquires a sensor signal having the signal pattern shown in the graphs (900) and (910), the electronic device may determine that the user has not performed the electronic device picking up action. The signals shown on the graph (900) of FIG. 9A and the signals shown on the graph (910) of FIG. 9B may correspond to sensor signals acquired based on detecting the movement of another electronic device by a plurality of sensors included in the other electronic device, from another electronic device that communicates with the electronic device by wire or wirelessly. Other electronic devices are described below in FIGS. 11 and 12.

[0128] FIG. 10 illustrates an electronic device according to various embodiments.

[0129] An electronic device (1000) according to various embodiments (e.g., the electronic device (101) of FIG. 1, the electronic device (500) of FIG. 5, or the electronic device (600) of FIG. 6) may include a first acceleration sensor (1010), a second acceleration sensor (1020), a gyro sensor (not shown), a first light sensor (1030), and a second light sensor (1040).

[0130] According to various embodiments, the electronic device (1000) may extract first motion information of a motion section from a first signal acquired from a first acceleration sensor (1010). In addition, the electronic device (1000) may extract second motion information of a motion section from a second signal acquired from a second acceleration sensor (1020). Furthermore, the electronic device (1000) may calculate an average signal for the first signal and the second signal, and extract third motion information from the motion section of the average signal. The electronic device (1000) may generate first input data including the first signal and the first motion information. In addition, the electronic device (1000) may generate second input data including the second signal and the second motion information. The electronic device (1000) may generate posture information based on inputting at least one of the first input data and the second input data into an artificial intelligence model. For example, the electronic device may generate first motion recognition information based on inputting the first input data into an artificial intelligence model. For example, the electronic device can generate second posture information based on inputting second input data into an artificial intelligence model. For example, the electronic device (1000) can generate third posture information based on inputting an average signal of the first signal and the second signal and third motion information into the artificial intelligence model. The electronic device (1000) can determine that a motion corresponding to posture information with a highest probability value among the first motion recognition information, the second posture information, and the third posture information corresponds to the motion of the electronic device (1000). For reference, the generated posture information can indicate information about a direction in which the user moved the electronic device (1000) (e.g., vertically or horizontally) and a rotational shape of the electronic device (1000) (e.g., vertically or horizontally). The electronic device (1000) can turn on the display in response to the motion determined based on the posture information.

[0131] An electronic device (1000) according to various embodiments may include a gyro sensor (not shown). The electronic device (1000) may obtain a rotation signal of the electronic device from the gyro sensor (not shown). The rotation signal of the electronic device may include a signal corresponding to an angle by which the electronic device is rotated based on a case where the electronic device is held vertically while the user lifts the electronic device. For example, a rotation signal when the user holds the electronic device vertically may have a value of 0, and a rotation signal when the user rotates the electronic device 90 degrees counterclockwise and holds it horizontally may have a value of 90. However, 0 or 90 are merely examples of rotation signals, and the value of the rotation signal is not limited thereto. The electronic device (1000) may generate first motion recognition information including movement information of the electronic device with respect to the ground and rotation information based on the eye axis of the user of the electronic device, based on inputting the first signal acquired by the first acceleration sensor (1010), the first motion information of the motion section among the first signals, and the rotation signal into the artificial intelligence model. For reference, the movement information may represent information corresponding to the direction in which the user picks up the electronic device (1000). For example, the movement information may represent information corresponding to the user picking up the electronic device (1000) in a vertical direction with respect to the ground. As another example, the movement information may represent information corresponding to the user picking up the electronic device (1000) in a horizontal direction with respect to the ground. The rotation information may represent the degree to which the electronic device rotates with respect to the eye axis of the user when the user picks up the electronic device. The rotation information may also be referred to as rotation state information. For example, the rotation information may have a value of 0 to 3 depending on the degree of rotation with respect to the eye axis of the user.For example, if a user holds the electronic device vertically, the rotation information may correspond to 0. For example, if a user holds the electronic device in a counterclockwise rotation 90 degrees from the vertical position, the rotation information may correspond to 1. For example, if a user holds the electronic device vertically upside down (in other words, if a user holds the device in a clockwise rotation about 180 degrees from the vertical position), the rotation information may correspond to 2. For example, if a user holds the electronic device in a clockwise rotation 90 degrees from the vertical position, the rotation information may correspond to 3. Additionally, if the degree of rotation of the electronic device is unknown, the electronic device may return -1 as the rotation information.

[0132] An electronic device (1000) according to various embodiments may include a first illuminance sensor (1030) and a second illuminance sensor (1040). The electronic device (1000) may receive light (1031) through the first illuminance sensor (1030). The electronic device (1000) may receive light (1041) through the second illuminance sensor (1040). For example, the electronic device (1000) may include a first illuminance sensor (1030) that receives light (1031) received in a display direction (hereinafter, inward). In addition, the electronic device (1000) may include a second illuminance sensor (1040) that receives light (1041) received in an opposite direction of the display (hereinafter, outward). The amount of light (1031) received by the first light sensor (1030) may be referred to as the first light signal, and the amount of light (1041) received by the second light sensor (1040) may be referred to as the second light signal. The first light signal and the second light signal may change based on the movement or state of the electronic device (1000). For example, if a user places the electronic device (1000) on a table with the display facing the ceiling, the first light sensor (1030) may receive a large amount of light (1031) from the fluorescent light, and the second light sensor (1040) may receive almost no light (1041). When a user lifts an electronic device (1000) on a table in front of his / her face, the amount of light (1031) received by the first illuminance sensor (1030) may decrease, and at the same time, the amount of light (1041) received by the second illuminance sensor (1040) may increase. The electronic device (1000) may generate first motion recognition information including brightness information about the surroundings of the electronic device based on inputting the acquired first illuminance signal and / or second illuminance signal into an artificial intelligence model.In other words, the electronic device (1000) can analyze a change pattern of a light signal (e.g., a first light signal or a second light signal) acquired by a light sensor (e.g., a first light sensor (1030) or a second light sensor (1040)) during a user's picking up action of the electronic device (1000) based on an artificial intelligence model, and recognize the user's picking up action for the electronic device based on the analyzed change pattern.

[0133] FIGS. 11 and 12 illustrate an electronic device and another electronic device communicating with at least one of a wired and wireless method according to various embodiments.

[0134] Other electronic devices (1100, 1200) illustrated in FIGS. 11 and 12 represent electronic devices that communicate with at least one of a wired and a wireless electronic device (e.g., the electronic device 101 of FIG. 1, the electronic device 500 of FIG. 5, the electronic device 600 of FIG. 6, or the electronic device 1000 of FIG. 10) according to various embodiments. In order to perform a pick-up-and-turn-on operation, the electronic device needs to more accurately recognize whether the motion performed by the user with respect to the electronic device is a pick-up operation. However, the user of the electronic device may also perform a pick-up operation, including a pick-up operation, a motion of sitting on a chair while holding the electronic device, or a motion of walking while holding the electronic device. Therefore, the electronic device can accurately recognize a pick-up operation for the electronic device by taking into account signals obtained through movements of other electronic devices that communicate with the electronic device.

[0135] For example, the electronic device may receive a sensor signal acquired based on detecting the movement of the other electronic device (1100) by the acceleration sensor (1110, 1210) from the other electronic device (1100) of FIG. 11 or the other electronic device (1200) of FIG. 12. For example, when a user is walking while wearing the other electronic device (1100, 1200), the electronic device may receive a sensor signal (e.g., a signal illustrated in the graph (700) of FIG. 7A) detected based on the movement of the other electronic device (1100, 1200) by a plurality of sensors (e.g., the acceleration sensor (1110) or the acceleration sensor (1210)) included in the other electronic device (1100, 1200). The electronic device may generate auxiliary posture information for the first motion recognition information based on inputting the received sensor signal into an artificial intelligence model. The auxiliary posture information may include information indicating that the motion of the electronic device is a motion other than a motion for picking up and turning on.

[0136] FIG. 13 is a drawing for explaining the structure and operation of a processor included in an electronic device according to various embodiments.

[0137] An electronic device (1300) according to various embodiments (e.g., the electronic device (101) of FIG. 1, the electronic device (500) of FIG. 5, the electronic device (600) of FIG. 6, or the electronic device (1000) of FIG. 10) may train an artificial intelligence model based on a training operation (601). In the training operation (601) performed by the electronic device illustrated in FIG. 13, the data collection operation (510), the feature extraction operation (620), the model update operation (520), the model miniaturization operation (530), and the model conversion operation (540) overlap with those described in FIGS. 5 and 6, and thus, a description thereof is omitted.

[0138] The electronic device (1300) can perform a pick-up and turn-on operation in the electronic device (1300) based on a trained artificial intelligence model. The electronic device (1300) can include an application processor (AP) (e.g., processor (120) of FIG. 1). The application processor (1310) can include a neural processing engine (NPE) (1320). For example, the neural processing engine (1320) can include a snapdragon neural processing engine (SNPE). The electronic device (1300) can generate an SSC (sensors core) driver based on executing firmware (1330) (e.g., digital signal processor; DSP (firmware)) through the neural processing engine (1320). The application processor (1310) of the electronic device (1300) can receive a sensor signal collected from a sensor at a specified sampling period based on the generated SSC driver. The electronic device (1300) can input the sensor signal received based on the SSC driver into an artificial intelligence model (e.g., a conversion model) through the application processor (1310). The electronic device (1300) can recognize a motion of the electronic device (1300) based on the result of the artificial intelligence model. The electronic device (1300) can turn on the display based on the recognized motion, and then perform a sampling period adjustment operation (590) for the sensor based on the time length of the recognized motion. In addition, the electronic device (1300) can collect data on a case in which the display turn-on operation is performed. The electronic device (1300) can store the collected data in a big data server and provide it to a training operation (601) of the artificial intelligence model.

[0139]

[0140] FIG. 14 is a block diagram specifically explaining a process of training and converting an artificial intelligence model for recognizing a user's electronic device picking-up motion by an electronic device according to various embodiments.

[0141] An electronic device according to various embodiments (e.g., an electronic device (101) of FIG. 1, an electronic device (500) of FIG. 5, an electronic device (600) of FIG. 6, an electronic device (1000) of FIG. 10, or an electronic device (1300) of FIG. 13) may first perform an artificial intelligence model update operation (520) and a model miniaturization operation (530). The model update operation (520) and the model miniaturization operation (530) are the same as those described in FIG. 5, and thus will not be described again.

[0142] An electronic device can generate a quantized simulation model (1401) through a model miniaturization operation (530). The quantized simulation model (1401) can correspond to a model in which the size of the operation weight is reduced compared to the artificial intelligence model before the model miniaturization operation (530). For example, the model before the model miniaturization operation (530) is a model that operates with 32-bit floating points, and the quantized simulation model (1401) can correspond to a model that operates with 8-bit floating points. However, this is only an example.

[0143] The electronic device can perform an evaluation operation (1410) on the quantized simulation model (1401). For example, the electronic device can evaluate whether motion recognition is performed well by inputting a sensor signal to the quantized simulation model (1401).

[0144] In the evaluation operation (1410), if the quantized simulation model (1401) satisfies the evaluation criteria, the electronic device may perform an output output operation (1420) for the quantized simulation model. In the output output operation (1420), the electronic device may convert the quantized simulation model (1401) into a quantized model (e.g., a model that operates on fixed-point numbers). In addition, in the output output operation (1420), the electronic device may generate JSON (Javascript object notion) related to quantization settings.

[0145] An electronic device may perform a model conversion operation (540) to enable an artificial intelligence model to operate in an embedded environment. The model conversion operation (540) may include a model integerization operation (1430) and a model byte array conversion operation (1440).

[0146] For example, in a model integerization operation (1430), the electronic device can convert a quantized model into an eAI model that performs integer operations.

[0147] In the byte array conversion operation (1440), the electronic device can convert the eAI model into a byte array.

[0148] An electronic device can perform a device driver implementation operation (550) for the electronic device based on a model converted into a byte array. Since the device driver implementation operation (550) has been described in FIG. 5, a description of duplicated content will be omitted.

[0149] FIG. 15 is a block diagram illustrating a process of training an artificial intelligence model in an embedded environment by an electronic device according to various embodiments.

[0150] In the data collection operation (510), the electronic device (1500) according to various embodiments (e.g., the electronic device (101) of FIG. 1, the electronic device (500) of FIG. 5, the electronic device (600) of FIG. 6, the electronic device (1000) of FIG. 10, or the electronic device (1300) of FIG. 13) may receive a sensor signal when a pick-up-and-turn-on operation of the electronic device (1500) is performed from a plurality of electronic devices (e.g., a foldable smartphone (1501), a smart watch (1502), a smart ring (1503), a general smartphone (1504), and a big data server (1505)). In addition, the electronic device (1500) may collect a sensor signal when a pick-up-and-turn-on operation is not performed on the electronic device (1500) based on its own unit test (UT).

[0151] In the feature extraction operation (620), the electronic device (1500) can extract features (e.g., motion information) for a motion pattern from the received sensor signal. For example, the electronic device (1500) can extract first motion information representing a motion pattern feature from a first signal (e.g., an acceleration signal) received from a first acceleration sensor (e.g., acc1 of the foldable smartphone (1501)) of the foldable smartphone (1501). For example, the electronic device (1500) can extract second motion information representing a motion pattern feature from a second signal (e.g., an acceleration signal) received from an acceleration sensor (e.g., acc) of a general smartphone (1504).

[0152] In the model update operation (520), the electronic device (1500) may perform an update operation of the artificial intelligence model using the prepared input data. For example, the electronic device (1500) may provide a sensor signal and a feature of the sensor signal as training data for the artificial intelligence model. The electronic device (1500) may provide the first motion information and the second motion information extracted in the feature extraction operation (620) as training data for learning the artificial intelligence model. The prepared input data may include first input data including a first signal and first motion information, and second input data including a second signal and second motion information.

[0153] In the model miniaturization operation (530), the electronic device (1500) can quantize the weights and biases of the updated artificial intelligence model.

[0154] In the model performance verification operation (1510), the electronic device (1500) can verify the performance of the quantized artificial intelligence model based on the comparison results between the performance of the quantized artificial intelligence model and the performance of the existing model. For example, the electronic device (1500) can determine that the model miniaturization operation (530) is successful if the error between the performance of the quantized artificial intelligence model and the performance of the existing model is within a specified error range.

[0155] In the model conversion operation (540), the electronic device (1500) can convert a quantized artificial intelligence model that has passed model performance verification into an eAI model. Based on the converted model, the electronic device can perform a device driver implementation operation (550). Since the model conversion operation (540) and the device driver implementation operation (550) have been described in FIG. 5, any duplicate details will be omitted.

[0156] In the firmware generation operation (1520), the electronic device (1500) can generate firmware based on a low power artificial intelligence digital signal processor (LPAI DSP).

[0157] FIG. 16 is a block diagram illustrating a process of performing a pick-up and turn-on function of an electronic device according to various embodiments.

[0158] An electronic device (1600) according to various embodiments (e.g., an electronic device (101) of FIG. 1, an electronic device (500) of FIG. 5, an electronic device (600) of FIG. 6, an electronic device (1000) of FIG. 10, an electronic device (1300) of FIG. 13, or an electronic device (1500) of FIG. 15) can recognize a user's picking-up motion for the electronic device (1600). The electronic device (1600) can perform a picking-up-and-turning motion of the electronic device (1600) based on recognizing the user's motion for the electronic device (1600) as a picking-up motion.

[0159] In the data collection operation (1601), the electronic device (1600) can collect sensor signals from sensors and other electronic devices connected to the electronic device (1600). For example, the electronic device (1600) can collect sensor signals detected based on each of an acceleration sensor, a gyro sensor, and a light sensor included in the electronic device. The electronic device (1600) can receive signals regarding a user's actions on other electronic devices from other electronic devices that communicate with the electronic device (1600) through at least one of wireless and wired means.

[0160] In the data preprocessing operation (1602), the electronic device (1600) may preprocess the collected sensor signals. For example, the electronic device (1600) may extract motion information for a motion section of an acceleration signal from among the collected sensor signals. For example, the electronic device (1600) may extract motion information for a section in which motion of the user's electronic device (1600) is expected to have occurred from among the acceleration signals. The motion information may represent information including pattern characteristics of a sensor signal generated according to the user's motion.

[0161] In motion inference operation (1603), the electronic device (1600) can infer a user's picking-up motion with respect to the electronic device (1600) based on inputting the collected data and the preprocessed data into an artificial intelligence model. For example, the electronic device (1600) can infer a user's motion corresponding to a pattern of sensor signals based on inputting an acceleration signal, motion information extracted from the acceleration signal, a light signal, a gyro signal, and signals corresponding to motions of other electronic devices received from other electronic devices connected to the electronic device (1600) into the artificial intelligence model. For example, if the user stands and lifts the electronic device (1600) vertically, the artificial intelligence model can return a value of 0 (e.g., ret==0).

[0162] The electronic device (1600) may perform a screen-on operation (1604) in response to a user motion recognized in the motion inference operation (1603) (e.g., a motion of the user standing and vertically lifting the electronic device (1600). For example, if the electronic device (1600) recognizes that the user is lifting the electronic device horizontally, the electronic device (1600) may perform the screen-on operation (1604) in landscape mode. As another example, if the electronic device (1600) recognizes that the user is lifting the electronic device vertically, the electronic device (1600) may perform the screen-on operation (1604) in portrait mode.

[0163] The electronic device (1600) may perform an operation (1605) of transmitting data (e.g., acceleration signals, motion information extracted from acceleration signals, light signals, gyro signals, and signals corresponding to the operation of other electronic devices received from other electronic devices connected to the electronic device (1600)) used to perform a motion inference operation (1603) and a screen-on operation (1604) to a big data server. The electronic device (1600) may train an artificial intelligence model based on the data transmitted to the big data server.

[0164] The electronic device (1600) can perform a screen-on operation (1604) and perform a sensor data sampling period adjustment operation (1606) based on preprocessed data. The sampling period adjustment operation (1606) overlaps with the description of the sampling period adjustment operation (590) of FIGS. 5 and 6, and therefore, a description thereof will be omitted.

[0165] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0166] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0167] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0168] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0169] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0170] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0171] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0172] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0173] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0174] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Claims

1. In electronic devices (101; 500; 600; 1000; 1300; 1500; 1600), display (160); A plurality of sensors including a first acceleration sensor; one or more processors (120); and Includes a memory (130) that stores instructions, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Extracting the first motion information of the motion section (701; 710; 720; 803) from the first signal obtained from the first acceleration sensor, Based on inputting the first signal and the first motion information into an artificial intelligence model, first motion recognition information (posture information) is generated, Turning on the display (160) based on the first motion recognition information generated above, Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

2. In paragraph 1, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Kurtosis (810; 820) for each of the first peak value (801) and the second peak value (802) among the sampling signals sampled at a specified period within the above motion section (701; 710; 720; 803), The slope (830) between the first peak value (810) and the second peak value (820), and The difference (850) between the first reference value (840) of the first signal corresponding to the start point of the motion section (701; 710; 720; 803) and the second reference value (841) of the first signal corresponding to the end point of the motion section is extracted as the first motion information. Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

3. In any one of paragraphs 1 and 2, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Adjusting the sampling period of the first acceleration sensor based on the time length of the above motion section (701; 710; 720; 803). Electronic devices.

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Based on the comparison result between the signal intensity of the first signal and the specified first reference intensity range (805), the start point of the motion section is determined, Based on the comparison result between the signal intensity of the first signal and the specified second reference intensity range (806), the end point of the motion section is determined. Electronic devices.

5. In any one of paragraphs 1 to 4, At least one of the above plurality of sensors is a gyro sensor, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Obtaining a rotation signal of the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) from the gyro sensor, Based on inputting the first signal, the first motion information, and the rotation signal into an artificial intelligence model, first motion recognition information is generated, including movement information about the ground of the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) and rotation information based on the user's gaze axis of the electronic device (101; 500; 600; 1000; 1300; 1500; 1600). Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

6. In any one of paragraphs 1 to 5, At least one of the plurality of sensors is a luminance sensor (1030; 1040), When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Obtain a luminance signal corresponding to the light (1031; 1041) received by the above luminance sensor (1030; 1040), Based on inputting the above light signal into the artificial intelligence model, first motion recognition information including brightness information of the surroundings of the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) is generated. Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

7. In any one of paragraphs 1 to 6, At least one of the plurality of sensors is a second acceleration sensor, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Extracting second motion information of a motion section from the second signal obtained from the second acceleration sensor, Generates posture information based on inputting at least one of first input data including the first signal and the first motion information and second input data including the second signal and the second motion information into the artificial intelligence model, Based on the above detailed information, turning on the display (160), Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

8. In any one of paragraphs 1 to 7, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Receive a sensor signal obtained based on detecting movement of the other electronic device (1100; 1200) by a plurality of sensors included in the other electronic device (1100; 1200) from another electronic device (1100; 1200) that communicates with the electronic device by at least one of wired and wireless means, Based on inputting the received sensor signal into the artificial intelligence model, auxiliary posture information for the first motion recognition information is generated. Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

9. In any one of paragraphs 1 to 8, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 500; 600; 1000; 1300; 1500; 1600) causes: Displaying screen information of a screen orientation determined based on the first motion recognition information generated above on the display (160). Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

10. In any one of paragraphs 1 to 9, The first signal and the first motion information are provided as training data for the artificial intelligence model, The above first motion information is, Including the kurtosis (810; 820) for each of the first peak value (801) and the second peak value (802) among the sampling signals sampled at a specified period within the motion section (701; 710; 720; 803), the slope (830) between the first peak value (801) and the second peak value (802), and the difference (850) between the first reference value (840) of the first signal corresponding to the start time of the motion section (701; 710; 720; 803) and the second reference value (841) of the first signal corresponding to the end time of the motion section (701; 710; 720; 803). Electronic devices (101; 500; 600; 1000; 1300; 1500; 1600).

11. In a method performed by an electronic device, An operation of extracting first motion information of a motion section from a first signal obtained from a first acceleration sensor among a plurality of sensors; An operation of generating first motion recognition information based on inputting the first signal and the first motion information into an artificial intelligence model; and An action of turning on the display based on the first motion recognition information generated above Including method.

12. A computer-readable storage medium storing one or more computer programs including commands for performing the method of claim 11.

13. In electronic devices, sensor; and Containing one or more processors, One or more of the above processors, Obtaining a first motion signal of the electronic device through the sensor, Extracting first motion information of the electronic device in a time interval specified through the first motion signal, wherein the first motion information includes movement direction information of the electronic device and final posture information of the electronic device at the end of the specified time interval, Distinguishing two or more different motions of the electronic device based on the first motion information; Electronic devices.

14. In paragraph 13, The electronic device further comprises a display, One or more of the above processors, Determining a target motion among two or more different motions distinguished above, and performing a turn-on operation of the display based on the determined target motion. Electronic devices.

15. In any one of paragraphs 13 to 14, The above sensor includes a first acceleration sensor, The above first motion signal is, A first axis acceleration signal corresponding to the motion of the electronic device in the first axis direction of the first acceleration sensor; a second-axis acceleration signal corresponding to the motion of the electronic device in a second-axis direction perpendicular to the first axis, and A third-axis acceleration signal corresponding to the motion of the electronic device along a third-axis direction perpendicular to both the first and second axes. The above first motion information is, At least one of the kurtosis for two peaks in the specified time interval of at least one of the first axis acceleration signal, the second axis acceleration signal, and the third axis acceleration signal, the slope between the two peaks, and the difference in acceleration between the start time and the end time of the specified time interval is extracted from the signal. Electronic devices.

Images