Methods, apparatus, sensors, devices, and media for waking an electronic device

Abstract

The present disclosure relates to a method, apparatus, sensor, device and medium for waking up an electronic device. In the method, image data is captured via a sensor; the captured image data is processed using a deep neural network (DNN); and the electronic device is woken up according to a processing result of the DNN. Based on the above scheme, by using the DNN to process the image data captured by the sensor to wake up the electronic device, the convenience and intelligence of the device wake-up can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of information processing, and more specifically, to methods, apparatus, sensors, devices, and media for waking up electronic devices in the field of information processing. Background Technology

[0002] With the development of technology, electronic devices are gradually becoming integrated into people's lives and have increasingly diverse functions. For example, smart glasses, as wearable devices, are receiving more and more attention. They can include integrated batteries, controllers, and cameras, as well as components such as microphones, speakers, and displays, to help meet user needs such as information retrieval, communication, and entertainment.

[0003] Currently, smart glasses that are in sleep mode to maintain low power consumption are often woken up by clicking a button or inputting voice. However, clicking a button requires the user to raise their hand to operate it. Frequent hand raising can be uncomfortable and even inconvenient for users. While voice input eliminates the need to raise the hand, it is unsuitable in quiet environments. Furthermore, the accuracy of voice recognition is also a concern, especially in noisy environments where accuracy drops significantly, making it difficult to wake up the device correctly. In addition, both button clicking and voice input methods can lead to the device not being properly woken up when needed due to user oversight, resulting in missed operations. Similar problems exist for other electronic devices that are woken up by clicking buttons or inputting voice.

[0004] Therefore, there is a need to provide a new way to wake up electronic devices, which can wake up electronic devices more conveniently and intelligently. Summary of the Invention

[0005] One aspect of this disclosure relates to a method for waking up an electronic device. The method may include: capturing image data via a sensor; processing the captured image data using a deep neural network (DNN); and waking up the electronic device based on the processing result of the DNN.

[0006] Another aspect of this disclosure relates to an electronic device. The electronic device may include: a sensor configured to capture image data and process the captured image data using a deep neural network (DNN); and a processor configured to receive the processing result from the DNN from the sensor and wake up the electronic device based on the processing result.

[0007] Another aspect of this disclosure relates to a sensor. The sensor may include: a photoelectric conversion unit configured to convert captured optical signals into electrical signals; and a deep neural network (DNN) module configured to process the electrical signals output from the photoelectric conversion unit using the DNN to generate a processing result when the electronic device including the sensor is in a sleep state, the processing result including a trigger signal related to waking up the electronic device.

[0008] Another aspect of this disclosure relates to an apparatus for waking up an electronic device. The apparatus may include: a processor; and a memory, including computer program instructions, wherein the memory and computer program instructions are configured to cause the apparatus to perform operations via the processor. The operations may include: processing image data captured via sensors included in the electronic device using a deep neural network (DNN); and waking up the electronic device based on the processing result of the DNN.

[0009] Another aspect of this disclosure relates to a computer-readable storage medium storing one or more computer program instructions. According to embodiments of this disclosure, the one or more computer program instructions can cause the processing device to perform the methods described above when executed by a processing device.

[0010] The above overview is provided to summarize some exemplary embodiments to provide a basic understanding of the aspects of the subject matter described herein. Therefore, the features described above are merely examples and should not be construed as narrowing the scope or spirit of the subject matter described herein in any way. Other features, aspects, and advantages of the subject matter described herein will become apparent from the following detailed description, taken in conjunction with the accompanying drawings. Attached Figure Description

[0011] A better understanding of this disclosure can be obtained by considering the following detailed description of the embodiments in conjunction with the accompanying drawings. The same or similar reference numerals are used in the drawings to denote the same or similar parts. The accompanying drawings, together with the following detailed description, are incorporated in and form a part of this specification to illustrate embodiments of the disclosure and explain the principles and advantages of the disclosure.

[0012] in:

[0013] Figure 1 This is a schematic diagram of smart glasses, an example of an electronic device in related technologies.

[0014] Figure 2A This is a schematic diagram of the processing sequence corresponding to the button triggering method used in related technologies to wake up electronic devices by clicking a button.

[0015] Figure 2BThis is a schematic diagram of the processing sequence corresponding to the voice triggering method for waking up electronic devices using voice input in related technologies.

[0016] Figure 3 This is a flowchart of a method for waking up an electronic device according to an embodiment of the present disclosure.

[0017] Figure 4A and 4B Examples of targets that can be detected by a DNN model according to embodiments of this disclosure.

[0018] Figure 5 This is a schematic diagram of a DNN model according to an embodiment of the present disclosure.

[0019] Figure 6 This is a schematic diagram of the processing sequence corresponding to the DNN triggering method for waking up an electronic device using a DNN according to an embodiment of the present disclosure.

[0020] Figure 7 This is a schematic diagram of sensor mode switching according to an embodiment of the present disclosure.

[0021] Figure 8 This is a schematic diagram illustrating an example of an electronic device application scenario according to embodiments of the present disclosure.

[0022] Figure 9 This is a block diagram illustrating an example of a schematic configuration of an electronic device according to embodiments of the present disclosure.

[0023] Figure 10 This is a block diagram of another example of an illustrative configuration of an electronic device according to embodiments of the present disclosure.

[0024] Figure 11 This is a block diagram illustrating yet another example of a schematic configuration of an electronic device according to embodiments of the present disclosure.

[0025] While the embodiments described in this disclosure may be readily modified and alternatively implemented, specific embodiments thereof are shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the drawings and the detailed description thereof are not intended to limit the embodiments to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the claims. Detailed Implementation

[0026] The following description illustrates representative applications of the devices and methods described herein. These examples are provided merely to provide context and aid in understanding the described embodiments. Therefore, it will be apparent to those skilled in the art that the embodiments described below can be practiced without some or all of the specific details provided. In other instances, well-known process steps have not been described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are also possible, and the scope of this disclosure is not limited to these examples.

[0027] First refer to Figure 1 ,exist Figure 1 The diagram shows a smart glasses, an example of an electronic device in the related technology.

[0028] Smart glasses can integrate electronic components such as processors, memory, cameras, microphones, and displays. When a user wears smart glasses, they remain in standby mode to conserve power. After the user wakes the smart glasses, the processor starts up from sleep mode and can then provide various services, including virtual reality (VR) or augmented reality (AR), based on information from the microphone, camera, etc., thereby expanding the user's capabilities and perceptual experience. Figure 1 As shown on the left, a camera can be installed at the end of the temple of the smart glasses, as indicated by the circled part. Figure 1 The right side shows the camera as seen from the front of the smart glasses.

[0029] Currently, users can wake up electronic devices such as smart glasses using either button or voice triggers. In the button trigger method, the user raises their hand to press the wake-up button on the electronic device, thus activating it from standby mode to normal operation. However, frequently raising the hand to wake up the device is cumbersome and unpleasant for users. Figure 2A The diagram shows a schematic of the processing sequence 200-A corresponding to the button triggering method of waking up an electronic device using a button click in the related technology.

[0030] In S210-A, the button used to wake up the electronic device (e.g., a power button, a designated area, etc.) receives a button click from the user and transmits the trigger signal generated by the button click to the system-on-a-chip (SoC) in the electronic device. In S220-A, the SoC responds to the trigger signal via I... 2 The C (integrated circuit bus) sends command signals to the image sensor corresponding to the camera to enable the image sensor to function properly. In S230-A, the image sensor responds to the command signals to capture images and output images or videos. Figure 2AAs shown, the normal operation of the image sensor is triggered by clicking the button, and once triggered, the image sensor will enter the normal working state.

[0031] In voice-triggered systems, users activate electronic devices from standby mode to normal operation by speaking (e.g., uttering specific commands). However, voice input is cumbersome and inconvenient for users when frequent speaking is required to wake up the device, or when silence is needed to wake it up. Figure 2B The diagram shows a schematic of the processing sequence 200-B corresponding to the voice triggering method for waking up an electronic device using voice input.

[0032] In S210-B, the microphone in the electronic device in standby mode remains on, and when voice input is received, the input voice is analyzed (e.g., the voice is converted to text and analyzed). In response to determining that the voice is a specific instruction for waking up the electronic device, a trigger signal is transmitted to the SoC in the electronic device. In S220-B, the SoC responds to the trigger signal via I... 2 C sends a command signal to the image sensor corresponding to the camera to enable the image sensor to function properly. In S230-B, the image sensor responds to the command signal to capture an image or video. Figure 2B As shown, the normal operation of the image sensor is triggered by voice input, and once triggered, the image sensor will enter the normal operating state.

[0033] Besides being cumbersome and inconvenient, button-triggered and voice-triggered methods may also lead to users neglecting to properly activate the electronic device when needed, resulting in missed operations. To alleviate or solve at least some of these problems, embodiments of this disclosure provide a method for waking up an electronic device using a deep neural network (DNN). This method not only avoids problems caused by users raising their hands or speaking, but also automatically wakes up the electronic device based on the DNN's processing of captured image data, thereby improving the convenience and intelligence of device wake-up.

[0034] Figure 3 This is a flowchart of a method 300 for waking up an electronic device according to an embodiment of the present disclosure. Although the following description uses smart glasses as an example of an electronic device, the present disclosure is not limited thereto. Those skilled in the art will readily realize that the electronic device can be other information processing devices with image sensors, such as AR devices with cameras (head-mounted or wrist-worn AR devices), smart cameras, or devices with smart cameras (e.g., smart home appliances, smart toys, etc.).

[0035] In S310, image data is captured via a sensor.

[0036] Because image sensors (e.g.) can be installed on electronic devices. Figure 1 (As shown in the image, a camera) allows the electronic device to capture image data via an image sensor. The captured image data may include image data containing a specific target. This specific target may be pre-defined to wake the device when it is in sleep mode. Such a target can be pre-set in the electronic device's processing program or set or changed by the user as needed. When the captured image data contains this specific target, it indicates that the electronic device needs to be woken up to enter normal operating mode.

[0037] In one or more examples, the specific target could be a user gesture. For instance, a user could make a specific gesture to wake up an electronic device, allowing the sensor to capture it and then the device to be woken up through the analysis of the gesture by a DNN. Since users can make gestures without raising their hands or speaking, and only need to place the gesture within the sensor's field of view to wake up the device, this greatly simplifies user operation.

[0038] In one or more examples, the specific target can be a predetermined object that requires an action from the electronic device. For example, certain items can be pre-defined as predetermined objects, such as keys, mobile phones, computers, blackboards, beds, study desks, or gaming devices. When the DNN receives an input image containing the predetermined object, in response to detecting the predetermined object, it can wake up the electronic device to take the relevant action. Thus, even if the user is unaware that the electronic device needs to be woken up to record something, the electronic device can be automatically woken up to perform the relevant operation.

[0039] With the development of LLM (Large Language Model), its memory capabilities are becoming increasingly powerful. When a DNN detects a predetermined object, an electronic device can be activated to record information related to a specific scene or environment, for example, by capturing images. This allows for querying and / or processing using the information recorded by the electronic device and the LLM when the user forgets something, thereby expanding the functionality and capabilities of the electronic device.

[0040] For example, when a sensor captures image data containing a blackboard, the DNN can process this data to wake up an electronic device to instruct the sensor to record video and / or instruct the microphone to record audio, facilitating the recording of content in potential teaching or meeting scenarios for review or retrieval. As another example, when a sensor captures image data containing both a key and a door, the DNN can process this data to wake up an electronic device to instruct the sensor to record video and / or take a picture, allowing users to check whether the door was locked when unsure. Those skilled in the art can conceive of many similar processes, such as automatically taking pictures upon detecting famous buildings, automatically recording video upon detecting collisions caused by traffic accidents, and automatically issuing an alarm when an obstacle is detected and the user is getting closer. Since the electronic device can be automatically woken up to record relevant information based on predetermined objects, users can obtain relevant information by asking the electronic device when they forget or cannot remember something. This greatly expands the user's memory capacity, thereby improving their learning, work, and life abilities.

[0041] Image data can be captured via sensors even when electronic devices are in sleep mode, facilitating intelligent wake-up of the devices. This requires the sensors to remain on even when the electronic device is in sleep mode. Although the sensors remain on, they can consume less power (or receive less power) when the electronic device is in sleep mode, thus maintaining a low-power always-on state.

[0042] The following is a detailed description of the sensor. Sensors can be directly integrated into electronic devices. Compared to augmented reality (AR) environments where special devices need to be worn on the user's arm or fingers to detect gestures or movements and transmit the results to head-mounted devices such as smart glasses, directly capturing information such as gestures or movements through sensors within electronic devices avoids the need for additional equipment and prevents the distraction caused by such equipment, thus reducing the user's burden.

[0043] An electronic device may contain only one sensor related to image capture (also referred to as an image sensor). This sensor may have two modes: a first mode (also referred to as DNN mode) and a second mode (also referred to as viewing mode or streaming mode). The camera containing this sensor may have corresponding two modes: always-on DNN mode and streaming mode. When the electronic device is in sleep mode, the sensor is in the first mode to capture image data for input to the DNN. When the electronic device is awakened and in normal working condition, the sensor switches from the first mode to the second mode for regular image capture. Because the same sensor can have two modes and switch between them, hardware resources and costs can be saved, hardware utilization efficiency can be improved, and the need for additional equipment can be avoided. Of course, those skilled in the art will understand that an electronic device may also have two, three, or more image sensors, where at least one image sensor has the above two modes.

[0044] The power consumed (or received) by the sensor in the first mode can be less than that consumed in the second mode, which helps reduce power consumption. Although the sensor consumes less power in the first mode, it may not be able to acquire image data with the same resolution or quality as regular image capture, but such image data is sufficient to provide the DNN with relevant information to determine whether to wake up the electronic device.

[0045] In the first mode, the image data acquired by the sensor does not need to be output to external memory or processor (such as a SoC); it only needs to be output to the DNN for processing. Since the DNN can be embedded in the sensor as a software program or a hardware module that implements such a program, the sensor does not need to communicate with the outside world in the first mode, except for receiving power.

[0046] After the DNN outputs its processing results to wake up the electronic device based on the image data captured by the sensor in the first mode, the electronic device is awakened, and the sensor is automatically switched to the second mode for regular image capture. In the second mode, the sensor can output images or videos with a higher resolution than the image data in the first mode to an external memory or processor. Furthermore, since waking up the electronic device is not required in the second mode, the DNN can be disabled and stop working, so the sensor's image data does not need to be output to the DNN.

[0047] When a DNN module is included within a sensor, it avoids increasing the device's size by adding extra hardware components. In this case, in addition to a photoelectric conversion unit as in a conventional sensor, the sensor can also include a DNN module related to waking up the electronic device. The photoelectric conversion unit can be configured to convert captured light signals into electrical signals, thereby obtaining the captured image data. The DNN module can be configured to generate a processing result based on the electrical signal output from the photoelectric conversion unit when the electronic device is in a sleep state. This processing result may include a trigger signal related to waking up the electronic device. The DNN module and its operation will be described in detail below.

[0048] In S320, a DNN is used to process the captured image data.

[0049] A Directed Neural Network (DNN) can be pre-trained. A DNN can be trained to output trigger signals related to waking up an electronic device based on input image data. For example, a high-level trigger signal indicates that an electronic device (e.g., an application processor within the device) needs to be woken up, while a low-level signal indicates that it does not. The DNN can be trained using any suitable supervised, unsupervised, semi-supervised, or reinforcement learning techniques. For instance, various image types that require waking up the electronic device (e.g., specific gestures, predetermined objects) can be pre-labeled for the DNN to learn. Furthermore, the DNN can be rewarded or penalized for its image analysis results, enabling it to automatically learn which situations require waking up the electronic device.

[0050] DNNs can be implemented using various structures, such as one or more convolutional layers, one or more fully connected layers, etc., as long as they can output trigger signals related to waking up electronic devices based on the input image data. According to embodiments of this disclosure, a DNN for object detection can be used to determine whether to set the output result in a form for waking up electronic devices based on whether a predetermined target is detected. For example, a classifier containing fully connected layers can be used to implement the DNN to detect whether a target is present in the input image. As another example, a DNN can be implemented based on a pruned MobileNet SSD (Single Shot MultiBox Detector). That is, a DNN is constructed by pruning an existing MobileNet SSD model to detect targets at a certain location in the input image. The pruned DNN can have a simpler network structure, less computation and model parameters, and faster processing speed, which is beneficial for use in resource-constrained devices such as smart glasses for object detection.

[0051] Figure 4A and 4BThe diagram illustrates an example of a target that a DNN model according to an embodiment of this disclosure can detect. When a target is detected, a trigger signal for waking up an electronic device can be output. Figure 4A As shown in the box, the on-chip DNN sensor, located on the sensor, can detect gestures in various environments. Figure 4B As shown, the on-chip DNN sensor can perform human detection to identify human bodies within the sensor's field of view. Of course, the on-chip DNN sensor can also detect other objects (not shown), such as beds or blackboards.

[0052] Figure 5 The diagram shows a schematic of a DNN model according to an embodiment of the present disclosure, which is implemented based on PrunedMobileNet SSD. Figure 5 In addition to showing the DNN structure, some parameters are also illustrated. Those skilled in the art will understand that... Figure 5 The DNN structures and parameters described herein are merely examples to aid in a better understanding of this disclosure and do not constitute a limitation thereof. Those skilled in the art, with the aid of the teachings of this disclosure and in conjunction with existing machine learning techniques, can implement other DNNs that output trigger signals related to waking up electronic devices based on image data, using other model architectures and / or parameters. Furthermore, Figure 5 The example described in this paper is the detection of user gestures, but those skilled in the art will understand that DNNs can also detect other targets such as actions, animals, humans, blackboards, keys, etc.

[0053] like Figure 5As shown, the image data captured by the sensor is input as a 120*160*1 image to the DNN, where 120*160 represents the size of the input image and 1 represents the number of channels. After passing through the downsampling module and convolutional layer, an intermediate result of 15*20*48 is obtained, where 15*20 represents the feature size and 48 represents the number of channels. Subsequent parameters have similar meanings and will not be elaborated further. Next, the 15*20*48 intermediate result is input to the next-level downsampling module and convolutional layer, resulting in an 8*10*80 intermediate result. Then, the 8*10*80 intermediate result is input to the next-level downsampling module, resulting in a 4*5*80 intermediate result. This result is then input to the next-level downsampling module, resulting in a 2*3*80 intermediate result, which is then input to the next-level downsampling module, resulting in a 1*1*64 intermediate result. Each of the aforementioned intermediate results is then transformed into an intermediate result of the same size through convolution and reshape operations, and concatenated to form a feature matrix. Next, an anchoring operation is performed based on the formed feature matrix to generate a large number of detection boxes on the input image, thereby helping to locate the target in the image. The anchoring operation can have 1242 parameters to add 1242 anchor boxes to the input image. Then, non-maximum suppression (NMS) is used to filter the anchor boxes and perform an offset operation to determine the anchor box with the highest confidence that contains the target, thus obtaining the final detection result for the target, such as a gesture in the input image. The anchoring and NMS operations are existing operations that, through their processing, can accurately detect targets based on features extracted from the input image.

[0054] When the DNN detects a user gesture or other predetermined target, it sets the trigger signal to a level used to wake up the electronic device (e.g., a high level). Otherwise, if the DNN does not detect a gesture or other target that needs to wake up the electronic device, it does not change the trigger signal (e.g., it remains at a low level).

[0055] In S330, the electronic device is woken up based on the processing result of the DNN in S320.

[0056] When the DNN outputs a signal in the S320 to wake up the electronic device, the electronic device is woken up in response to the signal. For example, the chip operates when the startup pin of the electronic device's processor chip has a startup level. In one or more examples, when the trigger signal output by the DNN meets a predetermined condition (e.g., goes high), the SoC or application processor (AP) of the electronic device is woken up, thereby bringing the electronic device into normal operation.

[0057] According to embodiments of this disclosure, after detecting a target such as a user gesture or a predetermined object, the DNN can issue a trigger signal based on this, and can also output feature data related to image data. This feature data can enable the electronic device to provide corresponding operations to the user after being woken up. The trigger signal and feature data can be jointly included in the output of the DNN to provide different information to the processor of the electronic device.

[0058] For example, during training, a DNN is not only trained to output trigger signals related to waking up an electronic device based on the input image, but also trained to output relevant feature data based on the input image. For instance, when a blackboard with text is detected in the image, feature data related to the blackboard with text is output, thereby instructing the electronic device to record video and / or audio after being woken up; as another example, when non-native language text is detected in the image, feature data related to the non-native language text is output, thereby instructing the electronic device to perform language conversion to the native language after being woken up; as yet another example, when a traffic jam scene is detected in the image, feature data related to the traffic jam scene is output, thereby instructing the electronic device to replan its route; and so on.

[0059] Feature data can indicate the operations that an electronic device should perform under normal operating conditions. Using feature data processed or analyzed by a DNN can help the electronic device perform more appropriate operations, thereby improving its intelligence. For example, an electronic device can use sensors to take pictures based on the feature data output by the DNN, rather than necessarily taking pictures upon waking up. Another example is that the electronic device can provide users with information for decision-making, such as providing path guidance for users including the blind, providing translation support to convert non-native languages ​​into native languages, and providing reading support to extract viewpoints and proofs from obscure materials. Yet another example is that the electronic device can provide information to the user terminal to launch relevant applications, such as enabling smart glasses to communicate with a mobile phone to launch relevant applications on the phone. Still another example is that the electronic device can communicate with remote servers (such as cloud servers, roadside devices, etc.) to enable AI services, such as enabling AI assistants. The above operations are merely examples, and they can appear individually or in combination in different embodiments.

[0060] According to the above technical solution, by utilizing DNN to process image data captured by sensors, electronic devices can be woken up based on the processing results of the DNN. This eliminates the need for user operations such as clicking or speaking, enhancing the user experience and providing a more convenient way to wake up electronic devices. Furthermore, since waking up electronic devices is achieved through DNN, the device can be automatically woken up when it is needed but the user is unaware of the need to wake it up, thereby improving the intelligence of device wake-up.

[0061] The above describes the process of using a DNN to wake up an electronic device. The following is a reference... Figure 6 This diagram illustrates a processing sequence 600 corresponding to a DNN triggering method for waking up an electronic device using a DNN according to an embodiment of the present disclosure.

[0062] In S610, when the electronic device is in standby mode, the sensor remains on in DNN mode (first mode) and captures image data. At this time, the sensor consumes less power than it would for regular image capture, capturing lower-resolution image data while maintaining low power consumption. The sensor sends the captured image data to the DNN to determine whether to wake up the electronic device. The DNN can be a software module or a hardware module embedded in the sensor (e.g., a sensor chip). Since such a sensor containing a DNN can function as part of a camera, it eliminates the need for additional sensors or other devices to generate trigger signals (e.g., interrupt signals to the processor) to wake up the processor, thus saving hardware overhead and cost.

[0063] In S620, the DNN processes the received input image to detect targets it contains. For example, it can detect predetermined objects or gestures by using bounding boxes. Alternatively, it can detect the presence of predetermined objects or gestures using a classifier. After detecting a target for waking up the electronic device, the DNN sends a trigger signal to the processor (e.g., the SoC) to wake up the electronic device. This trigger signal can be included in a multi-byte DNN output. In addition to the trigger signal, the DNN output may also contain feature data related to the input image, which the woken and normally functioning SoC can use to perform operations related to the feature data.

[0064] It should be noted that although the SoC is used as an example of a component that receives a trigger signal to wake up in the description herein, those skilled in the art will recognize other examples of components that can receive a trigger signal to wake up, such as a CPU (Central Processing Unit), MCU (Microcontroller Unit), DSP (Digital Signal Processor), host processor, etc. The wake-up of such a processor can correspond to the wake-up of an electronic device.

[0065] In the S630, in response to receiving a trigger signal to wake it up, the SoC switches the sensor that was kept on in the S610 from DNN mode to Viewing mode to obtain images or videos through regular image capture using the same sensor. At this time, the SoC can... 2 C sends a switching signal to the sensor to enter observation mode, for example, to provide the sensor with power for regular image capture and / or to activate the sensor's interface for transmitting signals to external memory or processor.

[0066] In the S640, the sensor enters Viewing mode (second mode) to capture images and output images or videos. For example, the sensor can capture images based on instructions issued by the SoC based on feature data contained in the DNN output, or the sensor can capture images directly after being switched to Viewing mode.

[0067] exist Figure 7 A schematic diagram of sensor mode switching according to an embodiment of the present disclosure is shown. Figure 7 The sensor shown includes a photoelectric conversion unit composed of pixels and a DNN module. The photoelectric conversion unit converts incident light signals into electrical signals. The DNN module receives the converted electrical signals as input images from the photoelectric conversion unit and processes the input images to determine whether to wake up the SoC. The DNN module here is a lightweight DNN, such as one implemented using a pruned MobileNet SSD, which has less computation and parameters, consumes less power, and can be easily implemented in existing image sensors, thus facilitating low-cost implementation.

[0068] like Figure 7 As shown, regardless of whether the electronic device is in standby or normal operation, the sensor in the electronic device remains on and can be in two modes: DNN mode in standby mode and Viewing mode in normal operation mode. In DNN mode, the sensor consumes less power because it does not need to communicate with external memory or processor and can only capture lower-resolution images for DNN processing. In Viewing mode, the sensor consumes normal power for regular image capture. It should be noted that image capture in this article includes at least one of still image capture and video capture.

[0069] Before the sensor detects a target, the electronic device is in a low-power standby state, and the SoC is in a sleep state, but the sensor remains on at low power to search for the target. In this case, the photoelectric conversion unit captures image data and inputs the image data as an input image to the DNN. When the DNN determines that there is no need to wake up the controller, it can stop producing output and continue processing the input image from the photoelectric conversion unit to detect the presence of a predetermined target (e.g., a gesture).

[0070] When the DNN detects a target based on the input image from the photoelectric conversion unit, the DNN outputs a trigger signal to wake up the SoC (e.g., an interrupt signal to the SoC), and can also output DNN results (e.g., feature data) to cause the SoC to take appropriate action. The trigger signal can be passed to the SoC via a GPO (Group Policy Object), and the DNN results can be passed via I... 2 C or I 3 The C (Inter-Integrated Circuit Communication Interface) is passed to the SoC. Of course, the trigger signal and the result can also be passed to the SoC as a whole.

[0071] Upon receiving a trigger signal, the SoC is woken up and enters normal operating mode at normal power consumption. At this time, the SoC can switch the sensor from DNN mode to Viewing mode, allowing the sensor to output an image stream to the SoC. In Viewing mode, the DNN can cease operation, and the photoelectric conversion unit can send the data obtained from image capture to the SoC for processing. The SoC can receive image output from the sensor in Viewing mode and analyze the image output using LLM to perform corresponding operations. For example, the SoC can recognize gestures, faces, predetermined objects, or scenes from images captured by the photoelectric conversion unit to instruct relevant components to take pictures, tell the user what is in front of them, access relevant applications, and use LLM to analyze the scene, etc.

[0072] exist Figure 8 The diagram illustrates an example scenario of an electronic device application according to embodiments of the present disclosure. Figure 8The example uses AI glasses as an illustration. These AI glasses are smart glasses that can be woken up via a DNN (Data Node Network). When the sensors detect a predetermined user gesture, the AI ​​glasses, which are in standby mode, are automatically woken up. After being woken up, the AI ​​glasses can communicate with a mobile terminal to launch relevant applications, such as enabling the mobile terminal to perform translation, reading support, route navigation, and payment based on the captured information transmitted by the AI ​​glasses. The AI ​​glasses can also communicate with remote servers (this communication can be conducted via or without the mobile terminal), such as cloud servers or edge servers (e.g., edge boxes). Among these remote servers, there may be more powerful LLMs (Local Level Managers) that can provide AI assistants for the AI ​​glasses users, thereby offering users various query and analysis services. Users can use gestures to wake up dormant electronic devices via the DNN to activate desired functions. For example, a user can use gestures to take photos while cycling; a user can use gestures to record videos while hiking; a user can use gestures to record videos while attending classes or meetings; a user can use gestures to translate what the sensors capture without interrupting other processing; a user can use gestures to activate their AI assistant to help summarize what they are reading; a user can use gestures to activate their AI assistant to help debug the code they are writing; a user can use gestures to activate route navigation, for example, in situations where the user has visual impairments or is in traffic jams; a user can use gestures to have the sensors scan QR codes and make payments; and so on. The operations that AI glasses awakened by a DNN can perform are not limited to these, and those skilled in the art will understand that other operations to meet user needs are also possible.

[0073] It should be noted that the various services provided to users do not necessarily require the AI ​​glasses to be activated to communicate with mobile terminals and / or remote servers, and the AI ​​glasses themselves may also process information.

[0074] The technology disclosed herein can be applied to various devices, such as smart glasses, AR devices with cameras, smart cameras, or devices with smart cameras. The specific form of the device is not limited to what is not disclosed, as long as the electronic device (e.g., its processor) can be woken up by processing images captured by a sensor using a DNN.

[0075] The foregoing has described various exemplary devices and methods according to embodiments of this disclosure. It should be understood that the operation or function of these devices can be combined with each other to achieve more or fewer operations or functions than described. Similarly, the operational steps of the methods can be combined with each other in any suitable order to similarly achieve more or fewer operations than described.

[0076] It should be understood that the machine-executable instructions in a machine-readable storage medium or program product according to embodiments of this disclosure can be configured to perform operations corresponding to the above-described device and method embodiments. Embodiments of the machine-readable storage medium or program product will be clear to those skilled in the art when referring to the above-described device and method embodiments, and therefore will not be described again. Machine-readable storage media and program products used to carry or include the above-described machine-executable instructions also fall within the scope of this disclosure. Such storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0077] Furthermore, it should be understood that the aforementioned series of processes and devices can also be implemented via software and / or firmware. In the case of implementation via software and / or firmware, data can be transferred from storage media or networks to computers with dedicated hardware architectures, such as… Figure 9 The example electronic device 1300 shown is equipped with programs that constitute the software, and when various programs are installed, the device is able to perform various functions, etc. Figure 9 This is a block diagram illustrating an example of a schematic configuration of an electronic device according to embodiments of the present disclosure.

[0078] exist Figure 9 In this system, the central processing unit (CPU) 1301 performs various processes based on the program stored in the read-only memory (ROM) 1302 or the program loaded into the random access memory (RAM) 1303 from the storage section 1308. The RAM 1303 also stores, as needed, the data required when the CPU 1301 performs various processes.

[0079] CPU 1301, ROM 1302 and RAM 1303 are connected to each other via bus 1304. Input / output interface 1305 is also connected to bus 1304.

[0080] The following components can be connected to the input / output interface 1305: input section 1306, including a keypad, etc.; output section 1307, including a display and speakers, etc.; storage section 1308, including a memory card, etc.; and communication section 1309, including a network interface card (such as a LAN card), modem, etc. The communication section 1309 can perform communication processing via a network (such as the Internet). A sensor (not shown) can also be connected to the input / output interface 1305.

[0081] As needed, drive 1310 is also connected to input / output interface 1305. Removable media 1311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 1310 as needed, so that computer programs read from them can be installed into storage section 1308 as needed.

[0082] When the above series of processes are implemented by software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1311.

[0083] Those skilled in the art will understand that such storage media are not limited to the removable medium 1311 shown, which stores a program and is distributed separately from the device to provide the program to the user. Examples of removable media 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be ROM 1302, a hard disk included in storage section 1308, etc., containing a program and distributed to the user along with the device containing them.

[0084] Figure 10 This is a block diagram illustrating another example of a schematic configuration of an electronic device 1600 to which the technologies of this disclosure can be applied. The electronic device 1600 includes a processor 1601, a memory 1602, a storage device 1603, an external connection interface 1604, a camera device 1606, a sensor 1607, a microphone 1608, an input device 1609, a display device 1610, a speaker 1611, a wireless communication interface 1612, one or more antenna switches 1615, one or more antennas 1616, a bus 1617, a battery 1618, and an auxiliary controller 1619. In one implementation, the electronic device 1600 herein may correspond to a device with a camera, such as smart glasses.

[0085] Processor 1601 may be, for example, a CPU or a System-on-a-Chip (SoC), and controls the application layer and other functions of electronic device 1600. Memory 1602 includes RAM and ROM, and stores data and programs executed by processor 1601. Storage device 1603 may include storage media such as semiconductor memory and hard disk. External connection interface 1604 is an interface for connecting external devices (such as memory cards and Universal Serial Bus (USB) devices) to electronic device 1600.

[0086] The camera device 1606 includes an image sensor (such as a charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS)) and generates captured images. The sensor 1607 may include a set of sensors, such as a measurement sensor, a gyroscope sensor, a magnetometer sensor, and an accelerometer sensor. The microphone 1608 converts sound input to the electronic device 1600 into an audio signal. The input device 1609 includes, for example, a touch sensor, keypad, keyboard, buttons, or switches configured to detect touches on the screen of the display device 1610 and receives operations or information input from the user. The display device 1610 includes a screen (such as a liquid crystal display (LCD) and an organic light-emitting diode (OLED) display) and displays the output image of the electronic device 1600. The speaker 1611 converts the audio signal output from the electronic device 1600 into sound.

[0087] The wireless communication interface 1612 supports any cellular communication scheme (such as LTE, LTE-Advanced, and NR) and performs wireless communication. The wireless communication interface 1612 typically includes, for example, a BB processor 1613 and RF circuitry 1614. The BB processor 1613 can perform, for example, encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and performs various types of signal processing for wireless communication. Meanwhile, the RF circuitry 1614 can include, for example, mixers, filters, and amplifiers, and transmits and receives wireless signals via antenna 1616. The wireless communication interface 1612 can be a single chip module on which the BB processor 1613 and RF circuitry 1614 are integrated. Figure 10 As shown, the wireless communication interface 1612 may include multiple BB processors 1613 and multiple RF circuits 1614. Although Figure 10 An example is shown in which the wireless communication interface 1612 includes multiple BB processors 1613 and multiple RF circuits 1614, but the wireless communication interface 1612 may also include a single BB processor 1613 or a single RF circuit 1614.

[0088] In addition to cellular communication schemes, wireless communication interface 1612 can support other types of wireless communication schemes, such as short-range wireless communication schemes, near-field communication schemes, and wireless local area network (LAN) schemes. In this case, wireless communication interface 1612 may include a BB processor 1613 and RF circuitry 1614 for each wireless communication scheme.

[0089] Each of the antenna switches 1615 switches the connection destination of the antenna 1616 among multiple circuits (e.g., circuits for different wireless communication schemes) included in the wireless communication interface 1612.

[0090] Each of the antennas 1616 includes one or more antenna elements (such as multiple antenna elements included in a MIMO antenna) and is used by the wireless communication interface 1612 to transmit and receive wireless signals. Figure 10 As shown, the electronic device 1600 may include multiple antennas 1616. Although Figure 10 An example is shown in which electronic device 1600 includes multiple antennas 1616, but electronic device 1600 may also include a single antenna 1616.

[0091] Furthermore, the electronic device 1600 may include an antenna 1616 for each wireless communication scheme. In this case, the antenna switch 1615 may be omitted from the configuration of the electronic device 1600.

[0092] Bus 1617 connects processor 1601, memory 1602, storage device 1603, external connection interface 1604, camera device 1606, sensor 1607, microphone 1608, input device 1609, display device 1610, speaker 1611, wireless communication interface 1612, and auxiliary controller 1619 to each other. Battery 1618 supplies power to... Figure 10 The various blocks of the illustrated electronic device 1600 are powered, and the feeders are partially shown as dashed lines in the figure. The auxiliary controller 1619 operates, for example, the minimum necessary functions of the electronic device 1600 in sleep mode.

[0093] Figure 11 This is a block diagram illustrating yet another example of a schematic configuration of an electronic device 1720 according to an embodiment of the present disclosure. The electronic device 1720 includes one or more of the following: a processor 1721, a memory 1722, a Global Positioning System (GPS) module 1724, a sensor 1725, a data interface 1726, a content player 1727, a storage medium interface 1728, an input device 1729, a display device 1730, a speaker 1731, a wireless communication interface 1733, one or more antenna switches 1736, one or more antennas 1737, and a battery 1738. In one implementation, the electronic device 1720 herein may correspond to a device with a camera, such as smart glasses.

[0094] The processor 1721 can be, for example, a CPU or a SoC, and controls various functions of the electronic device 1720. The memory 1722 includes RAM and ROM, and stores data and programs executed by the processor 1721.

[0095] GPS module 1724 uses GPS signals received from GPS satellites to measure the location (such as latitude, longitude, and altitude) of electronic device 1720. Sensor 1725 may include a set of sensors, such as a gyroscope sensor, a geomagnetic sensor, and an air pressure sensor, to collect relevant information. Sensor 1725 may also include an image sensor, for example, for waking electronic device 1720 from sleep mode based on captured image data. Data interface 1726 is connected to, for example, a data network 1741 via a terminal not shown.

[0096] Content player 1727 reproduces content stored on a storage medium (such as a memory card), which is inserted into storage medium interface 1728. Input device 1729 includes, for example, a touch sensor, button, or switch configured to detect touch on the screen of display device 1730, and receives operations or information input from the user. Display device 1730 includes a screen such as an LCD or OLED display and displays images or reproduced content for various functions. Speaker 1731 outputs sound or reproduced content for various functions.

[0097] The wireless communication interface 1733 supports any cellular communication scheme (such as LTE, LTE-Advanced, and NR) and performs wireless communication. The wireless communication interface 1733 typically includes, for example, a BB processor 1734 and RF circuitry 1735. The BB processor 1734 can perform, for example, encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and performs various types of signal processing for wireless communication. Meanwhile, the RF circuitry 1735 can include, for example, a mixer, filters, and amplifiers, and transmits and receives wireless signals via antenna 1737. The wireless communication interface 1733 can also be a chip module on which the BB processor 1734 and RF circuitry 1735 are integrated. Figure 11 As shown, the wireless communication interface 1733 may include multiple BB processors 1734 and multiple RF circuits 1735. Although Figure 11 An example is shown in which the wireless communication interface 1733 includes multiple BB processors 1734 and multiple RF circuits 1735, but the wireless communication interface 1733 may also include a single BB processor 1734 or a single RF circuit 1735.

[0098] In addition to cellular communication schemes, the wireless communication interface 1733 can support other types of wireless communication schemes, such as short-range wireless communication schemes, near-field communication schemes, and wireless LAN schemes. In this case, for each wireless communication scheme, the wireless communication interface 1733 may include a BB processor 1734 and an RF circuit 1735.

[0099] Each of the antenna switches 1736 switches the connection destination of the antenna 1737 among multiple circuits (such as circuits for different wireless communication schemes) included in the wireless communication interface 1733.

[0100] Each of the antennas 1737 includes one or more antenna elements (such as multiple antenna elements included in a MIMO antenna) and is used by the wireless communication interface 1733 to transmit and receive wireless signals. Figure 11 As shown, electronic device 1720 may include multiple antennas 1737. Although Figure 11 An example is shown in which electronic device 1720 includes multiple antennas 1737, but electronic device 1720 may also include a single antenna 1737.

[0101] Furthermore, the electronic device 1720 may include an antenna 1737 for each wireless communication scheme. In this case, the antenna switch 1736 can be omitted from the configuration of the electronic device 1720.

[0102] Battery 1738 can be a rechargeable battery and can be fed via a feeder to... Figure 11 The various blocks of the electronic device 1720 shown are powered, and the feeders are partially shown as dashed lines in the figure.

[0103] Exemplary embodiments of the present disclosure have been described above with reference to the accompanying drawings; however, the present disclosure is by no means limited to the examples described above. Various changes and modifications can be made by those skilled in the art within the scope of the appended claims, and it should be understood that such changes and modifications naturally fall within the technical scope of the present disclosure.

[0104] For example, the multiple functions included in one unit in the above embodiments can be implemented by separate devices. Alternatively, the multiple functions implemented by multiple units in the above embodiments can be implemented by separate devices respectively. In addition, one of the above functions can be implemented by multiple units. Needless to say, such a configuration is included within the scope of the present disclosure.

[0105] In this specification, the steps described in the flowchart include not only processes executed sequentially in the stated order, but also processes executed in parallel or individually, rather than necessarily sequentially. Furthermore, even within the steps of sequential processing, needless to say, the order can be appropriately altered.

[0106] While this disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of this disclosure as defined by the appended claims. Furthermore, the terms "comprising," "including," or any other variations thereof used in embodiments of this disclosure are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0107] As will be appreciated from the description herein, embodiments of this disclosure can be configured as follows:

[0108] 1. A method for waking up an electronic device, comprising:

[0109] Image data is captured via sensors;

[0110] The captured image data is processed using a deep neural network (DNN); and

[0111] The electronic device is woken up based on the processing result of the DNN.

[0112] 2. The method according to Clause 1, wherein, when the electronic device is in a sleep state, the sensor is in a first mode to capture image data for input to the DNN, wherein the power consumed by the sensor in the first mode is lower than the power consumed by the sensor in a second mode for image capture.

[0113] 3. The method described in Clause 2 further includes:

[0114] After waking up the electronic device, the sensor is switched to a second mode to capture images.

[0115] 4. The method described in Clause 3 further includes:

[0116] When the sensor is in the second mode, the DNN is disabled.

[0117] 5. The method according to Clause 1, wherein the image data includes image data containing a specific target, wherein the specific target is pre-designated for waking up the electronic device.

[0118] 6. The method according to Clause 5, wherein the image data is processed using a DNN for object detection to generate a processing result, the processing result including a trigger signal related to waking up the electronic device.

[0119] 7. The method according to Clause 6, wherein the electronic device is woken up when the trigger signal meets a predetermined condition.

[0120] 8. The method according to Clause 6, wherein the processing result further includes feature data relating to the image data, the feature data being used to enable the electronic device to provide corresponding operations to the user after being woken up.

[0121] 9. The method according to Clause 8, wherein the corresponding operation includes one or more of the following:

[0122] Image capture is performed using the sensor;

[0123] Provide information for decision-making;

[0124] Provide information to the user terminal to launch the relevant application; and

[0125] Communicate with a remote server to enable AI services.

[0126] 10. The method according to Clause 1, wherein the DNN is implemented based on pruned MobileNet SSD.

[0127] 11. The method according to Clause 1, wherein the electronic device is smart glasses, an AR device with a camera, a smart camera, or a device with a smart camera.

[0128] 12. An electronic device, comprising:

[0129] The sensor is configured to capture image data and process the captured image data using a deep neural network (DNN); and

[0130] The processor is configured to receive the processing results of the DNN from the sensor and wake up the electronic device based on the processing results.

[0131] 13. The electronic device according to Clause 12, wherein the sensor includes a photoelectric conversion unit and a DNN module.

[0132] The photoelectric conversion unit is configured to convert the captured optical signal into an electrical signal, and

[0133] The DNN module is configured to generate a processing result based on an electrical signal output from a photoelectric conversion unit when the electronic device is in a sleep state. The processing result includes a trigger signal related to waking up the electronic device.

[0134] 14. The electronic device according to Clause 13, wherein the DNN module ceases operation after the electronic device is woken up.

[0135] 15. The electronic device according to Clause 12, wherein the sensor is configured to remain on while the electronic device is in a sleep state, consuming less power than the power consumed by the sensor to capture an image, in order to capture image data for input to the DNN.

[0136] 16. The electronic device as described in Clause 12, wherein the electronic device is smart glasses, an AR device with a camera, a smart camera, or a device with a smart camera.

[0137] 17. A sensor, comprising:

[0138] A photoelectric conversion unit is configured to convert a captured optical signal into an electrical signal; and

[0139] A deep neural network (DNN) module is configured to process electrical signals output from a photoelectric conversion unit to generate a processing result when the electronic device including the sensor is in a sleep state. The processing result includes a trigger signal related to waking up the electronic device.

[0140] 18. A device for waking up an electronic device, comprising:

[0141] Processor; and

[0142] The memory includes computer program instructions, wherein the memory and computer program instructions are configured to cause the device to perform the following operations via the processor:

[0143] Image data captured via sensors included in an electronic device is processed using deep neural networks (DNNs); and

[0144] The electronic device is woken up based on the processing result of the DNN.

[0145] 19. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processing device, cause the processing device to perform the following operations:

[0146] Image data captured via sensors included in an electronic device is processed using deep neural networks (DNNs); and

[0147] The electronic device is woken up based on the processing result of the DNN.

Claims

1. A method for waking up an electronic device, comprising: Image data is captured via sensors; The captured image data is processed using a deep neural network (DNN); as well as The electronic device is woken up based on the processing result of the DNN.

2. The method according to claim 1, wherein, When the electronic device is in a sleep state, the sensor is in a first mode to capture image data for input to the DNN, wherein the power consumed by the sensor in the first mode is lower than the power consumed by the sensor in a second mode for image capture.

3. The method according to claim 2, further comprising: After waking up the electronic device, the sensor is switched to a second mode to capture images.

4. The method according to claim 3, further comprising: When the sensor is in the second mode, the DNN is disabled.

5. The method according to claim 1, wherein, The image data includes image data containing a specific target, wherein the specific target is pre-designated to wake up the electronic device.

6. The method according to claim 5, wherein, The image data is processed using a DNN for object detection to generate a processing result, which includes a trigger signal related to waking up the electronic device.

7. The method according to claim 6, wherein, When the trigger signal meets the predetermined conditions, the electronic device is woken up.

8. The method according to claim 6, wherein, The processing result also includes feature data related to the image data, which is used to enable the electronic device to provide corresponding operations to the user after being woken up.

9. The method according to claim 8, wherein, The corresponding operation includes one or more of the following: Image capture is performed using the sensor; Provide information for decision-making; Provide information to the user terminal to launch the relevant application; and Communicate with a remote server to enable AI services.

10. The method according to claim 1, wherein, The DNN is implemented based on a pruned MobileNet SSD.

Images