Method for waking up device, and system and chip

Abstract

The present application provides a method for waking up a device, and a system and a chip. The method may comprise: processing a collected first image, determining a motion area image in the first image, and by means of a primary neural network, determining whether a target object is present in the motion area image. When it is determined that the target object is present in the motion area image, by means of a secondary neural network cascaded with the primary neural network, secondary confirmation is performed on a result outputted by the primary neural network. Hence, determination on the target object is performed by means of the primary neural network and the secondary neural network, thereby reducing the false alarm frequency, reducing the system power consumption.

Description

Methods, systems, and chips for waking up devices

[0001] This application claims priority to Chinese Patent Application No. 202311702984.3, filed on December 12, 2023, entitled “Method, System and Chip for Wake-up Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and in particular to methods, systems and chips for waking up devices. Background Technology

[0003] With the advancement of technology, the way we interact with devices is changing, through human-computer interaction methods such as voice, gestures, and vision. These interaction methods lead to an increase in the amount of data computation on the devices, thus requiring these devices to use processors with stronger computing power and higher power consumption. To reduce the power consumption of these processors, smaller "wake-up" units are used to detect input from multiple sensor interfaces. Once the "wake-up" unit detects a target object, it will wake up the processor.

[0004] A common "wake-up" unit can be a pyroelectric infrared sensor (PIR), as shown in Figure 1A, which illustrates a scenario of a PIR-based wake-up device. As shown in Figure 1A, a PIR sensor is a passive sensor based on infrared radiation. It can detect the infrared radiation emitted by an object and use it as a trigger signal to wake up the detection device. The working principle of the PIR-based low-power wake-up strategy is as follows: when an object that can generate infrared radiation enters the detection area of ​​the PIR sensor, the infrared radiation is amplified by a Fresnel lens and focused onto the PIR sensor. Therefore, the PIR sensor can sense this radiation and output a trigger signal. In sleep mode, the device uses the detected trigger signal to determine whether an object has entered and wakes up the detection device to perform corresponding operations.

[0005] Because PIR sensors can wake up devices by sensing infrared radiation, they are not robust to non-target objects or changes in ambient temperature, which can easily lead to the device being woken up falsely and increase the device's power consumption. Summary of the Invention

[0006] This application provides a method, system, and chip for waking up a device, which can reduce the frequency of false wake-ups and lower system power consumption.

[0007] In a first aspect, this application provides a method for waking up a device, the method including:

[0008] Get the first image;

[0009] Determine the motion region image in the first image based on the first image;

[0010] The motion region image is input into a first neural network to obtain a first output result;

[0011] When the first output result indicates that a target object exists in the motion region image, the second neural network determines that the motion region image contains a target object and generates a trigger signal, wherein the trigger signal is used to wake up the device, and the first neural network is cascaded with the second neural network.

[0012] The first image is any frame from the video. This application allows for the initial determination of the moving region image within the image frame. Only when a target object is confirmed to exist in the moving region image based on a cascaded first and second neural network will a trigger signal be generated to wake up the device and perform corresponding operations, such as waking up the processor to identify the target object in the image. It can be seen that the first neural network first makes an initial judgment on the existence of a moving target object, filtering out false detections of both dynamic and static elements in the scene. The second neural network confirms the result of the first neural network. Only when a moving target object is confirmed will the device be woken up; otherwise, the device remains in sleep mode. This secondary confirmation by the second neural network reduces the frequency of false alarms and lowers system power consumption. Furthermore, the cascaded first and second neural networks require only one frame of data as input to both neural networks, reducing wake-up latency and increasing wake-up speed.

[0013] In one possible implementation of the first aspect, determining that the motion region image contains a target object via a second neural network includes:

[0014] The feature data of the intermediate layer of the first neural network is input into the second neural network to determine that the target object is contained in the motion region image, wherein the accuracy of the second neural network is greater than that of the first neural network.

[0015] It can be seen that the input of the second neural network comes from the intermediate layer of the first neural network, and the resolution of the feature data of the intermediate layer is smaller than that of the original image, which can reduce the input buffer of the second neural network and alleviate the buffer pressure.

[0016] In one possible implementation of the first aspect, determining the motion region image in the first image based on the first image includes:

[0017] A motion region image in the first image is determined based on the first image and the average image, wherein the first image includes an image acquired at a first time moment, and the average image is determined based on an image acquired before the first time moment.

[0018] As can be seen, when this application perceives changing targets or regions (i.e., moving area images) in an image frame within a scene, the reference frame is not the previous frame data, but rather an "average background frame" (i.e., an average image). Understandably, if the previous frame is used as the reference frame, it will be more sensitive to noise and illumination changes between consecutive frames, leading to continuous false alarms in scenes with frequent noise and illumination changes, increasing system power consumption. Using the average image as the reference frame can reduce noise and illumination changes, thereby reducing false alarms.

[0019] In one possible implementation of the first aspect, determining the motion region image in the first image based on the first image and the average image includes:

[0020] Obtain the average image, and determine the average image corresponding to the first image based on the first image and the average image;

[0021] Based on the first image and the average image corresponding to the first image, the motion region image in the first image is determined by inter-frame difference.

[0022] It can be seen that the calculation of the current average image (i.e. the average image of the first image) depends only on the current image (i.e. the first image) and the original average image. The original average image can reduce noise and illumination changes in the first image and reduce motion caused by noise and illumination changes, thereby reducing the number of false alarms.

[0023] In one possible implementation of the first aspect, the average image is an image obtained by averaging images acquired before the first moment according to preset parameters.

[0024] The preset parameters can be adaptively adjusted according to the actual needs of the scenario. For example, in low-contrast scenarios such as business applications, the sensitivity of motion detection can be improved by reducing the preset parameters.

[0025] In one possible implementation of the first aspect, after determining that the motion region image contains a target object via the second neural network and before generating the trigger signal, the method further includes:

[0026] The motion direction of the target object in the motion region image is determined to be consistent with a preset direction, wherein the preset direction includes the motion direction set by the user.

[0027] As can be seen, this application can perform targeted wake-up based on the characteristic motion state (i.e., preset direction) of the target object. For example, in the doorbell application scenario, assuming that the target object moving forward is the primary focus (such as the homeowner opening the door), and the target objects moving backward (such as the homeowner leaving) or moving left and right (such as neighbors) are secondary focus objects, the system can focus on waking up the target object in a specific motion direction in the scene (such as forward motion) and filter out the wake-up of target objects moving in other directions (such as backward motion or left and right motion). This not only achieves accurate wake-up but also reduces system power consumption.

[0028] In one possible implementation of the first aspect, determining that the motion direction of the target object in the motion region image is consistent with a preset direction includes:

[0029] Obtain the key points of the target object in the motion region image;

[0030] The direction of movement of the target object is determined based on the key points;

[0031] Determine that the direction of motion is consistent with the direction of motion.

[0032] As can be seen, key points can reflect the motion state of the target object. The method of determining the motion direction of the target object based on key points is easy to implement. Therefore, the wake-up mechanism based on the motion direction of the target object is easy to apply in real-world scenarios.

[0033] In one possible implementation of the first aspect, determining the movement direction of the target object based on the key points includes:

[0034] The key point includes at least one of a first key point, a second key point, and a third key point, wherein the first key point is a point located on the head of the target object, the second key point is a point located on the left shoulder of the target object, and the third key point is a point located on the right shoulder of the target object.

[0035] Obtain a first distance and a second distance, wherein the first distance is the perpendicular distance from the first key point to the first connecting line, the first connecting line is the connecting line between the second key point and the third key point, and the second distance is the distance between the second key point and the third key point;

[0036] The direction of motion of the target object is determined based on the ratio of the first distance to the second distance.

[0037] As can be seen, the key point in this application may be skeletal data. By analyzing the changes in multiple skeletal data of the target object, the movement direction of the target object can be comprehensively determined, which can improve the accuracy of the movement direction judgment, thereby reducing the number of false wake-ups and reducing system power consumption.

[0038] In one possible implementation of the first aspect, determining the direction of motion of the target object based on the ratio of the first distance to the second distance includes:

[0039] If the ratio of the first distance to the second distance is less than a first threshold, the direction of motion of the target object is determined to be forward or backward.

[0040] As can be seen, the first distance indicates the left-right movement trend of the target object, while the second distance indicates the forward-backward movement trend. When the first distance is less than the second distance, it means that the forward-backward movement trend of the target object is less than its left-right movement trend, and the target object is more likely to move forward or backward. The first threshold is a value determined by this application based on a large amount of data to determine the direction of movement of the target object. Therefore, the forward or backward movement of the target object determined based on the first threshold has reliability. The wake-up method based on forward or backward movement can reduce the number of false wake-ups and reduce system power consumption.

[0041] In one possible implementation of the first aspect, determining whether the target object's direction of motion is forward or backward includes:

[0042] Obtain motion region images of N frames of images acquired before the first moment, where N is a positive integer greater than or equal to 1;

[0043] Based on the second distance and the average of the second distances corresponding to the key points of the target object in the motion region image of the N frames, the motion direction of the target object is determined to be forward or backward.

[0044] As can be seen, this application estimates forward and backward motion by comparing the difference between the current frame (i.e., the first image) and the previous N frames (i.e., those acquired before the first moment), which can improve the reliability. The wake-up method based on forward and backward motion can reduce the number of false wake-ups and reduce system power consumption.

[0045] In one possible implementation of the first aspect, determining whether the target object's motion direction is forward or backward based on the average of the second distance and the motion region images of the N frames includes:

[0046] If the absolute value of the difference between the second distance and the mean is greater than the second threshold, the target object is moving forward; or,

[0047] The absolute value of the difference between the second distance and the mean is less than or equal to the second threshold, and the target object moves backward.

[0048] As can be seen, the second threshold is a value determined by this application based on a large amount of data that can determine whether the target object is moving forward or backward. Therefore, the determined forward or backward movement of the target object is reliable. The wake-up method based on forward or backward movement can reduce the number of false wake-ups and reduce system power consumption.

[0049] In one possible implementation of the first aspect, determining the direction of motion of the target object based on the ratio of the first distance to the second distance includes:

[0050] If the ratio of the first distance to the second distance is greater than or equal to a first threshold, the direction of movement of the target object is determined to be either leftward or rightward.

[0051] As can be seen, the first distance indicates the left-right movement trend of the target object, while the second distance indicates the forward-backward movement trend. When the first distance is greater than the second distance, it means that the forward-backward movement trend of the target object is greater than its left-right movement trend, and the target object is more likely to move left-right. The first threshold is a value determined by this application based on a large amount of data to determine the direction of movement of the target object. Therefore, the leftward or rightward movement of the target object determined based on the first threshold has credibility. The wake-up method based on leftward or rightward movement can reduce the number of false wake-ups and reduce system power consumption.

[0052] In one possible implementation of the first aspect, determining the movement direction of the target object based on the key points includes:

[0053] The key point includes the center point of the motion region image;

[0054] Based on the fact that the change of the center point of the motion region image on the X-axis is greater than the change on the Y-axis, the motion direction of the target object is determined to be either moving to the left or to the right.

[0055] In one possible implementation of the first aspect, determining the movement direction of the target object based on the key points includes:

[0056] The key point includes the center point of the motion region image;

[0057] Based on the fact that the change of the center point of the motion region image on the X-axis is less than or equal to the change on the Y-axis, the motion direction of the target object is determined to be forward or backward.

[0058] As can be seen, this application can determine the left-right or forward-backward movement of the target object based on the change of the center point on the motion region image. The implementation method is simple and feasible, and can improve the execution speed.

[0059] Secondly, this application provides a computing device, which includes a communication module and a processing module, wherein...

[0060] The communication module is used to acquire the first image;

[0061] The processing module is used to determine the motion region image in the first image based on the first image;

[0062] The processing module is further configured to input the motion region image into a first neural network to obtain a first output result;

[0063] The processing module is further configured to, when the first output result indicates that a target object exists in the motion region image, determine that the motion region image contains a target object through a second neural network and generate a trigger signal, wherein the trigger signal is used to wake up the device, and the first neural network is cascaded with the second neural network.

[0064] In one possible implementation of the second aspect, the processing module is specifically used for:

[0065] The feature data of the intermediate layer of the first neural network is input into the second neural network to determine that the target object is contained in the motion region image, wherein the accuracy of the second neural network is greater than that of the first neural network.

[0066] In one possible implementation of the second aspect, the processing module is specifically used for:

[0067] A motion region image in the first image is determined based on the first image and the average image, wherein the first image includes an image acquired at a first time moment, and the average image is determined based on an image acquired before the first time moment.

[0068] In one possible implementation of the second aspect, the processing module is specifically used for:

[0069] Obtain the average image, and determine the average image corresponding to the first image based on the first image and the average image;

[0070] Based on the first image and the average image corresponding to the first image, the motion region image in the first image is determined by inter-frame difference.

[0071] In one possible implementation of the second aspect, the processing module is further configured to:

[0072] The motion direction of the target object in the motion region image is determined to be consistent with a preset direction, wherein the preset direction includes the motion direction set by the user.

[0073] In one possible implementation of the second aspect, the processing module is specifically used for:

[0074] Obtain the key points of the target object in the motion region image;

[0075] The direction of movement of the target object is determined based on the key points;

[0076] Determine that the direction of motion is consistent with the direction of motion.

[0077] In one possible implementation of the second aspect, the processing module is specifically used for:

[0078] The key point includes at least one of a first key point, a second key point, and a third key point, wherein the first key point is a point located on the head of the target object, the second key point is a point located on the left shoulder of the target object, and the third key point is a point located on the right shoulder of the target object.

[0079] Obtain a first distance and a second distance, wherein the first distance is the perpendicular distance from the first key point to the first connecting line, the first connecting line is the connecting line between the second key point and the third key point, and the second distance is the distance between the second key point and the third key point;

[0080] The direction of motion of the target object is determined based on the ratio of the first distance to the second distance.

[0081] In one possible implementation of the second aspect, the processing module has the following functions:

[0082] If the ratio of the first distance to the second distance is less than a first threshold, the direction of motion of the target object is determined to be forward or backward.

[0083] In one possible implementation of the second aspect, the processing module is specifically used for:

[0084] Obtain motion region images of N frames of images acquired before the first moment, where N is a positive integer greater than or equal to 1;

[0085] Based on the second distance and the average of the second distances corresponding to the key points of the target object in the motion region image of the N frames, the motion direction of the target object is determined to be forward or backward.

[0086] In one possible implementation of the second aspect, the processing module is specifically used for:

[0087] If the absolute value of the difference between the second distance and the mean is greater than the second threshold, the target object is moving forward; or,

[0088] The absolute value of the difference between the second distance and the mean is less than or equal to the second threshold, and the target object moves backward.

[0089] In one possible implementation of the second aspect, the processing module is specifically used for:

[0090] If the ratio of the first distance to the second distance is greater than or equal to a first threshold, the direction of movement of the target object is determined to be either leftward or rightward.

[0091] In one possible implementation of the second aspect, the processing module is specifically used for:

[0092] The key point includes the center point of the motion region image;

[0093] Based on the fact that the change of the center point of the motion region image on the X-axis is greater than the change on the Y-axis, the motion direction of the target object is determined to be either moving to the left or to the right.

[0094] In one possible implementation of the second aspect, the processing module is specifically used for:

[0095] The key point includes the center point of the motion region image;

[0096] Based on the fact that the change of the center point of the motion region image on the X-axis is less than or equal to the change on the Y-axis, the motion direction of the target object is determined to be forward or backward.

[0097] Thirdly, embodiments of this application provide an electronic device including a processor, which is configured to implement the method described in any of the first aspects or the method described in any of the second aspects by executing instructions stored in a memory or by running logic circuits.

[0098] In one possible implementation, the communication device further includes a communication interface for receiving and / or sending data, and / or for providing input and / or output to the processor.

[0099] In one possible implementation, the communication device further includes a memory for storing at least one of instructions, configuration files for logic circuits, and data. Optionally, the processor and memory may also be integrated into a single device, i.e., the processor and memory can be combined.

[0100] The above embodiment illustrates an example of a processor (or general-purpose processor) that executes a method by invoking a computer-specified instruction. In practice, the processor can also be a dedicated processor, in which case the computer instructions are pre-loaded into the processor. Optionally, the processor can include both dedicated and general-purpose processors.

[0101] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on at least one processor, implement the method described in any of the first aspects or the method described in any of the second aspects.

[0102] Fifthly, this application provides a computer program product comprising computer instructions that, when executed on at least one processor, implement the method described in any of the first aspects or the method described in any of the second aspects.

[0103] Optionally, the computer program product can be a software installation package or an image package. If the aforementioned method is required, the computer program product can be downloaded and executed on a computing device.

[0104] In a sixth aspect, this application provides a chip system comprising at least one processor, a memory, and an interface circuit, wherein the memory, the interface circuit, and the at least one processor are interconnected via circuits, and the at least one memory stores a computer program; when the computer program is executed by the processor, it implements the method described in any of the first aspects or the method described in any of the second aspects.

[0105] In a seventh aspect, this application provides a communication system, which includes the communication apparatus described in the fourth aspect and the communication apparatus described in the fifth aspect.

[0106] The beneficial effects of the technical solutions provided in aspects two to seven of this application can be referred to the beneficial effects of the technical solutions in aspect one, and will not be repeated here. Attached Figure Description

[0107] Figure 1A shows a schematic diagram of a scenario based on a PIR wake-up device;

[0108] Figure 1B shows a schematic diagram of the architecture for low-power wake-up using PIR combined with motion detection.

[0109] Figure 1C shows a schematic diagram of the architecture of motion detection combined with CNN low-power wake-up;

[0110] Figure 2 is a schematic diagram of the wake-up system provided in an embodiment of this application;

[0111] Figure 3 is a schematic flowchart of a method for waking up a device according to an embodiment of this application;

[0112] Figure 4A is a flowchart illustrating a motion region image in a first image according to an embodiment of this application;

[0113] Figure 4B is a schematic flowchart of a method for determining an average image according to an embodiment of this application;

[0114] Figure 4C is a schematic diagram of key points of a target object provided in an embodiment of this application;

[0115] Figure 5 is a schematic diagram of a wake-up architecture provided in an embodiment of this application;

[0116] Figure 6 is a schematic diagram of another wake-up architecture provided in an embodiment of this application;

[0117] Figure 7 is a schematic diagram of the structure of a computing device 70 provided in an embodiment of this application;

[0118] Figure 8 shows a schematic diagram of the structure of an electronic device 80 provided in an embodiment of this application. Detailed Implementation

[0119] The embodiments of this application will now be described with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " signifies "or," for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0120] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0121] The following is a detailed analysis of the technical problems to be solved and the corresponding application scenarios of the embodiments of this application.

[0122] Object or scene recognition is a fundamental research topic in deep learning, and many deep learning optimization techniques are based on image recognition. Currently, low-power wake-up technology based on deep learning has been successfully applied in many fields such as security monitoring, doorbell access control, and autonomous driving. The biggest challenge facing low-power wake-up technology based on deep learning lies in its enormous computational and storage resource requirements. Many network miniaturization techniques, such as network pruning, network sparsity, and low-bit quantization, have emerged, making the application of deep learning possible. However, in the field of low-power edge devices, especially in the design of milliwatt and microwatt-level low-power chips, existing neural network optimization techniques still struggle to meet the computational resource and perception accuracy requirements of practical applications.

[0123] Please refer to Figure 1B, which shows a schematic diagram of the architecture for low-power wake-up using PIR combined with motion detection. The low-power wake-up architecture 100 shown in Figure 1B includes at least one of an image detection module 1001, a first-level wake-up module 1002, a second-level wake-up module 1003, and a processor 1004.

[0124] The image detection module 1001 obtains the region of interest (ROI) from the acquired images (e.g., the i-th frame image and the (i+1)-th frame image, where i is a positive integer) and inputs the ROI into the first-level wake-up module 1002. The ROI is the area to be processed delineated from the image being processed using methods such as rectangles, circles, ellipses, or irregular polygons. The ROI is usually a pre-defined area based on the range to be detected, such as entrances / exits, corridors, etc.

[0125] The first-level wake-up module 1002, acting as a first-level "wake-up" unit, detects the region of interest based on at least one of low-frame-rate smart motion detection (SMD) and a PIR sensor, detecting whether a moving object or a thermally moving object exists within the region of interest. When the first-level wake-up module 1002 detects the presence of at least one moving object or thermally moving object, an interrupt is triggered, and the second-level wake-up module 1003 performs further detection.

[0126] SMD, also known as motion detection, detects the motion of a target object by comparing pixel differences between adjacent frames using the frame difference method.

[0127] The secondary wake-up module 1003, acting as a secondary "wake-up" unit, further determines whether a target object (such as a human-shaped target) exists in the acquired image based on the high frame rate SMD. When a target object is detected, the secondary wake-up module 1003 wakes up the processor 1004 to perform corresponding processing. Conversely, if the secondary wake-up module 1003 does not detect a target object within a preset time, it returns to the primary "wake-up" state.

[0128] As can be seen, the secondary wake-up module 1003 can filter the results from the primary wake-up module 1002, thereby reducing the frequency of false wake-ups and saving power consumption.

[0129] Please refer to Figure 1C, which shows a schematic diagram of the architecture for low-power wake-up combining motion detection and CNN. The low-power wake-up architecture 101 shown in Figure 1C includes at least one of a primary wake-up module 1011, a secondary wake-up module 1012, and a processor 1013.

[0130] The first-level wake-up module 1011 is used to perform motion detection on the acquired video images (such as the i-th frame image and the i+1-th frame image). When a motion region is detected (such as a region where there is a difference between two consecutive frames), the first-level wake-up is triggered, and the second-level wake-up module 1012 performs further detection.

[0131] The secondary wake-up module 1012 is used to further detect the motion region and generate a motion region event region, which is then input into the neural network. The neural network further judges the motion region event region to determine whether a target object (such as a human-shaped target) exists within it. When a target object is detected, the secondary wake-up module 1012 wakes up the processor 1004 to perform corresponding processing.

[0132] It can be seen that the secondary wake-up module 1003 can filter the results from the primary wake-up module 1002, thereby improving the end-to-end wake-up accuracy.

[0133] In summary, the architectures shown in Figures 1B and 1C have the following problems:

[0134] 1. Uncertainty in ROI selection. The method shown in Figure 1B usually involves manually setting or using specific algorithms to detect ROI regions containing target objects in the entire scene image. However, in some doorbell and access control scenarios, target objects (such as human figures) are usually close to the acquisition device (such as a camera). Manually or algorithmically set ROI regions can easily lead to the target object being truncated, thus affecting the accuracy of target perception.

[0135] 2. High frequency of false wake-ups in motion detection. The methods shown in Figures 1B and 1C use frame difference analysis to detect pixel differences between consecutive frames in the acquired image, using this as a primary wake-up. However, frame difference analysis can only detect areas with pixel jumps in the image and cannot confirm whether the pixel jumps are caused by the movement of a target object. Therefore, falsely identified areas due to pixel differences between consecutive frames caused by the movement of non-target objects or changes in illumination may put computational and storage pressure on subsequent detection processes, and frequent primary false alarms or triggering of secondary wake-ups increase system power consumption.

[0136] Understandably, frame differencing is sensitive to noise and subtle changes in lighting between consecutive frames. Regarding noise, since frame differencing detects targets based on pixel differences, noise in the scene can be mistakenly identified as movement of the target object. This leads to continuous false alarms, increasing system power consumption and processing load. Furthermore, frame differencing is also sensitive to subtle changes in lighting. When the lighting in the scene changes slightly, the grayscale values ​​of pixels also change slightly, which may be misinterpreted as movement of the target object. Again, this leads to continuous false alarms, increasing system power consumption and processing load.

[0137] 3. Slow wake-up speed. SMD wake-up often requires at least two frames. For scenes where the target object is at the edge or moving rapidly, there may be missed wake-up issues if the target object leaves the scene. In addition, SMD is prone to misjudging the movement of non-target objects, generating a large number of misjudged areas and increasing additional storage overhead.

[0138] In view of this, this application provides a method, system, and chip for waking up a device. The method may include: processing a first image to determine a moving region image within the first image and determining whether a target object exists in the moving region image using a first-level neural network. If the presence of a target object in the moving region image is confirmed, a second-level neural network cascaded with the first-level neural network performs a secondary confirmation on the output of the first-level neural network. It can be seen that using both the first-level and second-level neural networks to determine the target object can reduce the frequency of false alarms and lower system power consumption.

[0139] The following section explains the parts related to neural networks to help those skilled in the art understand them.

[0140] (1) Deep Neural Networks (DNNs) is a broad concept that, in a sense, includes Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs). A DNN refers to a neural network containing multiple hidden layers. The neural network provided in this application embodiment may include a convolutional neural network.

[0141] (2) Convolutional Neural Network (CNN) is a multi-layered neural network. Each layer consists of multiple two-dimensional planes, and each plane consists of multiple independent neurons. The multiple neurons in each plane share weights, which reduces the number of parameters in the neural network. Currently, in CNNs, the processor usually performs convolution operations by convolving the input signal features with the weights, which is converted into matrix multiplication between the signal matrix and the weights.

[0142] (3) Convolutional kernels can be initialized as matrices of random size. During the training of a convolutional neural network, the kernels can learn to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the connections between layers of the convolutional neural network, while also reducing the risk of overfitting.

[0143] (4) A filter is a concatenation of multiple convolutional kernels, each assigned to a specific channel of the input. When the number of channels is 1, the filter is the convolutional kernel; when the number of channels is greater than 1, the filter refers to a concatenation of multiple convolutional kernels. For example, if an image is stored as a tensor in RGB format, the input includes three channels: the R matrix, the G matrix, and the B matrix (red, green, and blue, corresponding to three images of the same size). Each channel's matrix is ​​convolved with its corresponding convolutional kernel, and all the convolutional kernels corresponding to all channels form the filter. Each filter is used to extract different feature data. For example, an image has four channels (ARGB) (transparency and red, green, and blue, corresponding to four images of the same size). Assuming the convolution kernel size is 100*100, a total of 16 convolution kernels w1 to w16 are used. Kernel w1 to w4 form the first filter, kernels w5 to w8 form the second filter, kernels w9 to w12 form the third filter, and kernels w13 to w6 form the fourth filter. Different filters are used to extract different feature data from the input image. Performing a convolution operation on the ARGB image using the first filter, that is, using w1 to w4 corresponding to the four images on the four channels, yields the first image. The top-left pixel of this image is the weighted sum of the pixels within a 100*100 area in the top-left corner of the four input images, and so on. Similarly, including the other filters, the output of this layer corresponds to four "images." Each image pair is a response to different features in the original image.

[0144] (5) Convolutional neural networks can use the back propagation (BP) algorithm to correct the parameters in the initial neural network model during training, thereby reducing the reconstruction error loss of the neural network model. Specifically, forward propagation of the input signal to the output will generate error loss. By backpropagating the error loss information, the parameters in the initial neural network model are updated, thus converging the error loss. The back propagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weights and attention vectors in the embodiments of this application.

[0145] (6) Convolution is the extraction of feature data from the original input. Simply put, feature extraction involves extracting features from a small region of the original input. Mathematically, convolution is the operation between the convolution kernel and the input matrix of the convolutional layer. The input matrix is ​​typically extracted from the image matrix during convolution based on the stride of the convolution kernel. The convolution kernel is a small window that records the weights. The convolution kernel slides across the image matrix with the stride, each slide corresponding to a submatrix of the image matrix. The weights in the convolution kernel are multiplied and summed with the values ​​contained in that submatrix, and then assigned to an element in the current output feature map (output matrix). Convolution is not limited to convolution on the original input; it also includes re-convolution on the output result after convolution, which is not limited in this embodiment. For example, the first convolution extracts low-level feature data, the second convolution extracts mid-level feature data, the third convolution extracts high-level features, and so on. Features can be continuously extracted and compressed. The higher-level features obtained at the end can be understood as a further condensation of the original features, making the final features more reliable. The last layer of features can be used to process various tasks, such as classification and regression.

[0146] Next, we will introduce the application scenarios of the embodiments of this application.

[0147] Figure 2 shows a schematic diagram of the wake-up system provided in an embodiment of this application. The wake-up system 20 includes a data acquisition device 201, a data processing device 202, a working device 203, and a storage device 204. Wherein:

[0148] The acquisition device 201 may specifically be a camera, used to acquire multiple images of the surrounding environment. The camera may be a still camera or a video camera (i.e., a camcorder), or a visible light camera or an infrared camera; it can be any camera used to acquire images, and this embodiment of the application does not limit this.

[0149] The data processing device 202 is used to process the image to be identified acquired by the acquisition device 201 to identify the moving area in it, and then determine whether there is a target object in the moving area.

[0150] In one implementation, the data processing device 202 includes a primary wake-up module 2021 and a secondary wake-up module 2022. The primary wake-up module 2021, based on Smart Motion Detection (SMD), determines the motion region image in the image to be identified. Then, based on a primary CNN, it determines whether a target object exists in the motion region image. If the presence of a target object is confirmed, the secondary wake-up module 2022 is activated. The secondary wake-up module 2022, based on a secondary CNN cascaded with the primary CNN, confirms the output of the primary wake-up module 2021. If the output of the secondary CNN indicates the presence of a target object in the motion region image, the working device 203 is activated. It can be seen that using a secondary wake-up architecture reduces the frequency of false positives and lowers end-to-end system power consumption.

[0151] In one implementation, when the output of the secondary CNN indicates the presence of a target object in the motion region image, the secondary wake-up module 2022 is further used to determine the motion direction of the target object in the motion region image. When the motion direction of the target object matches a preset direction, a trigger signal is generated to wake up the working device 203 to perform corresponding work. The preset direction includes user-defined directions of interest, including but not limited to: forward movement, backward movement, left movement, and left movement.

[0152] The target object can be set by the user through the user device (not shown in Figure 2). The target object can be the object that the user wants to detect, such as human-shaped objects, vehicles, animals, etc.

[0153] The data processing device 202 can be of various types, such as cloud servers, network servers, application servers, and management servers—devices or servers with data processing capabilities. It can also be a device with communication and data processing capabilities, such as a chip, software module, or integrated circuit. The data processing device 202 receives detection requests from user equipment (not shown in Figure 2) through an interactive interface, and then performs target object detection processing in the moving area using a storage device and data processing stages, employing methods such as machine learning, deep learning, search, reasoning, and decision-making. The storage device in the data processing device is a general term, including local storage and a database storing historical data. The database can be located on the data processing device or on other network servers.

[0154] The working device 203 is used to wake up the processor to perform corresponding work when the data processing device 202 determines that there is a moving target object, such as displaying the moving target object identified by the data processing device 202, and / or generating alarm information, which is used to indicate that a moving target object has been detected.

[0155] Storage device 204 is used to store the motion region and target object in the motion region in each frame of image identified by data processing device 202.

[0156] For example, the acquisition device 201 and the data processing device 202 can be integrated into one device, such as in the same security monitoring device or the same vehicle. Alternatively, the acquisition device 201 and the data processing device 202 can be set up separately; for example, the acquisition device 201 and the data processing device 202 can be a separate camera and server.

[0157] For example, the acquisition device 201 and the data processing device 202 can be directly connected. For instance, when the acquisition device 201 and the data processing device 202 are integrated into the same device, they can be directly connected via corresponding connecting devices. Alternatively, the acquisition device 201 and the data processing device 202 can be indirectly connected. For instance, when the acquisition device 201 and the data processing device 202 are separately configured, they can be indirectly connected via wireless communication or other means.

[0158] The wake-up system shown in Figure 2 can be applied to various scenarios. The following description uses the acquisition device 201 as a camera as an example to illustrate these scenarios. In one scenario, the camera is typically positioned in a fixed location to ensure that the non-moving areas of each frame in the captured video are essentially the same, without significant changes. Three of these scenarios are described below, but this application is not limited to these three scenarios.

[0159] Scenario 1: The wake-up system shown in Figure 2 is applied to indoor or outdoor security monitoring scenarios. In homes, schools, construction sites, and other locations, to protect personal and property safety, monitoring equipment is installed in at least one location, such as corridors, doorways, and rooms. This equipment collects and displays surveillance footage. The surveillance footage indicates the image captured by the monitoring equipment and displayed on the screen.

[0160] In this scenario, the monitoring equipment can identify the motion region images in each frame of the video by waking up the system. These motion region images can then be used to locate target objects (such as human figures or animals). Once a human figure or animal is found in the motion region, the processor can be activated to perform corresponding tasks, such as identifying the target object and displaying alarm information.

[0161] Scenario 2: The wake-up system shown in Figure 2 is applied to traffic monitoring. At some highway intersections, toll gates, or crossroads, road monitoring equipment is usually installed to monitor and adjust the flow of traffic, and the monitoring equipment is used to collect and display the monitoring images.

[0162] In this scenario, the monitoring equipment uses a wake-up system to identify motion regions in each frame of the video. These motion regions can then be used to locate target objects (such as vehicles). When a particularly high number of vehicles traveling in a certain direction is detected, the processor can be activated to perform corresponding tasks, such as prompting relevant personnel to control and adjust traffic lights.

[0163] Scenario 3: The wake-up system shown in Figure 2 is applied to the field of photography. Taking the application of the wake-up system in a mobile phone as an example, in one scenario, when a user takes a picture with their mobile phone, in order to improve the shooting effect, the mobile phone can analyze the moving area image in the shooting scene based on intelligent motion detection. When a target is identified in the moving area image, the processor is woken up to perform corresponding work, such as marking and displaying the identified target object (such as a moving dog).

[0164] Please refer to Figure 3, which is a schematic flowchart of a method for waking up a device according to an embodiment of this application. This method can be applied to the system shown in Figure 2, and includes, but is not limited to, the following steps:

[0165] Step S301: Obtain the first image.

[0166] Specifically, electronic devices can acquire each frame of a video captured by a data acquisition device (such as a camera) of a target scene (such as a security monitoring scene, traffic monitoring scene, shooting scene, intelligent driving scene, etc.). The first image is a frame of the video, such as the image captured at the first moment.

[0167] It should be noted that electronic devices can be devices with communication and computing capabilities. In different scenarios, electronic devices can be different devices, such as smart cameras, monitoring doorbells, smart door locks, vehicles, and so on.

[0168] Step S302: Determine the motion region image in the first image based on the first image.

[0169] It is understandable that consecutive frames in the captured video have a continuous characteristic. If there are no moving objects in the target scene, the changes between consecutive frames are very subtle; if there are moving objects, there will be significant changes between consecutive frames. Because the objects in the target scene are moving, the positions of the object's image are different in different image frames. Therefore, the motion region image in the first image is the area where there are significant changes between the first image and other images caused by the moving object.

[0170] Please refer to Figure 4A, which is a flowchart illustrating a motion region image in a first image according to an embodiment of this application. As shown in Figure 4A, the electronic device performs Gaussian blur processing on the first image to obtain a processed first image. The resolution of the first image can be 80×64. The electronic device subtracts the processed first image from the average image to obtain a difference map (activation) of the inter-frame candidate motion region, and updates the average image by taking the average of the first image and the average image. Then, the electronic device binarizes the difference map (activation) to obtain a binary map (mask), and performs morphological processing on the binary map (mask), such as dilation and erosion, to obtain a morphologically processed binary map (mask), thereby obtaining a complete and accurate mask of the motion target region. Next, the electronic device performs grid processing on the morphologically processed binary map (mask) to obtain a grid binary map (grid mask), the resolution of which is 16×8. Finally, the electronic device performs connected component analysis (CCA) on the binary grid image to obtain bounding boxes, where the bounding boxes occupy the motion region image in the first image.

[0171] In one possible implementation, the electronic device determines the motion region image in the first image using the inter-frame difference method. Unlike the traditional inter-frame difference method, in addition to the first frame in the video, the reference frame used in this embodiment is not directly extracted from the previous frame of the current frame, but instead introduces the probability of an "average background frame" (i.e., an average image). For example, the average image is the image obtained by averaging the images acquired before the first moment.

[0172] Please refer to Figure 4B, which is a flowchart illustrating the process of determining the average image according to an embodiment of this application. As shown in Figure 4A, the i-th (i=0) frame is the first frame of the video captured after the acquisition device is powered on. Since there are no other images in the video before the i-th (i=0) frame, the i-th (i=0) frame can be directly considered as the average image. For the (i+1)-th frame after the i-th (i=0) frame, it can be averaged with the average image first, and the average of the (i+1)-th frame and the average image is used to update the average image. Therefore, except for the first frame after power-on, the average image can be updated by calculating the average value for each subsequent frame. Furthermore, the calculation of the average image of the current image depends only on the current image and the original average image.

[0173] In one implementation, the electronic device acquires an average image, determines the average image corresponding to the first image based on the first image and the average image, and then determines the moving region image in the first image using an inter-frame difference method based on the first image and the average image corresponding to the first image. That is, a difference operation is performed on the first image and the average image of the first image; the pixels corresponding to the first image and the average image of the first image are subtracted respectively, and the absolute value of the grayscale difference is determined. When the absolute value of the grayscale difference corresponding to a certain region image exceeds a certain threshold, the region image can be determined to be a moving region image.

[0174] As can be seen from Figure 4B, for the first image acquired at the first moment, before the first image is input, the average image is the average value of the images acquired before the first moment, and the average image of the first image is related to the first image and the average image before the first moment.

[0175] For example, avg_frm(i+1) = α * avg_frm(i) + (1-α) * cur_frm(i+1), where i is a positive integer, avg_frm(i+1) is the average image of the current image, avg_frm(i) is the original average image, cur_frm is the current image, and α is a preset parameter used to represent the update parameter of the average image. In one implementation, the value of α can be determined according to the usage scenario. For example, in low-contrast scenes such as nighttime, the sensitivity of motion detection can be improved by reducing the value of α. Traditional inter-frame difference methods are sensitive to noise and slight changes in illumination between consecutive frames, and will continue to generate false alarms in scenes with frequent changes in scene noise and illumination, increasing system power consumption. In the embodiments of this application, by constructing an average image to weaken noise and illumination changes, false alarms can be reduced.

[0176] Step S303: Input the motion region image into the first neural network to obtain the first output result.

[0177] Specifically, the first neural network can be a convolutional neural network, and it is trained based on sample data. During the process of the electronic device identifying objects in the moving region image using the first neural network, the first output result can be obtained directly based on the trained model parameters. It is understood that there may be one or more objects in the moving region image, the target object is one of these objects, and the sample data is sample data containing the target object. Therefore, the electronic device can identify the target object in the moving region image based on the first neural network trained according to this sample data.

[0178] Step S304: If the first output result indicates that a target object exists in the motion region image, the second neural network determines that the motion region image contains a target object and generates a trigger signal.

[0179] As can be seen, steps S302 and S303 can detect moving target objects in the target scene, thereby filtering out false detections of dynamic (such as fluttering curtains, leaves, etc.) and static (such as statues, posters, etc.) objects in the target scene. To avoid false wake-ups caused by false detections and missed detections, when the first output result indicates the presence of a target object in the moving area image, this application uses a second neural network to confirm the first output result of the first neural network. When the output result of the second neural network also indicates the presence of a target object in the moving area image, a trigger signal is generated to wake up the processor for corresponding processing. When the output result of the second neural network indicates the absence of a target object in the moving area image, no trigger signal is generated, and the processor is not woken up for corresponding processing.

[0180] The accuracy of the first neural network is lower than that of the second neural network. Therefore, the first neural network can be used for preliminary discrimination, while the second neural network can be used for secondary discrimination, thereby improving the accuracy of discrimination, reducing the number of false wake-ups, and reducing system power consumption.

[0181] In one possible implementation, a first neural network is cascaded with a second neural network. The electronic device inputs the intermediate layer feature data of the first neural network into the second neural network, and the second neural network identifies whether a target object exists in the moving area image. Therefore, in this application, wake-up can be achieved through the input of a single frame image (i.e., the first image). Compared to wake-up that relies on at least two frames or at least four frames, the wake-up based on a single frame image in this application can reduce wake-up latency and improve wake-up speed.

[0182] As can be seen, the input to the second neural network does not depend on the original first image, but instead uses the feature data from the intermediate layers of the first neural network as input. For example, the feature data output to the second neural network can come from the intermediate layers of the first neural network that are downsampled eight times. This reduces the input buffer of the second neural network and alleviates buffer pressure.

[0183] In one possible implementation, to achieve precise wake-up and reduce system power consumption, wake-up can be implemented along the direction of motion that the user is focused on. Therefore, after the electronic device determines that the motion region image contains a target object through a second neural network, it determines the direction of motion of the target object in the motion region image. If this direction of motion belongs to the direction of motion that the user is focused on, a trigger signal is generated to wake up the processor in the electronic device to perform corresponding work. For example, if the direction of motion that the user is focused on is forward motion, and the direction of motion of the target object in the motion region image is also forward motion, which is consistent with the direction of motion that the user is focused on, a trigger signal is generated to wake up the processor; if the direction of motion of the target object in the motion region image is at least one of leftward, rightward, or forward motion, which is inconsistent with the direction of motion that the user is focused on, the processor will not be woken up, and the system can continue to remain in sleep mode.

[0184] In one possible implementation, the electronic device acquires key points of the target object in a motion region image, and then determines the motion direction of the target object based on the motion trend of the key points.

[0185] In one implementation, the key point includes the center point of the motion region image, and the electronic device estimates the direction of motion of the target object based on the changing trend of this center point along the X and Y axes. It is understood that the computing device stores the motion region image up to the first moment; therefore, by combining the changes in the center point of the motion region image up to the first moment along the X and Y axes, the changing trend of the center point of the motion region image at the first moment along the X and Y axes can be obtained.

[0186] For example, if the change of the center point of the motion region image on the X-axis is greater than the change on the Y-axis, it indicates that in the time domain, the motion of the center point on the X-axis is significantly reduced or increased, while the motion on the Y-axis is a small fluctuation. The electronic device can then determine whether the target object is moving to the left or to the right.

[0187] For example, if the change of the center point of the motion region image on the X-axis is less than or equal to the change on the Y-axis, it indicates that in the time domain, the movement of the center point on the X-axis is a small fluctuation, while the movement on the Y-axis is a significant decrease or increase. The electronic device can then determine whether the target object is moving forward or backward.

[0188] In one implementation, the key points include at least one of a first key point, a second key point, and a third key point. These key points can be imaging coordinate points of skeletal points, where the imaging coordinate points are the coordinates of the skeletal points in the motion region image. Please refer to Figure 4C, which is a schematic diagram of key points of a target object provided in an embodiment of this application. As shown in Figure 4C, the first key point is located at the head of the target object, the second key point is located at the left shoulder of the target object, and the third key point is located at the right shoulder of the target object.

[0189] In one possible implementation, the electronic device can predict the motion direction of the target object based on key points of the target object in the motion region image. Specifically, it uses at least one of a first key point, a second key point, and a third key point to comprehensively determine the motion direction of the target object. Further, the electronic device acquires a first distance L1 and a second distance L2, where, as shown in Figure 4C, the first distance L1 is the perpendicular distance from the first key point to a first connecting line, the first connecting line is the line connecting the second and third key points, and the second distance L2 is the distance between the second and third key points. The electronic device can then determine the motion direction of the target object based on the ratio of the first distance L1 and the second distance L2 (r = L1 / L2). For example, the electronic device can determine whether the target object is facing the acquisition device (e.g., a camera) directly or sideways to the acquisition device based on the ratio of the first distance L1 and the second distance L2 (r = L1 / L2), and determine the motion direction of the target object accordingly.

[0190] In one implementation, when the ratio of the first distance to the second distance is less than a first threshold (for example, the first threshold = 0.8), it can be said that the target object is facing the camera, and the electronic device can determine whether the target object is moving forward or backward.

[0191] For example, for images acquired before the first moment, if it is determined that a motion region image exists in the images, the electronic device can store it. Therefore, the electronic device can acquire motion region images of N frames acquired before the first moment, calculate the second distance between the second and third key points of the target object in the motion region image of each of the N frames, and calculate the average of the N second distances, where N is a positive integer greater than or equal to 1. Then, the electronic device determines whether the target object's motion direction is forward or backward based on the second distance (the distance between the second key points of the target object in the motion region image of the first image acquired at the first moment) and the average of the second distances corresponding to the key points of the target object in the motion region images of the N frames.

[0192] For example, the electronic device can estimate the forward or backward movement of the target object based on the difference between the second distance and the aforementioned average. When the difference is greater than zero and the absolute value of the difference is greater than the second threshold (for example, the second threshold = the second distance * 0.05), the electronic device determines that the target object is moving forward. When the absolute value of the difference is less than or equal to the second threshold, the electronic device determines that the target object is moving backward.

[0193] In one implementation, when the ratio of the first distance to the second distance is greater than or equal to a first threshold (e.g., the first threshold = 0.8), it indicates that the target object is facing the camera, and the electronic device can determine whether the target object is moving to the left or right. For example, the electronic device can estimate the left or right movement of the target object based on the changes in the center point of the motion region image along the X and Y axes. If the change in the center point along the X-axis is greater than the change along the Y-axis, then the target object is moving to the left or right.

[0194] Please refer to Figure 5, which is a schematic diagram of a wake-up architecture provided in an embodiment of this application, applied to the system shown in Figure 2. As shown in Figure 5, the wake-up architecture 50 includes at least one of a primary wake-up module 501, a secondary wake-up module 502, and a processor 503. The primary wake-up module 501 includes a motion detection module 5011 and a first neural network 5012, and the secondary wake-up module 502 includes a second neural network 5021. The first neural network 5012 and the second neural network 5021 are cascaded.

[0195] As shown in Figure 5, the electronic device can perceive the changing targets or regions in the target scene between consecutive frames through the motion detection module 5011, thereby acquiring motion region images in consecutive frames. Specifically, the electronic device acquires motion region images using the inter-frame difference method based on average frames provided in this application. For a detailed description, please refer to Figures 4A and 4B, which will not be repeated here.

[0196] As shown in Figure 5, the motion region image input to the first neural network 5012 includes the region extracted from the original resolution image based on the bounding box in Figure 4A (for example, a resolution of 64×64) and a mask image of the motion region image (for example, a resolution of 64×64). In one implementation, the motion region image input to the first neural network 5012 is a mask image superimposed on the aforementioned region. It can be seen that, in order to introduce the location information of the target object to improve classification accuracy, this application adds a mask image channel corresponding to the motion region image to the input of the first neural network 5012.

[0197] In the first neural network 5012 shown in Figure 5, the motion region image is analyzed to obtain a first output result. The first output result indicates that when no target object is found in the motion region image, the first neural network 5012 detects that no target object is moving in the current image frame. Therefore, the second-level wake-up module 502 will not perform further processing on this image frame, that is, it will not wake up the processor 503 to perform corresponding work.

[0198] As shown in Figure 5, when the first output indicates the presence of a target object in the moving region image, the second-level wake-up module 502 is activated. Through two cascaded neural networks, both neural networks require only a single frame of data as input. The input of the second neural network does not depend on the original image frame data, but instead uses the intermediate layer feature data of the first neural network as input. In one implementation, the feature data obtained from feature extraction comes from the intermediate layer of the first-level neural network, which is downsampled eight times, reducing the input buffer of the second-level neural network to 40% of the original resolution (e.g., 160×64).

[0199] Therefore, the second neural network 5021 analyzes the input feature data to obtain a second output result. This second output result indicates that when no target object exists in the motion region image, the second neural network 5021 detects no motion of the target object in the current image frame and therefore will not wake up the processor 503 to perform corresponding work. It is understandable that the accuracy of the second neural network 5021 is higher than that of the first neural network 5012; therefore, the recognition accuracy of the second neural network 5021 is higher than that of the first neural network 5012, and the output result of the second neural network 5021 is reliable.

[0200] As shown in Figure 5, when the second output result indicates that a target object exists in the motion region image, the secondary wake-up module 502 generates a trigger signal to wake up the processor 503 to perform corresponding work.

[0201] Please refer to Figure 6, which is a schematic diagram of another wake-up architecture provided in an embodiment of this application, applied to the system shown in Figure 2. As shown in Figure 6, the wake-up architecture 60 includes at least one of a primary wake-up module 601, a secondary wake-up module 602, and a processor 603. The primary wake-up module 601 includes a motion detection module 6011 and a first neural network 6012, and the secondary wake-up module 602 includes a second neural network 6021 and a motion direction module 6022. The first neural network 6012 and the second neural network 6021 are cascaded.

[0202] For a description of the first-level wake-up module 601 in Figure 6 and the second neural network 6021 of the second-level wake-up module 602, please refer to the relevant descriptions of the first-level wake-up module 501 and the second neural network 5021 of the second-level wake-up module 502 in Figure 5, which will not be repeated here.

[0203] The motion direction module 6022 shown in Figure 6 is used to identify the motion direction of a moving target object. Specifically, it is used to determine the key points of the target object through a key point detection network, and also to determine the motion method of the target object based on the key points. For a description of "identifying the motion direction of a moving target object," please refer to step S304 in Figure 3, which will not be repeated here.

[0204] If the motion direction of the target object determined by the motion direction module 6022 is inconsistent with the motion direction that the user is interested in, it means that the motion direction is not the direction that the user is interested in, and there is no need to wake up the processor 603 to perform the corresponding work.

[0205] As can be seen from Figure 6, when the motion direction of the target object determined by the motion direction module 6022 is consistent with the motion direction that the user is interested in, it indicates that the motion direction is the direction that the user is interested in. The secondary wake-up module 602 generates a trigger signal to wake up the processor 603 to perform the corresponding work.

[0206] Understandably, introducing a motion direction module 6022 into the wake-up architecture 60 can accurately identify the motion direction of target objects in a scene, thereby guiding the system to wake up in the direction of user interest and filtering out wake-ups in directions the user doesn't care about. This allows users to flexibly configure the wake-up state of target objects and also saves system power consumption to a significant extent. For example, in a common smart doorbell system, the system is woken up when someone passes by to perform corresponding functions, without being able to identify whether the moving target object is moving left or right or forward or backward. If the user only needs to focus on waking up a forward-moving human figure, then left-right and backward movements are not considered directions, and the system can remain in sleep mode, theoretically saving 76% of power consumption.

[0207] The methods of the embodiments of this application have been described above, and the apparatus of the embodiments of this application is provided below.

[0208] Please refer to Figure 7, which is a schematic diagram of the structure of a computing device 70 provided in an embodiment of this application. The computing device 70 may include a communication module 701 and a processing module 702, the specific details of which are as follows:

[0209] The communication module 701 can implement the corresponding communication functions, and the processing module 702 is used for data processing. The communication module 701 can also be called a communication interface or a transceiver unit. Optionally, the processing module 702 can be implemented by at least one processor or processor-related circuitry.

[0210] Optionally, the communication module 701 may further include a storage module, which can be used to store instructions and / or data. The processing module 702 can read the instructions and / or data in the storage module to implement the aforementioned method embodiments.

[0211] Optionally, the communication module 701 may include a transmitting unit and a receiving unit. The transmitting unit is used to perform the transmitting operation in the above method embodiments. The receiving unit is used to perform the receiving operation in the above method embodiments. Optionally, the communication module 701 may be implemented by a transceiver or transceiver-related circuitry.

[0212] It should be noted that the computing device 70 may include a transmitting unit but not a receiving unit. Alternatively, the computing device 70 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the computing device 70 includes both transmitting and receiving actions.

[0213] Optionally, the computing device 70 can be used to perform the actions performed by the electronic device in the above method embodiments. The computing device 70 can be an electronic device or a component configurable on an electronic device (e.g., a processor, chip, or chip system). For example, the computing device 70 can be used to perform the following scheme:

[0214] The communication module is used to acquire the first image;

[0215] The processing module is used to determine the motion region image in the first image based on the first image;

[0216] The processing module is further configured to input the motion region image into a first neural network to obtain a first output result;

[0217] The processing module is further configured to, when the first output result indicates that a target object exists in the motion region image, determine that the motion region image contains a target object through a second neural network and generate a trigger signal, wherein the trigger signal is used to wake up the device, and the first neural network is cascaded with the second neural network.

[0218] In one possible implementation, the processing module is specifically used for:

[0219] The feature data of the intermediate layer of the first neural network is input into the second neural network to determine that the target object is contained in the motion region image, wherein the accuracy of the second neural network is greater than that of the first neural network.

[0220] In one possible implementation, the processing module is specifically used for:

[0221] A motion region image in the first image is determined based on the first image and the average image, wherein the first image includes an image acquired at a first time moment, and the average image is determined based on an image acquired before the first time moment.

[0222] In one possible implementation, the processing module is specifically used for:

[0223] Obtain the average image, and determine the average image corresponding to the first image based on the first image and the average image;

[0224] Based on the first image and the average image corresponding to the first image, the motion region image in the first image is determined by inter-frame difference.

[0225] In one possible implementation, the processing module is further configured to:

[0226] The motion direction of the target object in the motion region image is determined to be consistent with a preset direction, wherein the preset direction includes the motion direction set by the user.

[0227] In one possible implementation, the processing module is specifically used for:

[0228] Obtain the key points of the target object in the motion region image;

[0229] The direction of movement of the target object is determined based on the key points;

[0230] Determine that the direction of motion is consistent with the direction of motion.

[0231] In one possible implementation, the processing module is specifically used for:

[0232] The key point includes at least one of a first key point, a second key point, and a third key point, wherein the first key point is a point located on the head of the target object, the second key point is a point located on the left shoulder of the target object, and the third key point is a point located on the right shoulder of the target object.

[0233] Obtain a first distance and a second distance, wherein the first distance is the perpendicular distance from the first key point to the first connecting line, the first connecting line is the connecting line between the second key point and the third key point, and the second distance is the distance between the second key point and the third key point;

[0234] The direction of motion of the target object is determined based on the ratio of the first distance to the second distance.

[0235] In one possible implementation, the processing module has the following functions:

[0236] If the ratio of the first distance to the second distance is less than a first threshold, the direction of motion of the target object is determined to be forward or backward.

[0237] In one possible implementation, the processing module is specifically used for:

[0238] Obtain motion region images of N frames of images acquired before the first moment, where N is a positive integer greater than or equal to 1;

[0239] Based on the second distance and the average of the second distances corresponding to the key points of the target object in the motion region image of the N frames, the motion direction of the target object is determined to be forward or backward.

[0240] In one possible implementation, the processing module is specifically used for:

[0241] If the absolute value of the difference between the second distance and the mean is greater than the second threshold, the target object is moving forward; or,

[0242] The absolute value of the difference between the second distance and the mean is less than or equal to the second threshold, and the target object moves backward.

[0243] In one possible implementation, the processing module is specifically used for:

[0244] If the ratio of the first distance to the second distance is greater than or equal to a first threshold, the direction of movement of the target object is determined to be either leftward or rightward.

[0245] In one possible implementation, the processing module is specifically used for:

[0246] The key point includes the center point of the motion region image;

[0247] Based on the fact that the change of the center point of the motion region image on the X-axis is greater than the change on the Y-axis, the motion direction of the target object is determined to be either moving to the left or to the right.

[0248] In one possible implementation, the processing module is specifically used for:

[0249] The key point includes the center point of the motion region image;

[0250] Based on the fact that the change of the center point of the motion region image on the X-axis is less than or equal to the change on the Y-axis, the motion direction of the target object is determined to be forward or backward.

[0251] Figure 8 shows a schematic diagram of the structure of an electronic device 80 provided in an embodiment of this application. The electronic device 80 is a device with computing capabilities. The device can be a physical device, such as a controller, processor, server (such as a rack server), host, etc., or it can be a virtual device, such as a virtual machine, container, etc.

[0252] As shown in Figure 8, the electronic device 80 includes a processor 802 and a memory 801, and optionally includes a bus 804 and a communication interface 803. The processor 802 and the memory 801 communicate with each other via the bus 804. It should be understood that this application does not limit the number of processors and memories in the electronic device 80. The processor 802, the memory 801, and the optional bus 804 and communication interface can be integrated into a System on a Chip (SoC). An SoC is a chip that integrates multiple functional modules, achieving high integration, high performance, and low power consumption.

[0253] Memory 801 provides storage space, which may optionally store application data, user data, operating system, and computer programs. Memory 801 may include volatile memory, such as random access memory (RAM). Memory 801 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0254] Processor 802 is a module that performs calculations and may include any one or more of the following: controller (e.g., memory controller), central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), coprocessor (to assist the central processing unit in completing corresponding processing and applications), application-specific integrated circuit (ASIC), microcontroller unit (MCU).

[0255] The communication interface 803 is used to provide information input or output to the at least one processor. And / or, the communication interface 803 can be used to receive data transmitted externally and / or transmit data externally. The communication interface 803 can be a wired link interface, such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, and other wireless communication technologies). Optionally, the communication interface 803 may also include a transmitter (such as a radio frequency transmitter, antenna, etc.) or a receiver coupled to the interface.

[0256] Bus 804 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not indicate that there is only one bus or one type of bus. Bus 804 can include pathways for transmitting information between various components of electronic device 80 (e.g., memory 801, processor 802, communication interface 803).

[0257] In this embodiment, the memory 801 stores executable instructions, and the processor 802 executes these executable instructions to implement the aforementioned method for waking up the device, such as the method for waking up the device in the embodiments of FIG3, FIG5, or FIG6. That is, the memory 801 stores instructions for executing the method for waking up the device.

[0258] This application also provides a chip device including at least one processor, which is used to call a computer program or instructions stored in a memory to cause the processor to execute the wake-up device method in the embodiments of FIG3, FIG5 or FIG6 above.

[0259] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed on a processor, enable the method for waking up the device as described in the embodiments of Figures 3, 5, or 6 above.

[0260] This application also provides a computer program product, which includes a computer program or instructions that, when run on a processor, cause the method for waking up the device in the embodiments of FIG3, FIG5, or FIG6 to be executed.

[0261] It is understood that the processor in the embodiments of this application may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0262] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0263] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video optical disc; or it can be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both types of storage media.

[0264] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0265] In the description of this application, terms such as “first,” “second,” “S301,” or “S302” are used only for the purpose of distinguishing descriptions and for the convenience of context. The different sequence numbers themselves do not have specific technical meanings and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying the order of execution of operations. The order of execution of each process should be determined by its function and internal logic.

[0266] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural. Additionally, the character " / " in this document indicates that the preceding and following related objects have an "or" relationship.

[0267] In this application, "transmission" can include the following three situations: sending data, receiving data, or both sending and receiving data. In this application, "data" can include business data and / or signaling data.

[0268] The terms “comprising” or “having” and any variations thereof in this application are intended to cover a non-exclusive inclusion, such as a process / method that includes a series of steps, or a system / product / equipment that includes a series of units, not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes / methods / products / equipment.

[0269] In the description of this application, unless otherwise specified, the number of nouns refers to "singular nouns or plural nouns," that is, "one or more." "At least one" means one or more. "Including at least one of the following: A, B, C" means that it may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C. A, B, and C may be single or multiple.

Claims

1. A method for waking up a device, characterized in that, The method includes: Acquire a first image, wherein the first image is a frame from a video; Determine the motion region image in the first image based on the first image; The motion region image is input into a first neural network to obtain a first output result; When the first output result indicates that a target object exists in the motion region image, the second neural network determines that the motion region image contains a target object and generates a trigger signal, wherein the trigger signal is used to wake up the device, and the first neural network is cascaded with the second neural network.

2. The method according to claim 1, characterized in that, The step of determining that the motion region image contains a target object through a second neural network includes: The feature data of the intermediate layer of the first neural network is input into the second neural network to determine that the target object is contained in the motion region image, wherein the accuracy of the second neural network is greater than that of the first neural network.

3. The method according to claim 1 or 2, characterized in that, Determining the motion region image in the first image based on the first image includes: A motion region image in the first image is determined based on the first image and the average image, wherein the first image includes an image acquired at a first time moment, and the average image is determined based on an image acquired before the first time moment.

4. The method according to claim 3, characterized in that, The step of determining the motion region image in the first image based on the first image and the average image includes: Obtain the average image, and determine the average image corresponding to the first image based on the first image and the average image; Based on the first image and the average image corresponding to the first image, the motion region image in the first image is determined by inter-frame difference.

5. The method according to any one of claims 1-4, characterized in that, After determining that the motion region image contains a target object through the second neural network, and before generating the trigger signal, the method further includes: The motion direction of the target object in the motion region image is determined to be consistent with a preset direction, wherein the preset direction includes the motion direction set by the user.

6. The method according to claim 5, characterized in that, Determining that the motion direction of the target object in the motion region image is consistent with a preset direction includes: Obtain the key points of the target object in the motion region image; The direction of movement of the target object is determined based on the key points; Determine that the direction of motion is consistent with the direction of motion.

7. The method according to claim 6, characterized in that, Determining the movement direction of the target object based on the key points includes: The key point includes at least one of a first key point, a second key point, and a third key point, wherein the first key point is a point located on the head of the target object, the second key point is a point located on the left shoulder of the target object, and the third key point is a point located on the right shoulder of the target object. Obtain a first distance and a second distance, wherein the first distance is the perpendicular distance from the first key point to the first connecting line, the first connecting line is the connecting line between the second key point and the third key point, and the second distance is the distance between the second key point and the third key point; The direction of motion of the target object is determined based on the ratio of the first distance to the second distance.

8. The method according to claim 7, characterized in that, Determining the movement direction of the target object based on the ratio of the first distance to the second distance includes: If the ratio of the first distance to the second distance is less than a first threshold, the direction of motion of the target object is determined to be forward or backward.

9. The method according to claim 8, characterized in that, Determining whether the target object's direction of motion is forward or backward includes: Obtain motion region images of N frames of images acquired before the first moment, where N is a positive integer greater than or equal to 1; Based on the second distance and the average of the second distances corresponding to the key points of the target object in the motion region image of the N frames, the motion direction of the target object is determined to be forward or backward.

10. The method according to claim 9, characterized in that, Determining whether the target object's motion direction is forward or backward based on the average of the second distance and the motion region images of the N frames includes: If the absolute value of the difference between the second distance and the mean is greater than the second threshold, the target object is moving forward; or, The absolute value of the difference between the second distance and the mean is less than or equal to the second threshold, and the target object moves backward.

11. The method according to claim 7, characterized in that, Determining the movement direction of the target object based on the ratio of the first distance to the second distance includes: If the ratio of the first distance to the second distance is greater than or equal to a first threshold, the direction of movement of the target object is determined to be either leftward or rightward.

12. The method according to claim 6, characterized in that, Determining the movement direction of the target object based on the key points includes: The key point includes the center point of the motion region image; Based on the fact that the change of the center point of the motion region image on the X-axis is greater than the change on the Y-axis, the motion direction of the target object is determined to be either moving to the left or to the right.

13. The method according to claim 6, characterized in that, Determining the movement direction of the target object based on the key points includes: The key point includes the center point of the motion region image; Based on the fact that the change of the center point of the motion region image on the X-axis is less than or equal to the change on the Y-axis, the motion direction of the target object is determined to be forward or backward.

14. A computing device, characterized in that, The device includes a communication module and a processing module, wherein... The communication module is used to acquire the first image; The processing module is used to determine the motion region image in the first image based on the first image; The processing module is further configured to input the motion region image into a first neural network to obtain a first output result; The processing module is further configured to, when the first output result indicates that a target object exists in the motion region image, determine that the motion region image contains a target object through a second neural network and generate a trigger signal, wherein the trigger signal is used to wake up the device, and the first neural network is cascaded with the second neural network.

15. The apparatus according to claim 14, characterized in that, The processing module is specifically used for: The feature data of the intermediate layer of the first neural network is input into the second neural network to determine that the target object is contained in the motion region image, wherein the accuracy of the second neural network is greater than that of the first neural network.

16. The apparatus according to claim 14 or 15, characterized in that, The processing module is specifically used for: A motion region image in the first image is determined based on the first image and the average image, wherein the first image includes an image acquired at a first time moment, and the average image is determined based on an image acquired before the first time moment.

17. The apparatus according to claim 16, characterized in that, The processing module is specifically used for: Obtain the average image, and determine the average image corresponding to the first image based on the first image and the average image; Based on the first image and the average image corresponding to the first image, the motion region image in the first image is determined by inter-frame difference.

18. The apparatus according to any one of claims 14-17, characterized in that, The processing module is further configured to: The motion direction of the target object in the motion region image is determined to be consistent with a preset direction, wherein the preset direction includes the motion direction set by the user.

19. The apparatus according to claim 18, characterized in that, The processing module is specifically used for: Obtain the key points of the target object in the motion region image; The direction of movement of the target object is determined based on the key points; Determine that the direction of motion is consistent with the direction of motion.

20. The apparatus according to claim 19, characterized in that, The processing module is specifically used for: The key point includes at least one of a first key point, a second key point, and a third key point, wherein the first key point is a point located on the head of the target object, the second key point is a point located on the left shoulder of the target object, and the third key point is a point located on the right shoulder of the target object. Obtain a first distance and a second distance, wherein the first distance is the perpendicular distance from the first key point to the first connecting line, the first connecting line is the connecting line between the second key point and the third key point, and the second distance is the distance between the second key point and the third key point; The direction of motion of the target object is determined based on the ratio of the first distance to the second distance.

21. The apparatus according to claim 20, characterized in that, The processing module has the following functions: If the ratio of the first distance to the second distance is less than a first threshold, the direction of motion of the target object is determined to be forward or backward.

22. The apparatus according to claim 21, characterized in that, The processing module is specifically used for: Obtain motion region images of N frames of images acquired before the first moment, where N is a positive integer greater than or equal to 1; Based on the second distance and the average of the second distances corresponding to the key points of the target object in the motion region image of the N frames, the motion direction of the target object is determined to be forward or backward.

23. The apparatus according to claim 22, characterized in that, The processing module is specifically used for: If the absolute value of the difference between the second distance and the mean is greater than the second threshold, the target object is moving forward; or, The absolute value of the difference between the second distance and the mean is less than or equal to the second threshold, and the target object moves backward.

24. The apparatus according to claim 20, characterized in that, The processing module is specifically used for: If the ratio of the first distance to the second distance is greater than or equal to a first threshold, the direction of movement of the target object is determined to be either leftward or rightward.

25. The apparatus according to claim 19, characterized in that, The processing module is specifically used for: The key point includes the center point of the motion region image; Based on the fact that the change of the center point of the motion region image on the X-axis is greater than the change on the Y-axis, the motion direction of the target object is determined to be either moving to the left or to the right.

26. The apparatus according to claim 19, characterized in that, The processing module is specifically used for: The key point includes the center point of the motion region image; Based on the fact that the change of the center point of the motion region image on the X-axis is less than or equal to the change on the Y-axis, the motion direction of the target object is determined to be forward or backward.

27. An electronic device, characterized in that, The electronic device includes at least one processor and at least one memory, wherein the at least one memory stores computer instructions; the at least one processor is used to invoke the computer instructions to implement the method of any one of claims 1-13.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions that, when executed by a processor, implement the method of any one of claims 1-13.

29. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed, cause the method as described in any one of claims 1-13 to be performed.

30. A chip system, characterized in that, The chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected via lines. The at least one memory stores computer program instructions. When the computer program instructions are executed by the at least one processor, they implement the method according to any one of claims 1-13.

Images