Action detection method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an action detection method, electronic equipment and a computer readable storage medium, relates to the technical field of electronic equipment, and can improve the accuracy of action recognition. The method comprises the following steps: performing face detection on a first frame image in a first period to obtain position information of n face detection frames; performing similarity matching on the human face in the first frame image in the first period to obtain position information of p human face detection frames, and performing human face tracking on the rest frame images except the first frame image in the first period to obtain position information of n human face detection frames in the rest frame images except the first frame image in the first period; performing key point detection on the face in each frame of image in the first period to obtain position information of p groups of face key point sequences in each frame of image in the first period; and according to the position information of the p groups of face key point sequences in the x frames of images in the first period, performing action recognition on the x frames of images.

Description

Technical Field

[0001] This application relates to the field of electronic device technology, and in particular to an action detection method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the advancement of technology and the growth of demand, more and more functions in electronic devices can meet users' multiple needs in areas such as social, technical and emotional aspects. For example, users often use electronic devices to capture wonderful facial movements, such as shaking their heads, looking back, nodding or blinking.

[0003] Currently, commonly used face motion detection methods obtain 2D and 3D facial landmarks from the image to be detected. Based on these 2D and 3D facial landmarks, three pose angles—pitch, yaw, and roll—can be calculated. These pose angles can then be used to determine whether there are any significant facial movements in the image to be detected.

[0004] However, this detection method, which judges facial movements solely based on these three pose angles, is prone to false positives, leading to reduced accuracy in motion detection and impacting user experience. Summary of the Invention

[0005] This application provides an action detection method, an electronic device, and a computer-readable storage medium to address the problem of low accuracy in action recognition, which affects user experience.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, an action detection method is provided, which is applied to an electronic device, and the method includes:

[0008] Face detection is performed on the first frame image in the first period to obtain the position information of n face detection boxes. The position information of n face detection boxes is used to indicate the position of the face corresponding to the face detection box in the first frame image in the first period, where n is a positive integer.

[0009] Based on the position information of n face detection boxes, similarity matching is performed on the faces in the first frame image within the first period to obtain the position information of p face detection boxes in the first frame image within the first period. Face tracking is performed on the remaining frame images within the first period excluding the first frame image to obtain the position information of n face detection boxes in the remaining frame images within the first period excluding the first frame image, where p is a positive integer.

[0010] Based on the position information of p face detection boxes in the first frame image within the first period, and the position information of n face detection boxes in the remaining frames of the first period excluding the first frame image, key point detection is performed on the faces in each frame image within the first period to obtain the position information of p sets of face key point sequences in each frame image within the first period.

[0011] Based on the positional information of the p groups of facial key point sequences in the x frames of images within the first period, action recognition is performed on the x frames of images to obtain the recognition result. The recognition result is used to indicate whether there is a target action in the x frames of images, where x is a positive integer.

[0012] The method provided in the first aspect involves an electronic device performing face detection on the first frame image within a first period, obtaining the position information of n face detection boxes within the first frame image within the first period. Therefore, determining whether a face has performed a target action requires detecting multiple frames. Furthermore, the number of faces in the images may not change significantly over a certain time period. Thus, the electronic device only needs to detect whether a face exists in the first frame image within a certain period and determine the number and position of faces in that first frame image. It not only uses the faces in the first frame image of that period to match the faces in the first frame image of the previous period, but also uses the faces in the first frame image of that period to track the faces in the remaining frames of that period. There is no need to locate the faces in the remaining frames of that period, i.e., no need to perform face detection on every frame image within that period. Considering that the power consumption required for face tracking is much less than the power consumption required for face detection per frame image, and the power consumption required for face matching is also minimal, it is evident that the overall power consumption is reduced.

[0013] The electronic device performs similarity matching on n faces in the first frame of the first period and faces in the first frame of the fourth period to obtain the position information of p face detection boxes. Therefore, the electronic device can perform similarity matching on faces in the first frame of a certain period and faces in the first frame of previous periods to associate faces in the current period with faces in previous periods. This allows it to obtain information on changes in the number of faces in the images within that period, such as newly added faces and which faces are not displayed in the image, ensuring the continuity of faces in the images throughout the entire period.

[0014] The electronic device performs face tracking on the remaining frames (excluding the first frame) within the first period, obtaining the positional information of n face detection boxes in the remaining frames (excluding the first frame) within the first period. Thus, the electronic device can obtain the number and positional changes of faces in the images within a certain period, such as the number and position of newly added faces, the latest position of existing faces, and which faces(s) are not displayed in the image.

[0015] The electronic device performs keypoint detection on faces in each frame of the first period, obtaining the positional information of p sets of facial keypoint sequences in each frame of the first period. Therefore, the electronic device can accurately and comprehensively obtain the facial keypoints of each face in all frames of a given period, enabling precise detection of facial angles.

[0016] Electronic devices can perform action recognition on x frames of images within the first period based on the positional information of p sets of facial key point sequences. The electronic device can accurately identify whether a face in the x frames is performing a target action and output the recognition result. Therefore, using the positional information of facial key point sequences for action recognition, compared to using pose angles, can improve the accuracy of action recognition, reduce false detections, and enhance the user experience.

[0017] In one possible implementation of the first aspect, based on the position information of n face detection boxes, similarity matching is performed on the faces in the first frame image within the first period to obtain the position information of p face detection boxes in the first frame image within the first period, including:

[0018] Based on the position information of m face detection boxes and n face detection boxes, similarity matching is performed on the faces in the first frame image of the first period and the first frame image of the second period to obtain the position information of p face detection boxes in the first frame image of the first period.

[0019] The position information of m face detection boxes is used to indicate the position of the face corresponding to the face detection box in the first frame image of the second period, which is the period before the first period.

[0020] Electronic devices can correlate faces within two adjacent timeframes. This allows them to determine the changes in the number of faces within each timeframe, such as newly added faces and which faces are not displayed in the image, ensuring the continuity of faces throughout the entire timeframe.

[0021] In another possible implementation of the first aspect, the method specifically includes:

[0022] Face detection is performed on the first frame of the first period to obtain the detection results;

[0023] When the detection result indicates the presence of a face, determine the position information of n face detection boxes.

[0024] In another possible implementation of the first aspect, the method further includes:

[0025] When the detection result indicates that there is no human face and no human body, the duration of the period after the first period is increased until a human face is found in the first frame image of the third period. Then, the duration of the period after the third period is restored. The third period is one period after the first period.

[0026] In another possible implementation of the first aspect, the method further includes:

[0027] When the detection result indicates that there is no human face but there is a moving human body, the duration of the period after the first period is shortened until a human face is found in the first frame image of the third period. Then, the duration of the period after the third period is restored. The third period is one period after the first period.

[0028] The electronic device can dynamically adjust the duration of the first cycle based on the face detection results.

[0029] In another possible implementation of the first aspect, the method further includes:

[0030] Display the first interface, which includes the first control;

[0031] Receive the first operation on the first control, the first operation is used to enable action recognition;

[0032] Performing action recognition on the first frame image includes: performing action recognition on the first frame image in response to a first operation.

[0033] In another possible implementation of the first aspect, the method further includes:

[0034] Display the first interface, which includes the first control;

[0035] In response to a first operation on the first control, a second interface is displayed, the second interface including the second control;

[0036] Receive a second operation on the second control, the second operation being used to enable action recognition;

[0037] Performing action recognition on the first frame image includes: in response to a second operation, performing action recognition on the first frame image.

[0038] In another possible implementation of the first aspect, the method further includes:

[0039] The third interface is displayed, which includes a third control used to launch the first application.

[0040] Receive a third operation on a third control;

[0041] Displaying a first interface includes: responding to a third operation, displaying the first interface, and acquiring images within a first cycle.

[0042] For photography scenarios, electronic devices offer multiple ways to enable or disable motion recognition, allowing users to quickly and easily turn the motion recognition function on or off when needed.

[0043] In another possible implementation of the first aspect, the method further includes:

[0044] The fourth interface is displayed, which is used to play the first video. The fourth interface includes a fourth control.

[0045] Receive the fourth operation on the fourth control, which is used to enable action recognition;

[0046] Perform face detection on the first frame image within the first period, including: in response to the fourth operation, perform face detection on the first frame image within the first period of the first video.

[0047] In another possible implementation of the first aspect, the method further includes:

[0048] The fourth interface is displayed, which includes a fourth control and a fifth control. The fifth control is used to trigger the playback of the first video.

[0049] Receive the fourth operation on the fourth control, which is used to enable action recognition;

[0050] Receive the fifth operation on the fifth control;

[0051] Performing face detection on the first frame image within the first period includes: in response to the fifth operation, displaying the fifth interface, and performing face detection on the first frame image within the first period in the first video, wherein the fifth interface is used to play the first video.

[0052] For camera-based scenarios, electronic devices provide ways to enable or disable motion recognition, allowing users to quickly and easily turn the motion recognition function on or off when needed.

[0053] In another possible implementation of the first aspect, the types of target actions include looking back, shaking the head, nodding, and blinking.

[0054] Electronic devices can expand the types of target actions that can be recognized, thereby improving the user experience.

[0055] In another possible implementation of the first aspect, the x-frame images comprise p sets of facial key point sequences, representing multiple consecutive frames within the first period;

[0056] Alternatively, the x-frame images comprise multiple frames of p sets of facial landmark sequences spaced apart within the first period.

[0057] The electronic device can determine the x-frame image from all frame images within the first period. For example, the x-frame image can include all frame images within the first period, can include multiple consecutive frame images within the first period, or can include multiple spaced frame images within the first period. If the target action is continuous, it indicates that the x-frame image can include multiple consecutive frame images within the first period. If the target action does not change significantly between adjacent images, it indicates that the x-frame image can include multiple spaced frame images within the first period. Therefore, the electronic device can accurately identify whether the target action exists in the x-frame image based on the sequence position information of p groups of facial key points in the x-frame image within the first period.

[0058] In a second aspect, an action detection device is provided, the electronic device comprising: a module for performing the methods described in the first aspect and any possible design of the first aspect.

[0059] Thirdly, an electronic device is provided, comprising: a processor; when the processor executes one or more computer programs stored in a memory, the electronic device executes the action detection method described in the first aspect and any possible implementation thereof.

[0060] Fourthly, an electronic device is provided, comprising: one or more processors; and a memory; wherein the memory stores one or more computer programs, the one or more computer programs including instructions, which, when executed by the electronic device, cause the electronic device to perform the action detection method of the first aspect and any possible implementation thereof.

[0061] Fifthly, a computer-readable storage medium is provided, which stores instructions that, when executed on an electronic device, cause the electronic device to perform the action detection method described in the first aspect and any possible implementation thereof.

[0062] Sixthly, a computer program product is provided, comprising: execution instructions stored in a readable storage medium, at least one processor of an electronic device being able to read the execution instructions from the readable storage medium, and at least one processor executing the execution instructions causing the electronic device to implement the action detection method in the first aspect and any possible implementation thereof. Attached Figure Description

[0063] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0064] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0065] Figure 3 A flowchart illustrating an action detection method provided in this application embodiment. Figure 1 ;

[0066] Figure 4 A flowchart illustrating an action detection method provided in this application embodiment. Figure 2 ;

[0067] Figure 5A A schematic diagram of an action detection method provided in this application embodiment. Figure 1 ;

[0068] Figure 5B A schematic diagram of an action detection method provided in this application embodiment. Figure 2 ;

[0069] Figure 5C A schematic diagram of an action detection method provided in this application embodiment. Figure 3 ;

[0070] Figure 6 A flowchart illustrating an action detection method provided in this application embodiment. Figure 3 ;

[0071] Figure 7A This application provides an example of an interface for enabling motion recognition functionality. Figure 1 ;

[0072] Figure 7B This application provides an example of an interface for enabling motion recognition functionality. Figure 2 ;

[0073] Figure 7C This application provides an example of an interface for enabling motion recognition functionality. Figure 3 ;

[0074] Figure 7D This application provides an example of an interface for enabling motion recognition functionality. Figure 4 ;

[0075] Figure 8A Five schematic diagrams illustrating an interface for enabling action recognition function, provided as embodiments of this application;

[0076] Figure 8B This application provides an example of an interface for enabling motion recognition functionality. Figure 6 ;

[0077] Figure 8C This is a schematic diagram seven of an interface for enabling the action recognition function, provided as an embodiment of this application. Detailed Implementation

[0078] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0079] This application provides an action detection method applied to an electronic device. In this method, the electronic device performs face detection on the first frame of each cycle, and face tracking on all other frames within each cycle, locating faces in each frame. Keypoint detection is then performed on the faces in each frame within each cycle to obtain facial keypoints. Facial action recognition is then performed on the facial keypoints in some or all frames within each cycle to obtain a recognition result. This recognition result indicates whether a target action exists in the recognized image.

[0080] One of the cycles is a preset time period in the electronic device. During this time, the electronic device can acquire multiple frames of images or process multiple frames of images.

[0081] The target action is a specific facial movement, such as looking back, shaking the head, nodding, or blinking.

[0082] For example, the electronic device in the embodiments of this application may be a mobile phone, tablet computer, wearable device (such as a smartwatch or smart bracelet), ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc. The embodiments of this application do not impose special restrictions on the specific form of the electronic device.

[0083] In this embodiment of the application, a mobile phone is used as an example to describe the hardware structure of the electronic device.

[0084] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. Figure 1As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0085] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0087] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0088] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] In some embodiments, the processor 110 may include one or more interfaces. These interfaces are used to support interconnection and communication between the processor 110 and other devices (such as the mobile communication module 150, the audio module 170, and the sensor module 180) to realize various functions of the electronic device 100.

[0090] The charging management module 140 receives charging input from the charger. The power management module 141 connects to the battery 142, and the charging management module 140 connects to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory 120, display 194, camera 193, and wireless communication module 160, etc.

[0091] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0093] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0094] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0095] Electronic device 100 implements display functions through a GPU, display screen 194, and application processor. Electronic device 100 implements shooting functions through an ISP, camera 193, video codec, GPU, display screen 194, and application processor.

[0096] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0097] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0098] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0099] Buttons 190 include a power button, volume buttons, etc. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication.

[0100] The methods described in the following embodiments can all be implemented in the electronic device 100 having the above-described hardware structure.

[0101] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. Figure 2 As shown, the electronic device may include: a face detection module 201, a face matching module 202, a face tracking module 203, a key point detection module 204, and an action recognition module 205.

[0102] Among them, the face detection module 201, face matching module 202, face tracking module 203, key point detection module 204, and action recognition module 205 are software modules or software code.

[0103] The face detection module 201 includes a face detection algorithm. This algorithm can include any one of the following: YOLO (youonly look once), FCOS (fully convolutional one-stage object detection), or RetinaNet.

[0104] The face detection module 201 can use a face detection algorithm to detect faces in an image. The face detection module 201 can identify and locate information such as the features, position, size, and pose of a face. In some embodiments, the face detection module 201 can also generate a face detection box for any face in the image, which can mark the position of the face in the image. In some embodiments, the face detection box can frame or enclose the face in the image. The shape of the face detection box is not limited in this embodiment. For example, the face detection box can be a rectangle.

[0105] The face detection module 201 is typically trained using an image set, which includes multiple images containing faces.

[0106] Alternatively, there may be multiple images containing faces and multiple images not containing faces. The images containing faces may include images of faces at various angles, each angle being represented by at least one of three attitude angles: pitch, yaw, and roll.

[0107] In some embodiments, the image set may include images of profile views of faces. A profile view refers to an image in which the face is deviated from the plane of the image at a certain angle, i.e., the flip angle has a certain value, and only part of the facial features can be displayed in the image. This allows the face detection module 201 to accurately detect faces from various angles in practical use.

[0108] Considering the limited computing power and resources of electronic devices, the face detection module 201 is typically subject to certain specification limitations. In some embodiments, the face detection module 201 can detect a maximum of N faces. Here, N is a positive integer, and N can be pre-configured in the electronic device. When the actual number of faces in the image is less than or equal to N, the face detection module 201 can output the position information of the actual number of face detection boxes. When the actual number of faces in the image is greater than N, the face detection module 201 can output the position information of N face detection boxes. For example, the N face detection boxes are the N largest face detection boxes in the image.

[0109] The face matching module 202 includes a face matching algorithm. This algorithm can include any of the following: a person re-identification (ReID) algorithm, a deep feature-based matching algorithm, or the FaceNet algorithm.

[0110] The face matching module 202 can use a face matching algorithm to extract features from faces in multiple images and perform similarity matching on faces in multiple images based on the features of faces in multiple images.

[0111] For example, multiple images include Image 1 and Image 2, where Image 1 includes face 1 and Image 2 includes face 2. Face matching module 202 can extract features from face 1, and face matching module 202 can extract features from face 2. Based on the features of face 1 and face 2, face matching module 202 can perform similarity matching between face 1 and face 2.

[0112] During the matching process, the face matching module 202 can determine whether there are identical faces in multiple images.

[0113] When multiple images contain the same face, some faces in the multiple images are matched, and the same faces are represented by the same identification information. The remaining faces in the multiple images are not matched, and the remaining faces in the multiple images are represented by different identification information.

[0114] For example, image 1 includes face 1 and face 2, and image 2 includes face 1 and face 3. Face matching module 202 performs face matching on images 1 and 2. During the matching process, if face 1 in image 1 is the same as face 1 in image 2, and neither face 2 in image 1 nor face 3 in image 2 is identical, then face matching module 202 can output an identifier for face 1, and output identifiers for face 2 and face 3 respectively.

[0115] When the faces in multiple images are all different, the faces in the multiple images do not match, and different faces are represented using different identification information.

[0116] For example, image 1 includes face 1 and face 2, and image 2 includes face 3 and face 4. Face matching module 202 performs face matching on images 1 and 2. During the matching process, if no two faces (face 1 and face 2 in image 1, and face 3 and face 4 in image 2) are identical, then face matching module 202 outputs the respective identifier information for face 1, face 2, face 3, and face 4.

[0117] Therefore, the face matching module 202 can output the identification information of all faces in multiple images.

[0118] In addition, the face matching module 202 can store the features of faces in multiple images, eliminating the need to repeatedly extract features from the faces in the multiple images, which facilitates subsequent matching.

[0119] In addition, the face matching module 202 can also store the identification information of faces in multiple images, which facilitates the unification of matched faces in multiple images and the recording of all faces in multiple images.

[0120] The face tracking module 203 includes a face tracking algorithm. This algorithm can be any lightweight algorithm, such as a kernel correlation filter (KCF) algorithm or a SORT (simple online and real-time tracking) algorithm.

[0121] The face tracking module 203 can utilize a face tracking algorithm to perform face tracking on the remaining images (excluding the first image) based on all the faces in the first image out of a set of images. The first image may include one or more faces. Therefore, the face tracking module 203 can obtain the positions of all faces in the first image within the remaining images.

[0122] During the tracking process, for any face in the first image, the face tracking module 203 obtains the features of the face based on its position. For any of the remaining images, the face tracking module 203 determines whether each face in the first image is present in that image based on the features of each face in the first image.

[0123] When a face is in the image, the face tracking module 203 can obtain the position of the face in the image.

[0124] When a face is not in the image, the face tracking module 203 can determine the face's position in the image as a default value. This default value can be preset.

[0125] The keypoint detection module 204 includes a keypoint detection algorithm. This algorithm can include any one of the following: ResNet, MobileNet, or ShuffleNet.

[0126] The keypoint detection module 204 can use a keypoint detection algorithm to detect key points on a face in an image. The keypoint detection module 204 can locate key points on the face and obtain their location information. Key points refer to important feature points of various parts of the face, such as facial features, facial contours, and other features. Important feature points are typically contour points and corner points.

[0127] The action recognition module 205 includes an action recognition algorithm. This algorithm can include any one of the following: a deep neural network for action classification based on temporal information, temporal segment networks (TSN), temporal relational networks (TRN), or efficient convolutional networks (ECO). The action recognition module 205 can expand the types of target actions that can be recognized by modifying the classification head of the aforementioned networks. In some embodiments, the types of target actions can include looking back, shaking one's head, nodding, and blinking.

[0128] The action recognition module 205 can use an action recognition algorithm to perform action recognition on faces in multiple frames of images based on the position information of the key point sequence of faces in multiple frames of images.

[0129] The action recognition module 205 is typically trained using a sequence of facial landmarks, which includes a set of multiple sets of facial landmarks. These facial landmarks may include those corresponding to a target action, or a combination of those corresponding to a target action and those corresponding to a non-target action. This application does not limit the specific number of facial landmarks or the number of sets of facial landmarks in the set.

[0130] The face detection module 201 is connected to the face matching module 202 and the face tracking module 203. The face matching module 202 and the face tracking module 203 are also connected to the key point detection module 204. The key point detection module 204 is also connected to the action recognition module 205.

[0131] In addition, electronic devices can detect the presence of a human body in an image by detecting human contour information, limb information, and other human feature information. Human detection results can be obtained through other modules within the electronic device, such as the human detection module. The human detection module can then feed back the human detection results to the face detection module 201.

[0132] The human detection results can include subject detection values ​​and motion detection values. Subject detection values ​​indicate whether a human body is present in the image. A subject detection value of true indicates the presence of a human body. A subject detection value of false indicates the absence of a human body. Subject detection values ​​and motion detection values ​​indicate whether a human body is moving. A subject detection value of true and a motion detection value of true indicate the presence of a moving human body. A subject detection value of true and a motion detection value of false indicate the absence of a moving human body.

[0133] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. The following will be combined with the foregoing description. Figure 1 and Figure 2 The electronic device with the structure shown will be described in detail with respect to the motion detection method provided in the embodiments of this application.

[0134] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating an action detection method provided in an embodiment of this application. Figure 3 As shown, this method is applied to an electronic device and may include: S301-S305.

[0135] S301. Perform face detection on the first frame image in the first period to obtain the position information of n face detection boxes.

[0136] The position information of n face detection boxes is used to indicate the position of the face corresponding to the face detection box in the first frame image within the first period, where n is a positive integer.

[0137] Electronic devices can use cameras to capture multiple frames of images in real time. Additionally, electronic devices can store multiple frames of images in a storage device. In some embodiments, the electronic device can... Figure 1 The external memory interface 120 is connected to the storage device. Alternatively, the storage device may be... Figure 1 Internal memory 121.

[0138] A multi-frame image can be divided into multiple parts, each part corresponding to a cycle; that is, multiple frames correspond to multiple cycles. The duration of each cycle can be set in the electronic device. The duration of each cycle is set based on factors such as the camera's sampling frequency and / or the processing speed of the processor 110 in the electronic device.

[0139] For example, if a camera can capture 30 frames per second and the duration of one cycle is 200 milliseconds (ms), then there are 6 frames in that cycle.

[0140] For example, if processor 110 can process 20 frames per second and one cycle lasts 500ms, then there are 10 frames in that cycle.

[0141] One period can be any length, such as 100ms, 300ms, or 500ms. The number of periods is set based on factors such as the duration the camera is on and the duration of each period. For example, if the camera is on for 1 minute (min) and one period is 500ms, then the number of periods is 120.

[0142] The first cycle is one of multiple cycles. Within the first cycle, the electronic device can capture multiple frames of images using a camera. The electronic device, using the face detection module 201, can perform face detection on the first frame of the first cycle to obtain the position information of n face detection boxes.

[0143] For the first frame of a multi-frame image, the face detection module 201 can perform face detection on the first frame. The face detection module 201 can detect whether there is a face in the first frame, and thus determine whether the first frame is the first frame in the first period based on the detection result.

[0144] When a face is present in the first frame, the electronic device can determine that the first frame is the first image. The number of faces in the first frame is n. Therefore, for each of the n faces in the first frame, the electronic device can generate a face detection box corresponding to each face and obtain the position information of the n face detection boxes.

[0145] If there is no face in the first frame, the electronic device can detect the first frame of the next period until a face is found in the first frame, at which point the electronic device can perform face detection on that frame.

[0146] When the frame rate in the first cycle is fixed or adjustable, the electronic device can determine the frame number of each image within the first cycle; that is, the electronic device can determine which frame within the first cycle any currently acquired image belongs to. Therefore, the electronic device can identify the first frame within the first cycle, or the first frame containing a face, as the first frame image.

[0147] If a face is present in the first frame of the first period, the electronic device can identify that first frame as the first frame image. Therefore, if the number of faces in the first frame image is n, then the number of face detection boxes is also n.

[0148] If the first frame of the first period does not contain a face, the electronic device can determine this first frame as the first frame image. The electronic device can then continue to identify the first frame containing a face from the remaining frames, perform face detection on this first frame, and use the face detection bounding box corresponding to this first frame as the face detection bounding box for the first frame image. Alternatively, the electronic device can continue to identify the first frame containing a face from the remaining frames. Therefore, the electronic device can determine the first frame containing a face as the first frame image. If the number of faces in the first frame containing a face is n, then the number of face detection bounding boxes is also n.

[0149] The first frame containing a face refers to the earliest frame in the first period that contains a face and was captured at the earliest time.

[0150] In the first frame image, any face detection bounding box can at least outline or circle the face corresponding to that face detection box. The position information of any face detection box can represent the location of the face corresponding to that face detection box in the first frame image. In some embodiments, the position information of the face detection box may include: the size information of the face detection box and the coordinate information of the face detection box.

[0151] In summary, by using the face detection module 201, the electronic device can perform face detection only on the first frame of the first cycle, without having to perform face detection on the remaining frames of the first cycle, which can reduce the computational load of face detection and also reduce the power consumption of the electronic device.

[0152] S302. Based on the position information of n face detection boxes, perform similarity matching on the faces in the first frame image within the first period to obtain the position information of p face detection boxes in the first frame image within the first period, where p is a positive integer.

[0153] When the first cycle is the first of multiple cycles, the electronic device can determine the position information of n face detection boxes as the position information of p face detection boxes. At this time, p equals n.

[0154] When the first period is any of the multiple periods other than the first period, the electronic device uses the face matching module 202 to obtain the features of n faces in the first frame image based on the position information of n face detection boxes. It then performs similarity matching between the features of the n faces in the first frame image within the first period and the features of the faces in the first frame image within the fourth period to obtain the position information of p face detection boxes. At this time, p is greater than or equal to n.

[0155] The fourth period is any period preceding the first period. The fourth period has the same number of frames as the first period. In some embodiments, the fourth period is the period preceding the first period.

[0156] In some embodiments, facial features can be represented using methods such as facial feature vectors. For example, an electronic device can calculate the similarity between the feature vectors of multiple faces using methods such as cosine distance or Euclidean distance. Therefore, the electronic device can determine whether multiple faces match based on the similarity of their feature vectors.

[0157] Therefore, after matching, the electronic device can obtain the location information of p individual face detection boxes.

[0158] p is greater than n when the number of faces in the first frame of the first period is greater than the number of faces in the first frame of the fourth period, or when the number of faces in the first frame of the first period is less than the number of faces in the first frame of the fourth period.

[0159] In this embodiment, there is no limitation on whether the faces in the first frame image within the first period match the faces in the first frame image within the fourth period. The size of p is the sum of the number of faces in the first frame image within the first period and the number of faces in the first frame image within the fourth period, minus the number of faces that match between the first frame image within the first period and the first frame image within the fourth period.

[0160] For example, let n be the number of faces in the first frame of period 1, q be the number of faces in the first frame of period 2, where period 2 is the period preceding period 1, and r be the number of face matches between the first frame of period 1 and the first frame of period 2. If n is greater than q, then p is greater than n. If q is greater than n, then p is greater than n. Furthermore, p = n + qr. Here, q and r are positive integers.

[0161] When the number of faces in the first frame of the first period is equal to the number of faces in the first frame of the fourth period, p is greater than or equal to n.

[0162] If there is at least one face match between the first frame image in the first period and the first frame image in the fourth period, then p is greater than n. If there are n face matches between the first frame image in the first period and the first frame image in the fourth period, then p is equal to n.

[0163] For example, the number of faces in the first frame of period 1 is n, the number of faces in the first frame of period 2 is n, period 2 is the period preceding period 1, and the number of face matches between the first frame of period 1 and the first frame of period 2 is r. If r is greater than or equal to 0 and less than n, then p is greater than n, and p = n + qr. If r equals n, then p equals n. Here, r is a positive integer.

[0164] Specifically, for any face detection box among the p face detection boxes, the face corresponding to that face detection box may not exist in the first frame image within the first period. Therefore, the position information of that face detection box in the first frame image within the first period can be represented as a default value. Alternatively, if the face corresponding to that face detection box may exist in the first frame image within the first period, then the position information of that face detection box in the first frame image within the first period can be represented as the position of the face corresponding to that face detection box in the first frame image within the first period.

[0165] Furthermore, the electronic device can store the position information of p face detection boxes in the first frame image within the first period in the face detection information storage table. The electronic device can also store the features of the faces corresponding to the p face detection boxes in the face detection information storage table. Thus, through the face detection information storage table, the electronic device can easily read the position information of the face detection boxes and the features of the faces, enabling faster face similarity matching.

[0166] For example, a face detection information storage table can be as shown in Table 1. This table includes: the identifier information of the face detection box, the location information of the face detection box, and the features of the face corresponding to the face detection box. In Table 1, the identifier information of the face detection box is represented by its identifier (ID), the location information of the face detection box is represented by its coordinates, and the features of the face corresponding to the face detection box are represented by its feature vector.

[0167] Table 1

[0168] ID of face detection box Coordinates of the face detection bounding box Facial feature vector 1 Coordinates 1 of the face detection bounding box Face feature vector 1 … … … p The coordinates p of the face detection bounding box Face feature vector p

[0169] The coordinate information of the face detection box may include: the x-coordinate of the center point of the face detection box, the y-coordinate of the center point of the face detection box, the length and width of the face detection box, or the x-coordinate of the top left point, the y-coordinate of the top left point, the x-coordinate of the bottom right point, and the y-coordinate of the bottom right point of the face detection box.

[0170] In summary, the electronic device, using the face matching module 202, can match the faces in the first frame image within the first period with the faces in the first frame image within the fourth period, and obtain the position information of p face detection boxes in the first frame image within the first period.

[0171] S303. Based on the position information of the n face detection boxes, perform face tracking on the remaining frames in the first period excluding the first frame image to obtain the position information of the n face detection boxes in the remaining frames in the first period excluding the first frame image.

[0172] The electronic device uses a face tracking module 203 to track faces in the remaining frames of the first period, excluding the first frame, based on n face detection boxes in the first frame image within the first period.

[0173] The face detection module 201 can input n face detection boxes from the first frame image within the first period into the face tracking module 203. For the remaining frames within the first period excluding the first frame image, the face tracking module 203 can treat the n face detection boxes as n starting boxes. Therefore, the face tracking module 203 only tracks the faces within the n starting boxes.

[0174] Based on the position information of the n face detection boxes in the first frame image within the first period, the face tracking module 203 can locate the n faces in the first frame image within the first period. Based on the n faces, the face tracking module 203 can obtain the features of the n faces.

[0175] The face tracking module 203 performs n-thread tracking processing on the faces in the remaining frames (excluding the first frame) within the first period based on the features of n faces, in order to determine whether the n faces are in the remaining frames.

[0176] For any one of the n faces, that face may exist in one of the remaining frames. Then the face tracking module 203 can determine the position information of the face detection box corresponding to that face, which represents the position of that face in that frame.

[0177] For any one of the n faces, that face may not exist in a certain frame of the remaining frames. The face tracking module 203 can then determine that the position information of the face detection box corresponding to that face is a default value, indicating that the face does not exist in that frame. Typically, the default value is preset in the electronic device.

[0178] The execution order of S302 and S303 is not important; they can be executed simultaneously or sequentially. This application embodiment will not elaborate on this point.

[0179] The face detection module 201 inputs n face detection boxes from the first frame image within the first period to the face tracking module 203. These n face detection boxes are the n starting boxes of the face tracking module 203. The face tracking module 203 only tracks the faces within the n starting boxes. Thus, the electronic device performs face detection on the first frame image within the first period, while not performing face detection on the remaining frames within the first period.

[0180] S304. Based on the position information of p individual face detection boxes in the first frame image within the first period, and the position information of n individual face detection boxes in the remaining frames of the first period excluding the first frame image, key point detection is performed on the faces in each frame image within the first period to obtain the position information of p sets of face key point sequences in each frame image within the first period.

[0181] For the first frame and the remaining frames within the first period, the electronic device can obtain the position information of p face detection boxes in the first frame and the position information of n face detection boxes in the remaining frames. Therefore, the electronic device uses the keypoint detection module 204 to locate the faces in each frame within the first period based on the aforementioned position information, and performs keypoint detection on the faces in each frame within the first period to obtain the position information of the face keypoint sequence in each frame within the first period.

[0182] The facial landmark sequence comprises a set of multiple facial landmarks. The location information of the facial landmark sequence can be understood as the position of each facial landmark on the face, which may include coordinate information and other details.

[0183] In the first period, the first frame contains p face detection boxes, and each of the remaining frames contains n face detection boxes. Therefore, in the first period, each frame contains at most p face detection boxes. In other words, in the first period, each frame contains at most p faces.

[0184] Thus, the electronic device performs key point detection on the face in each frame of the first period, and can obtain the position information of p sets of face key point sequences in each frame of the first period.

[0185] Furthermore, the electronic device can store the positional information of p sets of facial key point sequences in each frame of the first period in a key information sequence storage table. This key information sequence storage table allows the electronic device to easily read the positional information of the facial key point sequences in x frames of the first period, enabling faster target action recognition.

[0186] For example, the key information sequence storage table can be as shown in Table 2. This key information sequence storage table includes: the total number of frames stored in the image within one period and the position information of the facial key point sequence corresponding to each face detection box. In Table 2, U represents the total number of frames stored in the image within the first period, where U is a positive integer.

[0187] Table 2

[0188]

[0189]

[0190] The key information sequence storage table uses a first-in-first-out (FIFO) replacement strategy. For the U+1 frame image, the electronic device can delete the position information of the p groups of facial key point sequences of the 1st frame image in Table 2, and then add the position information of the P groups of facial key point sequences of the U+1 frame image.

[0191] S305. Based on the position information of the p-group facial key point sequence in the x-frame images within the first period, perform action recognition on the x-frame images to obtain the recognition result.

[0192] The recognition result is used to indicate whether there is a target action in the x frames of the image, where x is a positive integer.

[0193] The electronic device can determine frame x from all frames in the first cycle.

[0194] Where x can be less than or equal to the number of image frames in the first period.

[0195] In some embodiments, when x equals the number of image frames in the first period, x frames include all frames in the first period. For example, if there are 6 frames in the first period and x is 6, then these six frames can include all frames. Referring to Table 2, the key information sequence storage table in this case can be as shown in Table 3.

[0196] Table 3

[0197]

[0198] In some embodiments, when x is less than the number of image frames in the first period, x frames may include multiple consecutive frames within the first period. Alternatively, x frames may include multiple spaced frames within the first period. Spaced frames refer to multiple non-consecutive images, and the intervals between these non-consecutive images may be the same or different.

[0199] For example, if there are 6 frames in the first period, and x is 3, then the 3 frames can include the first, second, and third frames from the 6 frames. Referring to Table 2, the key information sequence storage table in this case can be shown in Table 4.

[0200] Table 4

[0201]

[0202] For example, if there are 6 frames in the first period, and x is 3, then the 3 frames can include the first, third, and fifth frames from the 6 frames. Referring to Table 2, the key information sequence storage table in this case can be shown in Table 5.

[0203] Table 5

[0204]

[0205] Therefore, the electronic device utilizes the action recognition module 205 to perform action recognition based on the sequence position information of the p groups of facial key points in the x frames of images within the first period. The electronic device can identify whether the face in the x frames of images has performed the target action and output the recognition result.

[0206] The electronic device can output the recognition result in the form of a multidimensional array. The elements in the multidimensional array can include the image's identification information and the corresponding recognition result, which indicates whether the face in the image has performed the target action.

[0207] The action detection method provided in this application performs face detection on the first frame image within a first period using an electronic device, obtaining the position information of n face detection boxes in the first frame image within the first period. Therefore, determining whether a face is performing a target action requires detecting multiple frames. Furthermore, the number of faces in an image may not change significantly over a certain period. Thus, the electronic device only needs to detect whether a face exists in the first frame image within a certain period and determine the number and position of faces in that first frame image. It not only uses faces in the first frame image of that period to match faces in the first frame image of the previous period, but also uses faces in the first frame image of that period to track faces in the remaining frames within that period. There is no need to locate faces in the remaining frames within that period, i.e., no need to perform face detection on every frame image within that period. Considering that the power consumption required for face tracking is much less than the power consumption required for face detection per frame image, and the power consumption required for face matching is also minimal, the overall power consumption is reduced.

[0208] The electronic device performs similarity matching on n faces in the first frame of the first period and faces in the first frame of the fourth period to obtain the position information of p face detection boxes, where p is greater than or equal to n. Thus, the electronic device can perform similarity matching on faces in the first frame of a certain period and faces in the first frame of previous periods to associate faces in the current period with faces in previous periods. This allows it to obtain information on changes in the number of faces in the images within that period, such as newly added faces and which faces are not displayed in the image, ensuring the continuity of faces in the images throughout the entire period.

[0209] The electronic device performs face tracking on the remaining frames (excluding the first frame) within the first period, obtaining the positional information of n face detection boxes in the remaining frames (excluding the first frame) within the first period. Thus, the electronic device can obtain the number and positional changes of faces in the images within a certain period, such as the number and position of newly added faces, the latest position of existing faces, and which faces(s) are not displayed in the image.

[0210] The electronic device performs keypoint detection on faces in each frame of the first period, obtaining the positional information of p sets of facial keypoint sequences in each frame of the first period. Therefore, the electronic device can accurately and comprehensively obtain the facial keypoints of each face in all frames of a given period, enabling precise detection of facial angles.

[0211] Electronic devices can perform action recognition on x frames of images within the first period based on the positional information of p sets of facial keypoint sequences. The electronic device can accurately identify whether a face in the x frames has performed a target action and output the recognition result. Here, x can be less than or equal to the number of image frames within the first period. Therefore, using the positional information of facial keypoint sequences for action recognition, compared to using pose angles, can improve the accuracy of action recognition, reduce false detections, and enhance the user experience.

[0212] Based on the description of the above embodiments, in S302, the electronic device performs similarity matching on the faces in the first frame image within the first period. In some embodiments, the electronic device can perform similarity matching on the n faces in the first frame image within the first period and the m faces in the first frame image within the second period. The electronic device can obtain the position information of p face detection boxes in the first frame image within the first period.

[0213] Where m, n, and p are all positive integers, and the second period is the period preceding the first period. The number of frames in the second period is the same as that in the first period. The electronic device performs similarity matching on the first frame image within the first period and the first frame image within the period preceding the first period, enabling it to correlate and match faces in two consecutive periods. Thus, the electronic device can obtain the relationships between faces within each period.

[0214] The electronic device can obtain the features of m faces in the first frame image within the second period by reading the face detection information storage table. Alternatively, the electronic device can extract the features of m faces in the first frame image within the second period based on the position information of the m face detection boxes within the first frame image within the second period.

[0215] The electronic device can obtain the identification information of p faces by performing similarity matching on the features of n faces and m faces. Therefore, the electronic device can obtain the position information of the p face detection boxes in the first frame image within the first period.

[0216] When the number of faces in the first frame of the first period is equal to the number of faces in the first frame of the second period, i.e., n equals m, then p is greater than or equal to n and p is greater than or equal to m.

[0217] When m = n, if there is at least one face match between the first frame image in the first period and the first frame image in the second period, then p is greater than n and p is greater than m. If there are n face matches between the first frame image in the first period and the first frame image in the second period, then p is equal to n and p is equal to m.

[0218] In this embodiment, there is no limitation on whether the faces in the first frame image of the first period match the faces in the first frame image of the second period. The size of p is the sum of the number of faces in the first frame image of the first period and the number of faces in the first frame image of the second period, minus the number of faces that match between the first frame image of the first period and the first frame image of the second period.

[0219] For example, the number of faces in the first frame of the first period is n, and the number of faces in the first frame of the second period is m, where n equals m. The number of face matches between the first frame of the first period and the first frame of the second period is z. If z equals n, then p equals n and p equals m. If z is greater than or equal to 0 and less than n, then p is greater than n and p is greater than m. Furthermore, p = n + mz. Here, z is a positive integer.

[0220] When the number of faces in the first frame of the first period is greater than the number of faces in the first frame of the second period, or when the number of faces in the first frame of the first period is less than the number of faces in the first frame of the second period (i.e., n is greater than m or n is less than m), then p is greater than n and p is greater than m.

[0221] For example, the number of faces in the first frame of the first period is n, and the number of faces in the first frame of the second period is m. The number of face matches between the first frame of the first period and the first frame of the second period is z. If n is greater than m, then p is greater than n and p is greater than m. If m is greater than n, then p is greater than m and p is greater than m. Furthermore, p = n + mz. Where z is a positive integer.

[0222] In summary, electronic devices can perform correlation matching on faces in two adjacent timeframes. Therefore, the electronic devices can obtain information on the changes in the number of faces within each timeframe, such as newly added faces and which faces(s) are not displayed in the image, ensuring the continuity of faces in the image throughout the entire timeframe.

[0223] Based on the description of the above embodiments, the duration of the first cycle may include various representations.

[0224] In some embodiments, the duration of the first period can be a fixed period of time, that is, the frame rate of the first period is fixed.

[0225] In multiple cycles, the electronic device can determine whether a face exists in the first frame image of the first cycle. If a face exists, the electronic device can perform the motion detection method provided in this application on the image of the first cycle. If a face does not exist, the electronic device can continue to determine whether a face exists in the first frame image of the second cycle, without needing to perform subsequent processing on the remaining frame images of the first cycle, such as not needing to determine whether a face exists in the remaining frame images of the first cycle. The processing procedure for the first frame image of the second cycle can refer to the description of the first frame image of the first cycle, and the processing procedure for the remaining frame images of the second cycle can refer to the description of the remaining frame images of the first cycle, and will not be repeated here. Similarly, the image of the third cycle can refer to the description of the image of the first cycle, and when a face is detected in the first frame image of a certain cycle within multiple cycles, the electronic device can perform the motion detection method provided in this application on the image of that cycle until the multiple cycles end.

[0226] In some embodiments, the duration of the first period can be dynamically adjusted, that is, the frame rate of the first period is dynamic.

[0227] Within multiple cycles, the electronic device can determine whether a human face exists in the first frame of each cycle. If it does, the electronic device can execute the motion detection method provided in this application on the images within each cycle. If it does not exist, the electronic device can determine whether a human body exists in the first frame of each cycle, thereby dynamically adjusting the duration of each cycle.

[0228] Below, in conjunction with Figure 4 and Figures 5A-5C For example, let's illustrate the duration of the first cycle adjustment of an electronic device.

[0229] Please see Figure 4 and Figures 5A-5C , Figure 4 This is a flowchart of an action detection method provided in an embodiment of this application. Figures 5A-5C This is a schematic diagram of the interface for a face detection result provided in an embodiment of this application.

[0230] like Figure 4 As shown, the motion detection method provided in this application embodiment may include:

[0231] S401. Perform face detection on the first frame image in the first cycle and obtain the detection result.

[0232] S402. Determine whether the detection result indicates that a human face exists in the first frame image.

[0233] The test result is Figure 5A When a face is present in the first frame image shown, the electronic device executes S403. When the detection result is... Figure 5BWhen there is no human face in the first frame image shown, the electronic device executes S404.

[0234] S403. Determine the position information of the n face detection boxes.

[0235] S404. Determine whether the detection result indicates that a human body exists in the first frame image.

[0236] The test result is Figure 5C When the first frame image shown does not contain a human face but contains a moving human body, the electronic device executes S405. The detection result is... Figure 5B When there is no human face and no human body in the first frame image shown, the electronic device executes S406.

[0237] The presence of a human body can be characterized using a subject detection value. A subject detection value of true indicates the presence of a human body, while a subject detection value of false indicates the absence of a human body. Embodiments of this application include, but are not limited to, using subject detection values ​​to characterize the presence of a human body.

[0238] S405, shorten the duration of the third cycle.

[0239] For example, the duration of the first cycle is T, and the number of image frames in the first cycle is M1. The electronic device can shorten the duration of the third cycle from T to C, and the electronic device can reduce the number of image frames in the third cycle from M1 to Y. Where T is greater than C, and M1 is greater than Y.

[0240] The absence of a face in the first frame, coupled with the presence of a moving human body, indicates that the person may be temporarily far from the camera, but still within its field of view. The person may gradually move closer to the camera from a distance, during which time the electronic device can detect the face. In this situation, the electronic device can shorten the first frame duration, enabling face detection in a shorter time.

[0241] In this application, the presence or absence of a human body can be characterized by motion detection values. When both the subject detection value and the motion detection value are true, it indicates the presence of a moving human body. When both the subject detection value and the motion detection value are false, it indicates the absence of a moving human body. This application's embodiments include, but are not limited to, using subject detection values ​​and motion detection values ​​to characterize the presence or absence of a moving human body.

[0242] S406, the duration of the third growth cycle.

[0243] For example, the duration of the first cycle is T, and the number of image frames in the first cycle is M1. The electronic device can increase the duration of the third cycle from T to D, and the electronic device can increase the number of image frames in the third cycle from M1 to W. Where T is less than D, and M1 is less than W.

[0244] If the first frame does not contain a face or a human body, it means that the human body is not currently within the camera's field of view. In this case, the electronic device can extend the duration of the first cycle, reducing the computational load and saving power.

[0245] Furthermore, in some other embodiments, the electronic device performs face detection on the first frame of the first period and obtains the detection result. The electronic device can first determine whether the detection result indicates the presence of a moving human body in the first frame image.

[0246] If the detection result indicates the presence of a moving human body in the first frame image, the electronic device determines whether the detection result indicates the presence of a face in the first frame image. If the detection result indicates the presence of a face in the first frame image, the electronic device executes S403. If the detection result indicates the absence of a face in the first frame image, the electronic device executes S405.

[0247] When the detection result indicates that there is no human body in the first frame image, the electronic device executes S406.

[0248] In other embodiments, the electronic device may also simultaneously determine whether the detection result indicates the presence of a face and whether the detection result indicates the presence of a human body.

[0249] If the detection result indicates that there is no human body in the first frame image, then there must also be no human face in the first frame image, and the electronic device executes S406.

[0250] When the detection result indicates that there is a moving human body and a face in the first frame image, the electronic device executes S403.

[0251] When the detection result indicates that there is a moving human body in the first frame image but no human face, the electronic device executes S405.

[0252] For example, an electronic device can perform face detection on the first frame of an image across multiple cycles, with each cycle lasting T. The electronic device performs face detection on the first frame of the first cycle. If the detection result is that no face and no human body are found, the electronic device can increase the duration of subsequent cycles to D. The electronic device can continue performing face detection on the first frame of the second cycle. If the detection result is that no face and no human body are found, the electronic device can continue performing face detection on the first frame of the third cycle. If the detection result is that a face is found, the electronic device can restore the duration of subsequent cycles to T. The electronic device can continue performing face detection on the first frame of the fourth cycle, and so on, until the multiple cycles are completed.

[0253] For example, an electronic device can perform face detection on the first frame of an image across multiple cycles, with each cycle lasting T. The electronic device performs face detection on the first frame of the first cycle. If the detection result indicates no face is found but a moving human body is present, the electronic device can shorten the duration of subsequent cycles to C. The electronic device can then continue performing face detection on the first frame of the second cycle. If the detection result indicates no face is found but a moving human body is present, the electronic device can continue performing face detection on the first frame of the third cycle. If the detection result indicates a face is found, the electronic device can restore the duration of subsequent cycles to T. The electronic device can continue performing face detection on the first frame of the fourth cycle, and so on, until the multiple cycles are completed.

[0254] In some embodiments, the electronic device performs face detection on the first frame image within the first period. When the detection result is no face, the duration of the first period may remain unchanged or be dynamically adjusted based on the detection result. When the detection result is no face and no human body, the duration of the first period may be increased to D. When the detection result is no face and a moving human body is present, the duration of the first period may be shortened to C.

[0255] In summary, the duration of the first cycle can be dynamically adjusted based on the face detection results.

[0256] Therefore, the electronic device can adjust the duration of each cycle. After adjusting the duration of each cycle, the electronic device will continue to determine whether a face exists in the image within each cycle. When a face is detected in the first frame of a certain cycle, the electronic device can execute the motion detection method provided in this application on the image within that cycle. Similarly, the electronic device will continue to determine whether a face exists in the first frame of the subsequent cycles, a process identical to the aforementioned process, which will not be elaborated here, until multiple cycles have ended.

[0257] In a specific embodiment, taking a mobile phone as an example, the following is combined with... Figure 6 It describes in detail the specific implementation process of electronic devices recognizing target actions.

[0258] Please see Figure 6 , Figure 6 This is a flowchart illustrating an action detection method provided in an embodiment of this application. Figure 6 As shown, the target action identification method in this application embodiment may include:

[0259] S601. The mobile phone can acquire images in a variety of ways, such as image streaming.

[0260] The mobile phone can store the acquired images in its memory, and it can store the images frame by frame according to the frame number.

[0261] In a photo-taking scenario, the phone activates its camera. The camera can capture multiple consecutive frames of images (image stream) in real time. The phone can store the captured images in its memory. The duration of image capture is t, and the camera can capture a total of N frames. The number of cycles within duration t can be: a = N / M1, where a is the number of cycles within duration t. Therefore, the duration T of one cycle is t / a, and the number of frames in one cycle is N / a.

[0262] In a video recording scenario, a mobile phone can capture video from its own storage or receive video transmitted from other devices. The video duration can be 's', and it contains L frames. The number of cycles within a duration of 's' can be calculated using the formula b = L / M1, where b is the number of cycles in the video. Therefore, the duration T of one cycle is 's / b', and the number of frames in one cycle is L / b.

[0263] Thus, the mobile phone can determine which period any frame of an image or video it captures belongs to, and whether it is the first frame of that period.

[0264] S602. The mobile phone determines whether the image is the first frame image within time T.

[0265] Taking M1 as an example, the phone determines whether the image captured by the camera is the first frame within time T. If it is the first frame, the phone performs face detection on the first frame. If it is not the first frame, the phone performs face tracking on the remaining M1-1 frames within time T.

[0266] In a photo-taking scenario, the first frame captured by the camera is the first frame image in the first cycle, and the first frame images in other cycles can be determined through calculation.

[0267] In a video recording scenario, the first frame of the video is the first frame image in the first cycle, and the first frame image in the remaining cycles can be determined by calculation.

[0268] The mobile phone can determine whether the j-th frame is the first frame in a period by calculating the remainder of j / M1. Here, j takes all positive integers greater than or equal to 1, and j represents the sequence number of the frame.

[0269] If the remainder of j / M1 is 1, then the j-th frame is the first frame of a period. If the remainder of j / M1 is a positive integer greater than 1, then the j-th frame is not the first frame of a period.

[0270] In the first cycle, the mobile phone reads each frame of the image stored in the storage device. The phone performs face detection on the first frame and obtains the position information of the face detection bounding boxes for m faces in that first frame. Additionally, the phone acquires the facial feature vectors of the m faces and stores them in the face detection information storage table. The phone then performs face tracking on the remaining frames (excluding the first frame) and obtains the position information of the face detection bounding boxes for the remaining frames.

[0271] The mobile phone can respond to the user's activation of the motion recognition function and perform motion recognition on images within the first period. Based on the faces in each frame of the first period, the phone performs keypoint detection on the faces in each frame, obtaining the positional information of p sets of facial keypoint sequences in each frame. Using the positional information of the p sets of facial keypoint sequences in x frames, the phone performs motion recognition on the faces in x frames. The phone then outputs the recognition results.

[0272] In some embodiments, within the first cycle, the mobile phone determines that the target action exists in the x-frame images. In this case, the mobile phone may stop performing action recognition on the images in the remaining cycles. If, within the first cycle, the mobile phone determines that the target action does not exist in the x-frame images, the mobile phone may continue performing action recognition on the images in the remaining cycles.

[0273] S603: The mobile phone performs face detection on the first frame image and obtains the position information of n face detection boxes.

[0274] In the second cycle, the phone reads each frame of the image stored in the storage device for the second cycle. The phone performs face detection on the first frame image, obtaining the position information of n face detection boxes. Based on the position information of the n face detection boxes, the phone can locate the face in the first frame image.

[0275] S604. In the second cycle, the mobile phone calculates the facial feature vectors of n faces in the first frame image. The mobile phone reads the face detection information storage table to obtain the facial feature vectors of m faces in the first frame image in the first cycle. The mobile phone calculates the similarity between the feature vectors of the n faces and the m faces, and performs similarity matching on the n faces and m faces. Through similarity matching, the mobile phone obtains the position information of p face detection boxes in the first frame image. Additionally, the mobile phone replaces the face detection information storage table to store the feature vectors of the p faces in the first frame image and the position information of the face detection boxes.

[0276] S605. In the second cycle, the mobile phone performs face tracking on the remaining frames (excluding the first frame) based on the position information of the n face detection boxes in the first frame image. The mobile phone can obtain the position information of the n face detection boxes in each of the remaining frames.

[0277] When a user enables the phone's motion recognition function, the phone can perform motion recognition. When a user disables the phone's motion recognition function, the phone can choose not to perform motion recognition.

[0278] In photo-taking scenarios, users can enable motion recognition in the camera app. After receiving the user's activation of motion recognition, the phone performs motion recognition on the first frame of the image within the second period.

[0279] In camera-based scenarios, users can play pre-recorded videos in the Gallery app and enable motion recognition. After receiving the user's activation of motion recognition, the phone performs motion recognition on the first frame of the second period.

[0280] S606. The mobile phone reads each frame of the image stored in the storage device within the second cycle. Based on the position information of p face detection boxes in the first frame image and the position information of n face detection boxes in the remaining frames image excluding the first frame image, the mobile phone can locate the face in each frame image. The mobile phone performs key point detection on the face in each frame image to obtain the position information of p sets of face key point sequences in each frame image.

[0281] S607. During the second cycle, the mobile phone performs action recognition on the faces in the x-frame images using the positional information of p sets of facial key point sequences in the x-frame images. The mobile phone outputs the action recognition results for the p faces in the x-frame images.

[0282] The mobile phone can store the location information of p key point sequences in x frames of images, as well as the identification information of p face detection boxes in the key information sequence storage table.

[0283] During the second cycle, in a photo-taking scenario, when the phone recognizes a face in the image making a turning-over gesture, it sends a shooting command to the camera. The phone can capture memorable moments such as turning over in a glance.

[0284] In a video recording scenario, when the phone recognizes a face in the video making a turning-back gesture, the phone recommends a video editing app to the user, making it easy for the user to edit and enhance the video recording of the turning-back gesture.

[0285] In response to a user activating the phone's motion recognition function, the phone can continue motion recognition on images within the remaining period. In some embodiments, during the second period, the phone determines that a target action exists in x frames of images. In this case, the phone can stop motion recognition on images within the remaining period. During the second period, the phone determines that no target action exists in x frames of images. In this case, the phone can continue motion recognition on images within the remaining period. In response to a user deactivating the phone's motion recognition function, the phone can stop motion recognition on images within the remaining period.

[0286] In the third cycle, the phone processes the images in the third cycle similarly to the images in the second cycle. The difference lies in S603, where the first frame of the third cycle is matched with the first frame of the second cycle for facial similarity.

[0287] Therefore, in all cycles except the first cycle, the first frame image in any other cycle is matched with the first frame image in the previous cycle for facial similarity.

[0288] For example, the first frame image of the i-th period is used for similarity matching with the first frame image of the (i-1)-th period. Here, the i-th period is any period other than the first period within a periods, and i is a positive integer.

[0289] In summary, the electronic device can perform motion recognition in response to a user's activation of the motion recognition function. Within any given period, the electronic device can perform motion recognition on x frames of images within that period. If the recognition result indicates the presence of a target action in those x frames, the electronic device can stop performing motion recognition on images within the remaining periods. If the recognition result indicates the absence of a target action in those x frames, the electronic device can continue performing motion recognition on images within the remaining periods.

[0290] The above embodiments can be applied to photo-taking and photography scenarios of electronic devices. Electronic devices can capture the target movements of human faces in photo-taking and photography scenarios.

[0291] In the following embodiments, the electronic device is a mobile phone, and the target action is looking back.

[0292] In photography scenarios, users typically use their phones to capture memorable facial movements. Users can enable motion recognition on their phones, allowing the phone to perform motion recognition on the images captured by the camera.

[0293] As one possible implementation method, please refer to Figures 7A-7D . Figures 7A-7D This is a schematic diagram of an interface for enabling the action recognition function provided in this embodiment.

[0294] The phone can display Figure 7A An exemplary user interface 71. The user interface 71 may include a camera icon 701a and a gallery icon 702a, etc.

[0295] When the phone detects a user-instructed action to open the camera app, such as clicking on control 701a, it can display... Figure 7A The exemplary user interface 71 becomes a display Figure 7B The user interface 72 shown is an example.

[0296] Once the camera app is opened, the camera starts up, and the user interface 72 can display the image captured by the camera in real time. The user interface 72 may include controls 703a and 704a.

[0297] When the phone detects a user-instructed action to enable motion recognition, such as when the user clicks on control 703a, the phone can perform motion recognition on the face in interface 72.

[0298] When the phone detects a glance back, it can display... Figure 7D The user interface 72 is shown as an example. The user interface 72 may include a prompt box 706. Figure 7D If a person makes a turning-back gesture, and the phone can recognize the turning-back gesture, then prompt box 706 can display that the turning-back gesture has been recognized.

[0299] Control 701a is used to open the camera app on the phone. Control 702a is used to open the gallery app on the phone. Control 703a is used to enable the motion recognition function on the phone. Control 704a is used to enable... Figure 7C The user interface 73 is shown as an example.

[0300] As another feasible implementation, please refer to Figures 7A-7D .

[0301] When the mobile phone detects a user-instructed action to activate the motion recognition function, such as a user clicking on control 704a, it can display... Figure 7B The exemplary user interface 72 becomes a display Figure 7C The user interface 73 is shown as an example.

[0302] User interface 73 is the settings interface for the camera app, where users can configure the app's functions and performance parameters. User interface 73 may include control 705a.

[0303] When the mobile phone detects a user-instructed action to activate the motion recognition function, such as a user clicking on control 705a, it can display... Figure 7C The exemplary user interface 73 becomes a display Figure 7B The user interface 72 is shown as an example. After the user enables the motion recognition function, the mobile phone can begin motion recognition on the first frame image captured by the camera.

[0304] Control 705a is used to enable the motion recognition function on the phone. When the phone recognizes a person's gaze turning over, it can display... Figure 7D The user interface 72 shown is an example.

[0305] For details on the implementation of the first interface, please refer to [link / reference]. Figure 7B The description of the user interface 72 shown is provided below. For details on the implementation of the second interface, please refer to [link / reference needed]. Figure 7C The user interface 73 shown is described below. For details on the implementation of the third interface, please refer to [link / reference needed]. Figure 7A Description of the user interface 71 shown. The first operation includes the user clicking on control 703a and control 704a; the second operation includes the user clicking on control 705a; and the third operation includes the user clicking on control 701a. The first control includes 703a and 704a, the second control includes 705a, and the third control includes 701a.

[0306] In summary, regarding the photography scenarios, Figures 7A-7D This implementation provides two ways for users to enable motion recognition on their phones. Users can enable motion recognition directly from the camera app interface, or they can enable it through the camera app's settings. Thus, electronic devices can provide users with multiple entry points to enable motion recognition.

[0307] In photography scenarios, users typically use their phones to capture facial movements. Users can pre-record videos and then enable motion recognition on their phones, allowing the phones to perform motion recognition within the video.

[0308] As one possible implementation method, please refer to Figure 7A and Figures 8A-8C . Figures 8A-8C This is a schematic diagram of an interface for enabling the action recognition function, provided in an embodiment of this application.

[0309] The phone can display Figure 7A Example user interface 71.

[0310] When the phone detects a user-instructed action to open the gallery app, such as clicking on control 702a, it can display... Figure 7A The exemplary user interface 71 becomes a display Figure 8A The user interface 81 is shown as an example.

[0311] Once the Gallery app is opened, the user interface 81 can display images and videos stored in the Gallery app. The user interface 81 may include video thumbnails 801.

[0312] When the phone detects a user-instructed action to enable video thumbnails, such as clicking on video thumbnail 801, it can display... Figure 8A The exemplary user interface 81 becomes a display Figure 8B The user interface 82 shown is an example.

[0313] The user interface 82 displays an unplayed video. The user interface 82 may include controls 802a and 803a.

[0314] When the phone detects a user-instructed action to play a video, such as clicking on control 803a, the phone begins playing the recorded video.

[0315] When the mobile phone detects a user-instructed action to activate the motion recognition function, such as when the user clicks on control 802a, it can display... Figure 8B The exemplary user interface 82 becomes a display Figure 8C The user interface 83 is shown as an example.

[0316] After the phone's motion recognition function is enabled, motion recognition begins from the first frame of the video. When the phone detects a person's face turning its head in the video, it recommends a video editing app to the user. The user interface 83 may include control 804a.

[0317] When the phone detects a user-instructed action to open a video editing app, such as clicking on control 804a, it can display... Figure 8C The user interface 83 shown in the example becomes the video editing APP interface.

[0318] Among them, control 802a is used to enable the motion recognition function, control 803a is used to play the video on the current interface, and control 804a is used to jump to the video editing APP.

[0319] For details on the implementation of the fourth interface, please refer to [link / reference]. Figure 8BThe description of the user interface 82 shown is provided below. For the specific implementation of the fifth interface, please refer to [link / reference]. Figure 8C The user interface 83 is shown in the description. The fourth operation includes a user clicking on control 802a, and the fifth operation includes a user clicking on control 803a. The fourth control includes 802a, and the fifth control includes 803a.

[0320] In summary, for camera scenarios, Figures 8A-8C This document describes a method for users to enable motion recognition functionality on their mobile phones. Users open the video they want to perform motion recognition on within the Gallery app and then enable the motion recognition function on the video playback screen.

[0321] As can be seen, in photo and video shooting scenarios, electronic devices can activate motion recognition via controls on the user interface. Once motion recognition is enabled, the electronic device can capture target facial movements using the methods described above. When a target movement is detected in the image, the electronic device can provide functions such as editing short videos, creating emojis, facial unlocking, and security verification based on the captured image, enriching the user experience.

[0322] For example, this application provides an electronic device, including: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions in the memory to cause the electronic device to execute the action detection method in the preceding embodiments.

[0323] For example, this application provides a chip system applied to an electronic device including a memory and a sensor; the chip system includes a processor; when the processor executes computer instructions stored in the memory, the electronic device executes the motion detection method described in the preceding embodiments.

[0324] For example, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program being processed by a processor to cause an electronic device to implement the action detection method described in the preceding embodiments.

[0325] For example, this application provides a computer program product, including: execution instructions, the execution instructions are stored in a readable storage medium, at least one processor of an electronic device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to enable the electronic device to implement the action detection method in the foregoing embodiments.

[0326] In the above embodiments, all or part of the functionality can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0327] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A motion detection method, characterized by, The method is applied to an electronic device, and the method comprises: performing face detection on a first frame of images in a first period to obtain position information of n face detection frames, the position information of the n face detection frames being used to indicate positions of faces corresponding to the face detection frames in the first frame of images in the first period, n being a positive integer; performing similarity matching on the faces in the first frame of images in the first period according to the position information of the n face detection frames to obtain position information of p face detection frames in the first frame of images in the first period, and performing face tracking on remaining frame images other than the first frame of images in the first period to obtain position information of the n face detection frames in the remaining frame images other than the first frame of images in the first period respectively, p being a positive integer; performing key point detection on the faces in each frame of images in the first period according to the position information of the p face detection frames in the first frame of images in the first period and the position information of the n face detection frames in the remaining frame images other than the first frame of images in the first period to obtain position information of p groups of face key point sequences in each frame of images in the first period; performing action recognition on x frames of images in the first period according to the position information of the p groups of face key point sequences in the x frames of images in the first period to obtain a recognition result, the recognition result being used to indicate whether a target action exists in the x frames of images, x being a positive integer.

2. The method of claim 1, wherein, performing similarity matching on the faces in the first frame of images in the first period according to the position information of the n face detection frames to obtain position information of p face detection frames in the first frame of images in the first period, comprising: performing similarity matching on the faces in the first frame of images in the first period and the first frame of images in a second period according to position information of m face detection frames and the position information of the n face detection frames to obtain the position information of the p face detection frames in the first frame of images in the first period; wherein the position information of the m face detection frames is used to indicate positions of faces corresponding to the face detection frames in the first frame of images in the second period, the second period being a period before the first period.

3. The method according to claim 1 or 2, characterized in that, The method specifically comprises: performing face detection on the first frame of images in the first period to obtain a detection result; when the detection result is that a face exists, determining the position information of the n face detection frames.

4. The method of claim 3, wherein, The method further comprises: when the detection result is that a face does not exist and no human body exists, increasing a time length of a period after the first period until a first frame of images in a third period has a face, and restoring the time length of the period after the third period, the third period being a period after the first period.

5. The method of claim 3, wherein, The method further comprises: when the detection result is that a face does not exist and a human body in motion exists, shortening a time length of a period after the first period until a first frame of images in a third period has a face, and restoring the time length of the period after the third period, the third period being a period after the first period.

6. The method according to any one of claims 1-5, characterized in that, The method further comprises: display a first interface, the first interface including a first control; receive a first operation on the first control, the first operation being used to start action recognition; perform action recognition on the first frame of image, including: in response to the first operation, performing action recognition on the first frame of image.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: display a first interface, the first interface including a first control; in response to a first operation on the first control, display a second interface, the second interface including a second control; receive a second operation on the second control, the second operation being used to start action recognition; perform action recognition on the first frame of image, including: in response to the second operation, performing action recognition on the first frame of image.

8. The method according to claim 6 or 7, characterized in that, The method further includes: display a third interface, the third interface including a third control, the third control being used to start a first application; receive a third operation on the third control; display a first interface, including: in response to the third operation, display the first interface, and collect images in the first period.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: display a fourth interface, the fourth interface being used to play a first video, the fourth interface including a fourth control; receive a fourth operation on the fourth control, the fourth operation being used to start action recognition; perform face detection on a first frame of image in the first period, including: in response to the fourth operation, perform face detection on the first frame of image in the first period in the first video.

10. The method according to any one of claims 1-8, characterized in that, The method further includes: display a fourth interface, the fourth interface including a fourth control and a fifth control, the fifth control being used to trigger playing of a first video; receive a fourth operation on the fourth control, the fourth operation being used to start action recognition; receive a fifth operation on the fifth control; perform face detection on a first frame of image in the first period, including: in response to the fifth operation, display a fifth interface, and perform face detection on the first frame of image in the first period in the first video, the fifth interface being used to play the first video.

11. The method according to any one of claims 1-10, characterized in that, The type of target action includes looking back, shaking head, nodding and blinking.

12. The method of any one of claims 1-11, wherein, the x frames of image include a plurality of consecutive frames of image in the first period for the p sets of face key point sequences; or, the x frames of image include a plurality of frames of image at intervals in the first period for the p sets of face key point sequences.

13. An electronic device, comprising: comprise: a processor; when the processor executes one or more computer programs stored in the memory, the electronic device performs the action detection method of any one of claims 1-12.

14. An electronic device, comprising: comprise: one or more processors; a memory; wherein the memory stores one or more computer programs, the one or more computer programs comprising instructions that, when executed on the electronic device, cause the electronic device to perform the action detection method of any one of claims 1-12.

15. A computer-readable storage medium having stored therein instructions, the computer-readable storage medium comprising: The instructions, when executed on the electronic device, cause the electronic device to perform the action detection method of any one of claims 1-12.

Images