Interaction method and apparatus, electronic device, and storage medium

Abstract

The application discloses an interaction method and device, electronic equipment and storage medium, and belongs to the technical field of communication. The method comprises the following steps: acquiring N images, wherein the images comprise an eye area and a brow area, the eye area comprises a first eye and a second eye, the brow area comprises a first brow and a second brow, N is a positive integer; calculating opening-closing difference value information of the first eye and the second eye and swing difference value information of the first brow and the second brow according to each image in the N images; in the case that the opening-closing difference value information satisfies a first preset condition and the swing difference value information satisfies a second preset condition, counting a blink state according to the N images; and performing a target operation according to the blink state.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication, and particularly relates to an interaction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the development of science and technology, a user can touch an electronic device to interact with the electronic device. For example, in the interaction logic of an instant messaging application, if text, pictures, audio and video information needs to be transmitted and shared between users, the user needs to perform a large number of clicking and swiping operations on the screen by using a finger to complete the operation. As a result, the interaction process between the user and the electronic device is complicated, the operation is complex, and the efficiency of the user using the electronic device is affected. SUMMARY

[0003] The embodiments of the present application provide an interaction method, device, electronic equipment and storage medium, and can solve the problem that the interaction process between a user and an electronic device is complicated, the operation is complex, and the efficiency of the user using the electronic device is low in the prior art.

[0004] In a first aspect, the embodiments of the present application provide an interaction method applied to an electronic device, which can include the following steps.

[0005] N images are acquired, the image includes an eye region and a brow region, the eye region includes a first eye and a second eye, the brow region includes a first brow and a second brow, N is a positive integer;

[0006] According to each of the N images, the opening and closing difference value information of the first eye and the second eye and the swing difference value information of the first brow and the second brow are calculated;

[0007] In a case where the opening and closing difference value information meets a first preset condition and the swing difference value information meets a second preset condition, the blinking state is counted according to the N images;

[0008] According to the blinking state, a target operation is performed.

[0009] In a second aspect, the embodiments of the present application provide an interaction device applied to an electronic device, which can include the following steps.

[0010] The acquisition module is configured to acquire N images, the image includes an eye region and a brow region, the eye region includes a first eye and a second eye, the brow region includes a first brow and a second brow, N is a positive integer;

[0011] The calculation module is configured to calculate, according to each of the N images, the opening and closing difference value information of the first eye and the second eye and the swing difference value information of the first brow and the second brow;

[0012] The statistical module is configured to, in a case where the opening-closing difference information meets a first preset condition and the swing difference information meets a second preset condition, statistically determine the blink state according to the N images.

[0013] The execution module is configured to execute a target operation according to the blink state.

[0014] In a third aspect, an electronic device is provided. The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the interaction method according to the first aspect.

[0015] In a fourth aspect, a readable storage medium is provided. The readable storage medium stores a program or instructions, and the program or instructions are executed by a processor to implement the steps of the interaction method according to the first aspect.

[0016] In a fifth aspect, a chip is provided. The chip includes a processor and a display interface. The display interface is coupled to the processor. The processor is configured to execute a program or instructions to implement the steps of the interaction method according to the first aspect.

[0017] In a sixth aspect, a computer program product is provided. The program product is stored in a storage medium. The program product is executed by at least one processor to implement the steps of the interaction method according to the first aspect.

[0018] In the embodiments of the present application, N images including an eye region and a brow region are acquired. The eye region includes a first eye and a second eye, and the brow region includes a first brow and a second brow. Then, opening-closing difference information of the first eye and the second eye and swing difference information of the first brow and the second brow are calculated according to each of the N images. Then, in a case where the opening-closing difference information meets a first preset condition and the swing difference information meets a second preset condition, a blink state is statistically determined according to the N images. Then, a target operation is executed according to the blink state. Here, the operation of the electronic device is controlled by the eyes, and the user does not need to perform a large number of clicking, swiping and other operations on the screen of the electronic device by fingers. Thus, the interaction process between the user and the electronic device is simple and convenient to operate. In addition, in the embodiments of the present application, the opening-closing difference information related to the two eyes and the swing difference information related to the two brows are used to determine whether the user is in a state of operating the electronic device by the eyes, so as to avoid the case that the electronic device is mistakenly operated due to the blinking of the user in a normal state. The accuracy of determining that the user is in a state of operating the electronic device by the eyes is improved while maintaining the sensitivity of the electronic device controlled by the eyes, so as to improve the efficiency of the user using the electronic device. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1A flow chart of an interaction method provided for an embodiment of the present application;

[0020] Figure 2 A structural diagram of an electronic device for an interaction method provided for an embodiment of the present application;

[0021] Figure 3 An angle diagram of image collection for an interaction method provided for an embodiment of the present application;

[0022] Figure 4 An eye region and eyebrow region diagram for an interaction method provided for an embodiment of the present application;

[0023] Figure 5 An eye region diagram for an interaction method provided for an embodiment of the present application;

[0024] Figure 6 A first distance diagram for an interaction method provided for an embodiment of the present application;

[0025] Figure 7 A third distance diagram for an interaction method provided for an embodiment of the present application;

[0026] Figure 8 An eyebrow curve diagram for an interaction method provided for an embodiment of the present application;

[0027] Figure 9 A structural diagram of an interaction device provided for an embodiment of the present application;

[0028] Figure 10 A structural diagram of an electronic device provided for an embodiment of the present application;

[0029] Figure 11 A hardware structural diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and are not limited in number, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the objects before and after are in an "or" relationship.

[0032] To solve the problems in the related art, the embodiments of the present application provide an interaction method, which confirms whether a user is in a state of operating an electronic device through eye operation through open-close difference value information related to two eyes and swing difference value information related to two eyebrows, and if the user wears glasses, it can also determine whether the user is in a state of operating the electronic device through eye operation according to the open-close difference value information, the swing difference value information and the height change difference value information of the frame in the adjacent image, avoiding the case that the user blinks in the normal state to cause the electronic device to be misoperated, and realizing that the accuracy of determining that the user is in a state of operating the electronic device through eye operation can be improved while maintaining the sensitivity of eye control of the electronic device.

[0033] In addition, if the electronic device is connected with a control device (such as a gamepad, a keyboard, a mouse), it can be defaulted that the user will operate the electronic device through the control device, and then, if the user sometimes places the control device connected with the electronic device in a position that the user cannot reach or does not want to operate the electronic device through the control device, etc. cannot operate the electronic device through the control device, the interaction method of operating the electronic device through eye operation provided in the embodiments of the present application can be used, so as to reduce the false triggering and perform power optimization, and further improve the efficiency of the user using the electronic device.

[0034] It should be noted that the electronic device in the embodiments of the present application can be a mobile phone, a tablet computer, a notebook computer, etc., and can also be a wearable device (specifically, a head-mounted device) such as a virtual reality (VR) glasses (headgear), an augmented reality (AR) glasses (headgear), a mixed reality (MR) glasses (headgear) and the like, which is a device with image acquisition and calculation function. The embodiments of the present application take the head-mounted device as an example for description, and the electronic device is not limited herein.

[0035] Based on this, the interaction method provided by the embodiments of the present application will be described in detail below in combination with the accompanying Figures 1-8 , through specific embodiments and application scenarios.

[0036] First, in conjunction with Figure 1 The interactive method provided by the embodiments of the present application is described in detail.

[0037] Figure 1 The flowchart of the interactive method provided by the embodiments of the present application.

[0038] As Figure 1 shown, the interactive method provided by the embodiments of the present application can be applied to an electronic device, and the method can include the following steps:

[0039] Step 110, obtaining N images, the images including an eye region and a brow region, the eye region including a first eye and a second eye, the brow region including a first brow and a second brow, N being a positive integer; Step 120, according to each of the N images, calculating open-close difference value information of the first eye and the second eye and swing difference value information of the first brow and the second brow; Step 130, in the case that the open-close difference value information meets a first preset condition and the swing difference value information meets a second preset condition, according to the N images, counting a blink state; Step 140, according to the blink state, performing a target operation.

[0040] In this way, the operation of the electronic device is controlled by the eyes, without the user needing to perform a large number of clicking, swiping and other operations on the screen of the electronic device by fingers, so that the interaction process between the user and the electronic device is simple and the operation is convenient; and in the embodiments of the present application, whether the user is in the state of operating the electronic device by the eyes is determined according to the open-close difference value information related to the two eyes and the swing difference value information related to the two brows, so as to avoid the case that the electronic device is mistakenly operated due to blinking of the user in a normal state, and to realize the improvement of the accuracy of determining that the user is in the state of operating the electronic device by the eyes while maintaining the sensitivity of the electronic device controlled by the eyes, thereby improving the efficiency of the user using the electronic device.

[0041] The above steps are described in detail as follows.

[0042] First, in relation to step 110, in one or more possible embodiments, the N images can be captured by a camera module of the electronic device.

[0043] Exemplarily, as Figure 2 shown, the electronic device can be a head-mounted device, which can include display modules corresponding to the two eyes, such as display module 001 and display module 002, and at least two camera modules, such as camera module 003 and camera module 004, as shown in Figure 2 . Among them, the display module is used to display content for browsing, and the camera module is used to collect images of the brow and eye of the user wearing the head-mounted device.

[0044] Specifically, the display module 001 and the display module 002 can display the interface content after the user operates the eye operation device. In addition, as shown in Figure 3 The camera module 003 can capture images including the user's eyebrow, the facial region in the preset range above the eyebrow, and the eye, and the camera module 004 can capture images of the user's eye and the facial region below the eye, so as to ensure that the camera module 003 and the camera module 004 capture more complete images of the user's eye and eyebrow, and ensure that more clear images of the eye and eyebrow are captured.

[0045] It should be noted that, as shown in Figure 4 The eye region 40 in the embodiment of the present application includes a first eye (such as the left eye) and a second eye (such as the right eye), and the eye includes the eyeball and the region around the eyeball, such as the eyeball 401, the upper eyelid 402, and the lower eyelid 403. Among them, the upper eyelid 402 and the lower eyelid 403 are located in front of the eyeball, and the gap between the upper eyelid 402 and the lower eyelid 403 is called the palpebral fissure. Specifically, as shown in Figure 5 , the eyeball 401 can include the pupil 4011, the iris 4012, and the sclera 4013. The upper eyelid 404 can include the upper eyelid edge 4041 and the (partial) eyelid region above the upper eyelid edge, and the lower eyelid 405 can include the lower eyelid edge 4042 and the (partial) eye bag region below the lower eyelid edge.

[0046] As shown in Figure 4 The eyebrow region 41 in the embodiment of the present application includes a first eyebrow (left eyebrow) corresponding to the first eye and a second eyebrow (right eyebrow) corresponding to the second eye. The eyebrow is an arc-shaped protrusion located between the forehead and the eye region, and is composed of a cluster of short hairs, i.e., the eyebrow. The eyebrow includes the brow 411, the brow peak 412, and the brow tip 413.

[0047] In another or more possible embodiments, the electronic device can further include an infrared sensor for capturing the up-and-down jumping state of the muscles of the user's eye and eyebrow, the distance from the infrared sensor, and the like, so as to determine the eye region and the eyebrow region, facilitate the calculation of the opening-closing difference value information, the swinging difference value information, and the height change difference value information, and the like in the later stage.

[0048] Next, referring to step 120, the opening-closing difference value information and the swinging difference value information involved in the embodiment of the present application can be calculated in the following manner.

[0049] The opening-closing difference value information can be calculated in the following manner. In one or more possible embodiments, the first eye includes a first upper eyelid and a first lower eyelid, and the second eye includes a second upper eyelid and a second lower eyelid. Based on this, the step 120 can specifically include:

[0050] According to each image, a first distance between the first upper eyelid and the first lower eyelid at the i-th moment and a second distance between the second upper eyelid and the second lower eyelid at the i-th moment are obtained, the first distance being the maximum vertical distance between the first upper eyelid and the first lower eyelid, and the second distance being the maximum vertical distance between the second upper eyelid and the second lower eyelid;

[0051] A difference between the first distance and the second distance is taken as opening-closing difference information of the first eye and the second eye at the i-th moment, i being a positive integer, i corresponding to a time of shooting the image.

[0052] Exemplarily, since the shooting time of each image in the N images is different, each image corresponds to a shooting moment, and thus the first distance between the first upper eyelid and the first lower eyelid at the i-th moment in the image can be obtained. Specifically, as shown in Figure 6 for ease of understanding, here, the highest point of the first upper eyelid in the first eye, i.e., the left eye, along the upper eyelid to the lowest point of the lower eyelid in the first lower eyelid is taken as the first distance d 左眼i , and the second distance d 右眼i of the second eye, i.e., the right eye, is taken as the distance between the highest point of the second upper eyelid along the upper eyelid to the lowest point of the lower eyelid along the lower eyelid. i 左眼i 右眼i .

[0053] Further, the swing difference information can be calculated in the following two ways, as shown below. In one or more possible embodiments, the first eyebrow includes a first eyebrow peak, and the second eyebrow includes a second eyebrow peak, the first eyebrow corresponding to the first eye, and the second eyebrow corresponding to the second eye, based on which the step 120 specifically can include:

[0054] According to each image, a third distance between the first eyebrow peak and the first lower eyelid at the i-th moment and a fourth distance between the second eyebrow peak and the second lower eyelid at the i-th moment are obtained, the third distance being the maximum vertical distance between the first eyebrow peak and the first lower eyelid, and the fourth distance being the maximum vertical distance between the second eyebrow peak and the second lower eyelid;

[0055] A difference between the third distance and the fourth distance is taken as swing difference information of the first eyebrow and the second eyebrow at the i-th moment.

[0056] Exemplarily, as shown in Figure 7 , still taking the first eyebrow (left eyebrow) and the first eye (left eye) as an example for illustration, the distance from the first eyebrow peak (i.e., the highest point of the first eyebrow) of the first eyebrow to the lowest point of the first lower eyelid is determined as the third distance d 左眉i , and the fourth distance d 右眉i corresponding to the second eyebrow (right eyebrow) and the second eye (right eye) is determined as the distance between the highest point of the second upper eyelid along the upper eyelid to the lowest point of the lower eyelid along the lower eyelid. i 左眉i 右眉i .​​​​

[0057] In another or more possible embodiments, based on this, the step 120 can specifically include:

[0058] image recognition is performed on the first eyebrow and the second eyebrow in the adjacent first image in the N images, to obtain a first eyebrow curve corresponding to the first eyebrow and a second eyebrow curve corresponding to the second eyebrow at the jth moment, and a third eyebrow curve corresponding to the first eyebrow and a fourth eyebrow curve corresponding to the second eyebrow at the (j+1)th moment, j being a positive integer, the jth moment and the (j+1)th moment corresponding to the adjacent first image;

[0059] comparing the first eyebrow curve and the third eyebrow curve to obtain a first swing curvature corresponding to the first eyebrow, and comparing the second eyebrow curve and the fourth eyebrow curve to obtain a second swing curvature corresponding to the second eyebrow;

[0060] determining the first swing curvature and the second swing curvature as swing difference information.

[0061] Exemplarily, as shown in Figure 8 if the adjacent first image in the N images is the first image 1 and the first image 2, the first eyebrow and the second eyebrow in the first image 1 and the first eyebrow and the second eyebrow in the first image 2 can be image-recognized to obtain the first eyebrow curves 1(1) and 1(2), and the second eyebrow curves 2(1) and 2(2), and the first swing curvature is calculated according to the first eyebrow curve 1(1) and the second eyebrow curve 2(1), and the second swing curvature is calculated according to the first eyebrow curve 1(2) and the second eyebrow curve 2(2).

[0062] Further, in relation to the step 130, the user can be determined to be in the state of operating the electronic device through the eye in two ways, which are specifically described in detail as follows.

[0063] In one or more possible embodiments, if the value of the opening-closing difference information is greater than or equal to a first preset opening-closing difference value, and the value of the swing difference information is greater than or equal to a second preset opening-closing difference value, it is determined that the opening-closing difference information satisfies the first preset condition, and the swing difference information satisfies the second preset condition.

[0064] In another or more possible embodiments, the interactive method provided by the embodiments of the present application can be applied not only to the scene where the user is naked eye, i.e. without wearing glasses, but also to the scene where the user wears glasses, i.e. with lenses. Therefore, in addition to determining whether the user is in the state of operating the electronic device through the eye according to the above opening-closing difference information and swing difference information, if the user wears glasses, the user can also be determined to be in the state of operating the electronic device through the eye according to the opening-closing difference information, the swing difference information and the height change difference information of the frame in the adjacent image. Based on this, before the step 130, the interactive method can further include:

[0065] In a case where the image further comprises a glasses region, the glasses region comprising a first glasses frame corresponding to the first eye and a second glasses frame corresponding to the second eye, the first height change difference information of the first glasses frame at the rth moment and the first glasses frame at the (r+1)th moment and the second height change difference information of the second glasses frame at the rth moment and the second glasses frame at the (r+1)th moment are calculated according to the second adjacent image in the N images, r being a positive integer, the rth moment and the (r+1)th moment corresponding to the second adjacent image.

[0066] Based on this, the step 130 can specifically include:

[0067] In a case where the first height change difference information or the second height change difference information satisfies a third preset condition, the blink state is counted according to the N images.

[0068] Exemplarily, when the user wears glasses, in addition to the manner of determining whether the user is in the state of operating the electronic device through the eyes according to the opening-closing difference information and the swinging difference information, the height change difference information of the glasses frame can also be combined to determine whether the user is in the state of operating the electronic device through the eyes.

[0069] It should be noted that before the step 130 is performed, the interaction method can further include a step of setting the preset threshold (preset value), preset condition and the like in the present application. That is, if the preset information corresponding to the user in the N images is not included in the electronic device, a tutorial recording mode is entered, and a prompt message is played or displayed to prompt the user to make a specified action, such as blinking both eyes, blinking the left eye and blinking the right eye, so as to record the following information by the electronic device: the eye opening distance (or height) in the open left and right eye states determined as the opening-closing difference value, and the preset swinging difference value information can be taken as the minimum value of the eyebrow peak (i.e. the maximum value of the eyebrow) to the lower eyelid line when the eyes are open, and the minimum value of the eyebrow peak (i.e. the maximum value of the eyebrow) to the lower eyelid line when the eyes are closed, that is, H1 and H2. In this way, the preset swinging difference value information can be taken as (H1+H2) / 2, and the glasses frame is the same as this, which will not be described here.

[0070] Here, based on the above two cases of determining that the user is in the state of operating the electronic device through the eyes, the present application provides the following two ways of calculating the blink state.

[0071] In one or more possible embodiments, the step 130 can specifically include:

[0072] M third images in a first preset time period are extracted from the N images, M being a positive integer greater than 1;

[0073] calculate a fifth distance of the first upper eyelid and the first lower eyelid, target opening-closing difference value information corresponding to each third image and a first eyebrow center height of the first eyebrow according to each third image in the M third images;

[0074] In a case that a preset number of fifth distances in the M fifth distances are less than or equal to a preset distance, a value of the target opening-closing difference value information is greater than or equal to a value of preset opening-closing difference value information, and the first eyebrow center height is less than or equal to a preset eyebrow center height, an eye blinking state is obtained based on the first preset time period and the preset number.

[0075] Exemplarily, if the fifth distance is less than 4 mm for 3 times within 2 seconds, and the target opening-closing difference value information is greater than or equal to 3 mm for 2 times, and the first eyebrow center height is less than a threshold value of 2 mm at the same time, it is determined that the preset number is 2 times.

[0076] In this way, the case that the user blinks in a normal state to cause a false operation of the electronic device is avoided, and the accuracy of determining that the user is in a state of operating the electronic device by the eye is improved while maintaining the sensitivity of the eye in controlling the electronic device.

[0077] In another or more possible embodiments, the eye blinking state can also be calculated according to the pupil and iris area, the eye white area and the area covering the pupil and iris, and based on this, the step 130 can specifically include:

[0078] extract L fourth images in a second preset time period from N images, L being a positive integer greater than 1;

[0079] According to each fourth image in the L fourth images, obtain first eyelid fissure information of the first eye and second eyelid fissure information of the second eye in each fourth image;

[0080] Compare the first eyelid fissure information and the second eyelid fissure information between at least two adjacent images in the L fourth images respectively, to obtain a blinking frequency in the L fourth images;

[0081] Obtain an eye blinking state according to the second preset time period and the blinking frequency;

[0082] The first eyelid fissure information includes a first pupil and iris area, a first eye white area and a first area covering the pupil and iris, and the second eyelid fissure information includes a second pupil and iris area, a second eye white area and a second area covering the pupil and iris.

[0083] Exemplarily, S p左眼1 is the first eye white area, S p左眼2 is the first area covering the pupil and iris and S p左眼3 is the first pupil and iris area, at this time, S p左眼2 / S p左眼1= 10:1 or S p左眼2 / S p左眼3 = 15:1, it indicates that the left eye is in the open state, and by analogy, in the case of S p左眼2 / S p左眼1 = 15:1 or S p左眼2 / S p左眼3 = 30:1, it indicates that the left eye is in the closed state, and by contrast, if S p左眼2 / S p左眼1 is less than the preset value, it indicates that the left eye is in the open state, and S p左眼2 / S p左眼1 is greater than or equal to the preset value, it indicates that the left eye is in the closed state, and the right eye is by analogy, which will not be repeated here.

[0084] Thus, based on the above two cases where the user is not in the state of operating the electronic device through the eye operation, no response is given.

[0085] Then, step 140 is involved, which in one or more possible embodiments can specifically include:

[0086] According to the preset blinking state and the associated information of the preset operation, the target operation corresponding to the blinking state is executed;

[0087] Among them, the blinking state includes at least one of the following: the blinking frequency of the first eye, the blinking frequency of the second eye, and the blinking state of the first eye and the second eye.

[0088] Exemplarily, the single eye working time can be separated for the left eye and the right eye, or combined together, wherein if the blinking frequency of the single eye is 1 time in 1 second, the target operation is a single game machine; if the blinking frequency of the single eye is 2 times in 1 second, the target operation is a double game machine; if the left and right eyes are sequentially blinked 1 time in 1 second, the target operation is the display screen theme switching. By analogy, the left eye blinks 2 times in 1 second, and the left and right eyes each blink 1 time, different target operations are executed.

[0089] In addition, in other embodiments, if the electronic device is connected with a control device (such as a handle, a keyboard, a mouse), it can be defaulted that the user will operate the electronic device through the control device, and then, if the user sometimes places the control device connected with the electronic device in a position that the user cannot reach or does not want to operate the electronic device through the control device, etc. cannot operate the electronic device through the control device, the above-mentioned eye operation interaction mode for operating the electronic device can be used. In order to reduce (power consumption), the embodiments of the present application can further include, before step 220:

[0090] In the case of connecting the control device, the control signal sent by the control device is received;

[0091] Based on this, the condition triggering the execution of step 220 is that, in the case that no control signal sent by the control device is received within a third preset time period, the opening-closing difference value information of the first eye and the second eye and the swinging difference value information of the first eyebrow and the second eyebrow are calculated according to each of the N images.

[0092] Thus, if the electronic device is connected with the control device (such as a gamepad, a keyboard, or a mouse), it can be assumed that the user will operate the electronic device through the control device. Then, if the user sometimes places the control device that has been connected with the electronic device in a position that cannot be reached by the user or does not want to operate the electronic device through the control device, and the like, the electronic device cannot be operated through the control device, and the interactive mode of operating the electronic device through the eye provided in the embodiment of the present application can be used, so as to reduce the false triggering and optimize the power, and further improve the efficiency of the user using the electronic device.

[0093] In summary, the embodiment of the present application provides an interactive method, which confirms whether the user is in the state of operating the electronic device through the eye by using the opening-closing difference value information related to the two eyes and the swinging difference value information related to the two eyebrows, and if the user wears glasses, the state of the user operating the electronic device through the eye can also be determined according to the opening-closing difference value information, the swinging difference value information, and the height change difference value information of the adjacent images in the frame, so as to avoid the case that the user blinks in the normal state and causes the electronic device to be operated by mistake, and the accuracy of determining that the user is in the state of operating the electronic device through the eye can be improved while maintaining the sensitivity of the eye controlling the electronic device.

[0094] In addition, if the electronic device is connected with the control device (such as a gamepad, a keyboard, or a mouse), it can be assumed that the user will operate the electronic device through the control device. Then, if the user sometimes places the control device that has been connected with the electronic device in a position that cannot be reached by the user or does not want to operate the electronic device through the control device, and the like, the electronic device cannot be operated through the control device, and the interactive mode of operating the electronic device through the eye provided in the embodiment of the present application can be used, so as to reduce the false triggering and optimize the power, and further improve the efficiency of the user using the electronic device.

[0095] The interactive method provided in the embodiment of the present application can be executed by the interactive device.

[0096] Based on the same inventive concept, the present application also provides an interactive device. The interactive device will be described in detail in combination with Figure 9

[0097] Figure 9 FIG. 1 is a structural schematic diagram of an interactive device provided in an embodiment of the present application.

[0098] As​Figure 9 As shown, the interaction device 90 is applied to an electronic device, and specifically can include:

[0099] The acquisition module 901 is configured to acquire N images, the images including an eye region and a brow region, the eye region including a first eye and a second eye, the brow region including a first brow and a second brow, N being a positive integer;

[0100] The calculation module 902 is configured to calculate, according to each of the N images, open-close difference value information of the first eye and the second eye and swing difference value information of the first brow and the second brow.

[0101] The statistics module 904 is configured to, in a case where the open-close difference value information satisfies a first preset condition and the swing difference value information satisfies a second preset condition, statistically analyze a blink state according to the N images.

[0102] The execution module 904 is configured to perform a target operation according to the blink state.

[0103] The interaction device 90 is described in detail as follows:

[0104] In one or more possible embodiments, the calculation module 902 can be specifically configured to, in a case where the first eye includes a first upper eyelid and a first lower eyelid, and the second eye includes a second upper eyelid and a second lower eyelid, acquire, according to each image, a first distance between the first upper eyelid and the first lower eyelid and a second distance between the second upper eyelid and the second lower eyelid at the ith moment, the first distance being a maximum vertical distance between the first upper eyelid and the first lower eyelid, and the second distance being a maximum vertical distance between the second upper eyelid and the second lower eyelid, i being a positive integer, i corresponding to a time of photographing the image.

[0105] The difference between the first distance and the second distance is taken as the open-close difference value information of the first eye and the second eye at the ith moment.

[0106] In another or more possible embodiments, the calculation module 902 can be specifically configured to, in a case where the first brow includes a first brow peak, the second brow includes a second brow peak, the first brow corresponds to the first eye, and the second brow corresponds to the second eye, acquire, according to each image, a third distance between the first brow peak and the first lower eyelid and a fourth distance between the second brow peak and the second lower eyelid at the ith moment, the third distance being a maximum vertical distance between the first brow peak and the first lower eyelid, and the fourth distance being a maximum vertical distance between the second brow peak and the second lower eyelid.

[0107] The difference between the third distance and the fourth distance is taken as the swing difference value information of the first brow and the second brow at the ith moment.

[0108] In still another or more possible embodiments, the calculation module 902 can be specifically configured to obtain, according to each image, a third distance between the first eyebrow peak and the first lower eyelid and a fourth distance between the second eyebrow peak and the second lower eyelid at the ith moment, the third distance being the maximum vertical distance between the first eyebrow peak and the first lower eyelid, and the fourth distance being the maximum vertical distance between the second eyebrow peak and the second lower eyelid.

[0109] The difference between the third distance and the fourth distance is taken as the swing difference information of the first eyebrow and the second eyebrow at the ith moment.

[0110] In still another or more possible embodiments, the calculation module 902 can be specifically configured to perform image recognition on the first eyebrow and the second eyebrow in the first image adjacent to the N images to obtain a first eyebrow curve corresponding to the first eyebrow and a second eyebrow curve corresponding to the second eyebrow at the jth moment, and a third eyebrow curve corresponding to the first eyebrow and a fourth eyebrow curve corresponding to the second eyebrow at the j+1th moment, j being a positive integer, the jth moment and the j+1th moment corresponding to the adjacent first image.

[0111] The first swing curvature corresponding to the first eyebrow is obtained by comparing the first eyebrow curve and the third eyebrow curve, and the second swing curvature corresponding to the second eyebrow is obtained by comparing the second eyebrow curve and the fourth eyebrow curve.

[0112] The first swing curvature and the second swing curvature are determined as the swing difference information.

[0113] In still another or more possible embodiments, the calculation module 902 in the embodiments of the present application can also be configured to, the image further comprising a glasses region, the glasses region comprising a first frame corresponding to the first eye and a second frame corresponding to the second eye, according to the second image adjacent to the N images, calculate first height change difference information of the first frame at the rth moment and the first frame at the r+1th moment, and second height change difference information of the second frame at the rth moment and the second frame at the r+1th moment, r being a positive integer, the rth moment and the r+1th moment corresponding to the adjacent second image.

[0114] The statistical module 904 can be specifically configured to, in a case where the first height change difference information or the second height change difference information satisfies a third preset condition, statistically obtain the blink state according to the N images.

[0115] In still another or more possible embodiments, the statistical module 904 can also be configured to, in a case where the first eye comprises a first upper eyelid and a first lower eyelid, extract M third images in a first preset time period from the N images, M being a positive integer greater than 1.

[0116] According to each of the M third images, a fifth distance of the first upper eyelid and the first lower eyelid, target opening-closing difference value information corresponding to each of the third images, and a first eyebrow center height of the first eyebrow are calculated.

[0117] In a case where a preset number of the M fifth distances are less than or equal to a preset distance, a value of the target opening-closing difference value information is greater than or equal to a preset opening-closing difference value information, and the first eyebrow center height is less than or equal to a preset eyebrow center height, an eye blinking state is obtained based on a first preset time period and the preset number.

[0118] In still another or more possible embodiments, the statistical module 904 can also be configured to extract L fourth images in a second preset time period from the N images, L being a positive integer greater than 1.

[0119] According to each of the L fourth images, first eyelid fissure information of the first eye and second eyelid fissure information of the second eye in each of the fourth images are obtained.

[0120] The first eyelid fissure information and the second eyelid fissure information between at least two adjacent images of the L fourth images are compared respectively to obtain a number of eye blinks in the L fourth images.

[0121] According to the second preset time period and the number of eye blinks, an eye blinking state is obtained.

[0122] The first eyelid fissure information includes a first pupil and iris area, a first white eye area, and a first area covering the pupil and iris, and the second eyelid fissure information includes a second pupil and iris area, a second white eye area, and a second area covering the pupil and iris.

[0123] In still another or more possible embodiments, the interaction device 90 in the embodiments of the present application can further include a receiving module.

[0124] The receiving module is configured to receive a control signal sent by the control device in a case where the control device is connected.

[0125] The calculation module 902 can be specifically configured to, in a case where no control signal sent by the control device is received in a third preset time period, calculate opening-closing difference value information of the first eye and the second eye and swing difference value information of the first eyebrow and the second eyebrow according to each of the N images.

[0126] In still another or more possible embodiments, the execution module 904 can be specifically configured to execute a target operation corresponding to the eye blinking state according to association information of the preset eye blinking state and the preset operation.

[0127] The interactive device in the embodiments of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiments of the present application are not limited in this regard.

[0128] The interactive device in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an IOS operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0129] The interactive device provided in the embodiments of the present application can implement the method embodiments to achieve the same technical effects, and thus repeated descriptions are omitted here. Figures 1 to 8 The method embodiments implement various processes to achieve the same technical effects, and thus repeated descriptions are omitted here.

[0130] Therefore, the interactive device provided in the embodiments of the present application confirms whether the user is in a state of operating an electronic device through eye operation by using the opening-closing difference value information related to the eyes and the swinging difference value information related to the eyebrows, and if the user wears glasses, the interactive device can further determine whether the user is in a state of operating an electronic device through eye operation according to the opening-closing difference value information, the swinging difference value information, and the height change difference value information of the glasses frames in the adjacent images, thereby avoiding the case that the user blinks in a normal state to cause an error operation of the electronic device, and achieving the accuracy of determining that the user is in a state of operating an electronic device through eye operation while maintaining the sensitivity of eye control of the electronic device.

[0131] In addition, if the electronic device is connected with a control device (such as a gamepad, a keyboard, or a mouse), the user can operate the electronic device through the control device by default. Then, if the user sometimes places the control device connected with the electronic device in a position that the user cannot reach or does not want to operate the electronic device through the control device, and the like, the user can operate the electronic device through the eye operation method provided in the embodiments of the present application, so as to reduce the false triggering and optimize the power, and further improve the efficiency of the user in using the electronic device.

[0132] Optionally, as shown in Figure 10 The embodiments of the present application further provide an electronic device 100, including a processor 1001 and a memory 1002, the memory 1002 stores a program or instructions which can be run on the processor 1001, the program or instructions are executed by the processor 1001 to realize the steps of the above-mentioned interaction method embodiments, and the same technical effects can be achieved, to avoid repetition, which will not be described here.

[0133] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device.

[0134] Figure 11 A hardware structure schematic diagram of an electronic device provided in the embodiments of the present application.

[0135] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, a processor 1110, and the like.

[0136] Those skilled in the art can understand that the electronic device 1100 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected with the processor 1110 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Figure 11 The electronic device structure shown in the above-mentioned figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which will not be described here.

[0137] In the embodiments of the present application, the processor 1110 is configured to acquire N images, the images comprising an eye region and a brow region, the eye region comprising a first eye and a second eye, the brow region comprising a first brow and a second brow, N being a positive integer. The processor 1110 is further configured to calculate, according to each of the N images, open-close difference information of the first eye and the second eye and swing difference information of the first brow and the second brow. The processor 1110 is further configured to, in a case where the open-close difference information satisfies a first preset condition and the swing difference information satisfies a second preset condition, statistically analyze a blink state according to the N images. The processor 1110 is further configured to perform a target operation according to the blink state.

[0138] In one or more possible embodiments, the processor 1110 can be specifically configured to, in a case where the first eye comprises a first upper eyelid and a first lower eyelid, and the second eye comprises a second upper eyelid and a second lower eyelid, acquire, according to each image, a first distance between the first upper eyelid and the first lower eyelid and a second distance between the second upper eyelid and the second lower eyelid at the ith moment, the first distance being a maximum vertical distance between the first upper eyelid and the first lower eyelid, and the second distance being a maximum vertical distance between the second upper eyelid and the second lower eyelid, i being a positive integer, i corresponding to a time of photographing the image.

[0139] The difference between the first distance and the second distance is taken as the open-close difference information of the first eye and the second eye at the ith moment.

[0140] In another or more possible embodiments, the processor 1110 can be specifically configured to, in a case where the first brow comprises a first brow peak, the second brow comprises a second brow peak, the first brow corresponds to the first eye, and the second brow corresponds to the second eye, acquire, according to each image, a third distance between the first brow peak and the first lower eyelid and a fourth distance between the second brow peak and the second lower eyelid at the ith moment, the third distance being a maximum vertical distance between the first brow peak and the first lower eyelid, and the fourth distance being a maximum vertical distance between the second brow peak and the second lower eyelid.

[0141] The difference between the third distance and the fourth distance is taken as the swing difference information of the first brow and the second brow at the ith moment.

[0142] In yet another or more possible embodiments, the processor 1110 can be specifically configured to acquire, according to each image, a third distance between the first brow peak and the first lower eyelid and a fourth distance between the second brow peak and the second lower eyelid at the ith moment, the third distance being a maximum vertical distance between the first brow peak and the first lower eyelid, and the fourth distance being a maximum vertical distance between the second brow peak and the second lower eyelid.

[0143] The difference between the third distance and the fourth distance is taken as the swing difference information of the first brow and the second brow at the ith moment.

[0144] In still another or more possible embodiments, the processor 1110 can be specifically configured to perform image recognition on the first eyebrow and the second eyebrow in a first image adjacent to the N images to obtain a first eyebrow curve corresponding to the first eyebrow and a second eyebrow curve corresponding to the second eyebrow at a jth moment, and a third eyebrow curve corresponding to the first eyebrow and a fourth eyebrow curve corresponding to the second eyebrow at a (j+1)th moment, j being a positive integer, the jth moment and the (j+1)th moment corresponding to the first image adjacent.

[0145] comparing the first eyebrow curve and the third eyebrow curve to obtain a first swing curvature corresponding to the first eyebrow, and comparing the second eyebrow curve and the fourth eyebrow curve to obtain a second swing curvature corresponding to the second eyebrow;

[0146] determining the first swing curvature and the second swing curvature as swing difference information.

[0147] In still another or more possible embodiments, the processor 1110 in the embodiments of the present application can also be configured to perform image recognition on the first image adjacent to the N images to obtain a first height change difference value information of the first mirror frame at a rth moment and a first mirror frame at a (r+1)th moment, and a second height change difference value information of the second mirror frame at the rth moment and the second mirror frame at the (r+1)th moment, r being a positive integer, the rth moment and the (r+1)th moment corresponding to the second image adjacent.

[0148] The processor 1110 can be specifically configured to, in a case where the first height change difference value information or the second height change difference value information satisfies a third preset condition, count the blink state according to the N images.

[0149] In still another or more possible embodiments, the processor 1110 can also be configured to, in a case where the first eye includes a first upper eyelid and a first lower eyelid, extract M third images in a first preset time period from the N images, M being a positive integer greater than 1.

[0150] According to each of the M third images, calculate a fifth distance of the first upper eyelid and the first lower eyelid, target opening and closing difference value information corresponding to each third image, and a first eyebrow hair center height of the first eyebrow;

[0151] In a case where a preset number of fifth distances in the M fifth distances are less than or equal to a preset distance, the value of the target opening and closing difference value information is greater than or equal to the value of the preset opening and closing difference value information, and the first eyebrow hair center height is less than or equal to a preset eyebrow hair center height, the blink state is obtained based on the first preset time period and the preset number.

[0152] In still another or more possible embodiments, the processor 1110 can also be configured to extract L fourth images in a second preset time period from the N images, L being a positive integer greater than 1.

[0153] According to each of the L fourth images, first eyelid crack information of the first eye and second eyelid crack information of the second eye in each fourth image are obtained.

[0154] The first eyelid crack information and the second eyelid crack information between at least two adjacent images in the L fourth images are compared respectively to obtain the number of blinks in the L fourth images.

[0155] According to the second preset time period and the number of blinks, a blink state is obtained.

[0156] The first eyelid crack information includes the first pupil and iris area, the first white eye area, and the area of the first covered pupil and iris, and the second eyelid crack information includes the second pupil and iris area, the second white eye area, and the area of the second covered pupil and iris.

[0157] In still another or more possible embodiments, the network module 1102 in the embodiments of the present application is configured to receive a control signal sent by a control device in the case of connecting the control device.

[0158] The processor 1110 can be specifically configured to, in the case that no control signal sent by the control device is received within a third preset time period, calculate the opening and closing difference value information of the first and second eyes and the swinging difference value information of the first and second eyebrows according to each image in the N images.

[0159] In still another or more possible embodiments, the processor 1110 can be specifically configured to execute a target operation corresponding to the blink state according to the association information of the preset blink state and the preset operation.

[0160] In this way, through the opening and closing difference value information related to the two eyes and the swinging difference value information related to the two eyebrows, it is determined whether the user is in the state of operating the electronic device through the eyes, and if the user wears glasses, it is also determined whether the user is in the state of operating the electronic device through the eyes according to the opening and closing difference value information, the swinging difference value information, and the height change difference value information of the frame in adjacent images, avoiding the case that the user blinks in the normal state to cause the electronic device to be operated by mistake, and realizing that the accuracy of determining that the user is in the state of operating the electronic device through the eyes can be improved while maintaining the sensitivity of controlling the electronic device through the eyes.

[0161] In addition, if the electronic device is connected with a control device (such as a gamepad, a keyboard, a mouse), the user can operate the electronic device by default through the control device, and then, if the user sometimes places the control device connected with the electronic device in a position that the user cannot reach or does not want to operate the electronic device through the control device, etc. cannot operate the electronic device through the control device, the user can operate the electronic device by using the eye operation method provided in the embodiments of the present application, thereby reducing the mis-triggering and optimizing the power, and further improving the efficiency of the user using the electronic device.

[0162] It should be understood that the input unit 1104 can include a graphics processor (GPU) 11041 and a microphone 11042. The graphics processor 11041 processes image data of a still image or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 can include a display panel, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 can include two parts of a touch detection device and a touch display. The other input devices 11072 can include, but are not limited to, a physical keyboard, function keys (such as volume display keys, on-off keys, etc.), a trackball, a mouse, a joystick, etc., which will not be described here.

[0163] The memory 1109 can be used to store software programs and various data, and can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 1109 can include a volatile memory or a non-volatile memory, or the memory 1109 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1109 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0164] The processor 1110 can include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes a wireless display signal, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1110.

[0165] The embodiments of the present application also provide a readable storage medium, and the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned interaction method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0166] The processor is a processor in the electronic device in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.

[0167] In addition, the embodiment of the present application further provides a chip, which comprises a processor and a display interface. The display interface is coupled with the processor. The processor is used to run programs or instructions, to realize various processes of the above-mentioned interactive method embodiments, and to achieve the same technical effects. To avoid repetition, details are not described herein.

[0168] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, and the like.

[0169] The embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to realize various processes of the above-mentioned interactive method embodiments, and to achieve the same technical effects. To avoid repetition, details are not described herein.

[0170] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0171] In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved, for example, the described method can be performed in an order different from the described order, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0173] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. An interaction method, characterized in that, include: Obtain N images, each image including an eye region and an eyebrow region. The eye region includes a first eye and a second eye, and the eyebrow region includes a first eyebrow and a second eyebrow. N is a positive integer. Based on each of the N images, calculate the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow; If the opening and closing difference information meets the first preset condition and the swing difference information meets the second preset condition, the blinking state is statistically analyzed based on the N images. Execute the target operation based on the blinking state; The first eye includes a first upper eyelid and a first lower eyelid, and the second eye includes a second upper eyelid and a second lower eyelid; the first eyebrow includes a first brow peak, and the second eyebrow includes a second brow peak, the first eyebrow corresponding to the first eye, and the second eyebrow corresponding to the second eye; the step of calculating the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow based on each of the N images includes: Based on each image, a first distance between the first upper eyelid and the first lower eyelid, and a second distance between the second upper eyelid and the second lower eyelid are obtained at time i. The first distance is the maximum vertical distance between the first upper eyelid and the first lower eyelid, and the second distance is the maximum vertical distance between the second upper eyelid and the second lower eyelid. The difference between the first distance and the second distance is used as the opening and closing difference information of the first eye and the second eye at time i, where i is a positive integer and corresponds to the time of image capture. Based on each image, a third distance between the first brow peak and the first lower eyelid, and a fourth distance between the second brow peak and the second lower eyelid are obtained at the i-th time. The third distance is the maximum vertical distance between the first brow peak and the first lower eyelid, and the fourth distance is the maximum vertical distance between the second brow peak and the second lower eyelid. The difference between the third distance and the fourth distance is used as the swing difference information between the first eyebrow and the second eyebrow at the i-th time.

2. The method according to claim 1, characterized in that, The step of calculating the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow based on each of the N images includes: Image recognition is performed on the first eyebrow and the second eyebrow in the adjacent first images among the N images to obtain the first eyebrow curve corresponding to the first eyebrow and the second eyebrow curve corresponding to the second eyebrow at time j, and the third eyebrow curve corresponding to the first eyebrow and the fourth eyebrow curve corresponding to the second eyebrow at time j+1, where j is a positive integer, and the time j and the time j+1 correspond to the adjacent first images. By comparing the first eyebrow curve and the third eyebrow curve, a first oscillation curvature corresponding to the first eyebrow is obtained; and by comparing the second eyebrow curve and the fourth eyebrow curve, a second oscillation curvature corresponding to the second eyebrow is obtained. The first oscillation curvature and the second oscillation curvature are determined as the oscillation difference information.

3. The method according to claim 1, characterized in that, The image also includes an eyeglasses region, which includes a first frame corresponding to the first eye and a second frame corresponding to the second eye; Before performing the target operation corresponding to the target blinking state based on the target blinking state of the eye region, the method further includes: Based on the adjacent second images among the N images, calculate the first height change difference information of the first frame at time r and the first frame at time r+1, and the second height change difference information of the second frame at time r and the second frame at time r+1, where r is a positive integer, and the time r and the time r+1 correspond to the adjacent second images; The step of calculating the blinking state based on the target blinking state of the eye region and based on the N images includes: If either the first height change difference information or the second height change difference information meets the third preset condition, blinking status is statistically analyzed based on the N images.

4. The method according to claim 1, characterized in that, The blinking state includes blinking frequency; the first eye includes a first upper eyelid and a first lower eyelid; the step of statistically analyzing the blinking state based on the N images includes: Extract M third images from the N images within a first preset time period, where M is a positive integer greater than 1; Calculate the fifth distance between the first upper eyelid and the first lower eyelid, the target opening and closing difference information corresponding to each of the M third images, and the first eyebrow center height of the first eyebrow based on each of the M third images; If, among M fifth distances, a predetermined number of fifth distances are less than or equal to a predetermined distance, the value of the target opening / closing difference information is greater than or equal to the value of the predetermined opening / closing difference information, and the height of the first eyebrow center is less than or equal to a predetermined eyebrow center height, the blinking frequency is obtained based on the first predetermined time period and the predetermined number of times.

5. The method according to claim 1, characterized in that, The blinking state includes blinking frequency; the step of statistically analyzing the blinking state based on the N images includes: Extract L fourth images within a second preset time period from the N images, where L is a positive integer greater than 1; Based on each of the L fourth images, obtain the first palpebral fissure information of the first eye and the second palpebral fissure information of the second eye in each fourth image; By comparing the first palpebral fissure information and the second palpebral fissure information between at least two adjacent images in the L fourth images, the number of blinks in the L fourth images is obtained; The blink frequency is obtained based on the second preset time period and the number of blinks; The first palpebral fissure information includes the area of ​​the first pupil and iris, the area of ​​the first sclera, and the area of ​​the first covering pupil and iris; the second palpebral fissure information includes the area of ​​the second pupil and iris, the area of ​​the second sclera, and the area of ​​the second covering pupil and iris.

6. The method according to claim 1, characterized in that, Before calculating the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow based on each of the N images, the method further includes: When connected to a control device, receive control signals sent by the control device; The step of calculating the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow based on each of the N images includes: If no control signal is received from the control device within the third preset time period, the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow are calculated based on each of the N images.

7. The method according to claim 1, characterized in that, The step of performing the target operation based on the blinking state includes: Based on the association information between the preset blinking state and the preset operation, the target operation corresponding to the blinking state is executed; wherein, the blinking state includes at least one of the following: the blinking frequency of the first eye, the blinking frequency of the second eye, and the blinking state of the first eye and the second eye combined.

8. An interactive device, characterized in that, include: An acquisition module is used to acquire N images, each image including an eye region and an eyebrow region. The eye region includes a first eye and a second eye, and the eyebrow region includes a first eyebrow and a second eyebrow. N is a positive integer. The calculation module is used to calculate the opening and closing difference information of the first eye and the second eye and the swing difference information of the first eyebrow and the second eyebrow based on each of the N images; The statistics module is used to count the blinking state based on the N images when the opening and closing difference information meets a first preset condition and the swing difference information meets a second preset condition. An execution module is used to perform a target operation based on the blinking state; The first eye includes a first upper eyelid and a first lower eyelid, and the second eye includes a second upper eyelid and a second lower eyelid; the first eyebrow includes a first brow peak, and the second eyebrow includes a second brow peak, the first eyebrow corresponds to the first eye, and the second eyebrow corresponds to the second eye; The calculation module is specifically used to, based on each image, obtain a first distance between the first upper eyelid and the first lower eyelid at time i, and a second distance between the second upper eyelid and the second lower eyelid, where the first distance is the maximum vertical distance between the first upper eyelid and the first lower eyelid, and the second distance is the maximum vertical distance between the second upper eyelid and the second lower eyelid; and use the difference between the first distance and the second distance as the opening and closing difference information of the first eye and the second eye at time i, where i is a positive integer and corresponds to the time of image capture; Based on each image, a third distance between the first brow peak and the first lower eyelid, and a fourth distance between the second brow peak and the second lower eyelid are obtained at the i-th time. The third distance is the maximum vertical distance between the first brow peak and the first lower eyelid, and the fourth distance is the maximum vertical distance between the second brow peak and the second lower eyelid. The difference between the third distance and the fourth distance is used as the swing difference information between the first eyebrow and the second eyebrow at the i-th time.

9. An electronic device, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the interactive method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the interactive method as described in any one of claims 1-7.

Images