Interaction method and device

Abstract

The invention discloses an interaction method and device, and the method can achieve the interaction with a target page only through facial actions, and greatly improves the operation convenience. In the aspect of interaction accuracy, the interaction instruction is determined by obtaining the feature points of the nose, eyebrows and eyes of the face, and compared with a traditional mode, the intention of the user can be reflected more meticulously. In the aspect of interaction modes, the design of the combination of various facial actions enables the user to freely select different operation modes according to own habits. And in the aspect of scene application, as the method only depends on facial actions and image information, the user can still complete interaction operation in a special environment that both hands are occupied or a screen cannot be touched, and the application range is greatly expanded. In terms of technical implementation cost, due to the fact that complex and expensive hardware does not need to be added and only a software algorithm needs to be optimized mainly based on an existing facial image recognition technology and a common equipment camera, cost is low, the technology is easier to popularize, and more users are benefited.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to an interaction method and device. Background Art

[0002] In the era of the rapid development of modern technology, smartphones have become an indispensable tool in people's lives. Their rich functions and convenient operations have greatly changed people's lifestyles. Currently, most smartphones mainly rely on touch screens for operation, such as swiping the screen to view content, clicking on icons to launch applications, dragging files for organization, etc. This touch operation method is simple, intuitive, and efficient for the vast majority of normal users.

[0003] However, for some special groups, such as people with physical disabilities, due to limited physical functions, they are unable to interact with the touch screen as flexibly as normal people using their fingers; there are also the elderly. As they age, the flexibility of their hands decreases, and it is also difficult to operate the touch screen. In order to meet the needs of this special group, non-contact operation methods have emerged.

[0004] Currently, traditional non-contact alternative operation methods mainly rely on speech recognition technology or specific hardware devices. Although speech recognition technology has achieved non-contact operation to a certain extent, it has many limitations. In a noisy environment, the accuracy of speech recognition will drop significantly, resulting in the inability to accurately convey operation instructions and affecting the user experience; moreover, for some users with speech impairments, the speech recognition operation method is simply inapplicable. The operation method relying on specific hardware devices often requires users to purchase and carry relevant devices additionally, which not only increases the usage cost but also reduces the convenience of use. At the same time, the compatibility of such devices may also be a problem and they cannot be perfectly adapted to all smartphones.

[0005] Therefore, it is urgent to develop a more stable, wider applicable range, and convenient non-contact operation technology for smartphones. The smartphone sliding interaction system based on facial motion control was born precisely in such a background, aiming to provide a more efficient and convenient smartphone operation method for special groups. Summary of the Invention

[0006] This application provides an interaction method and device, which can achieve interacting with the target page only by relying on facial movements, thus greatly improving the operation convenience.

[0007] In a first aspect, this application provides an interaction method, and the method includes:

[0008] Respond to the facial control instruction triggered by the user for the target page, and obtain the user's facial image information;

[0009] Obtain a first feature point corresponding to the nose, a second feature point corresponding to the eyebrows, and a third feature point corresponding to the eyes in the user facial image information according to the user facial image information;

[0010] Determine an input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows;

[0011] Determine a sliding interaction operation instruction according to the first feature point corresponding to the nose and the third feature point corresponding to the eyes;

[0012] Complete an interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction.

[0013] In a second aspect, the present application provides an interaction device, and the device includes:

[0014] A first unit, configured to obtain user facial image information in response to a facial control instruction triggered by a user for a target page;

[0015] A second unit, configured to obtain a first feature point corresponding to the nose, a second feature point corresponding to the eyebrows, and a third feature point corresponding to the eyes in the user facial image information according to the user facial image information;

[0016] A third unit, configured to determine an input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows;

[0017] A fourth unit, configured to determine a sliding interaction operation instruction according to the first feature point corresponding to the nose and the third feature point corresponding to the eyes;

[0018] A fifth unit, configured to complete an interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction.

[0019] In a third aspect, the present application provides a readable medium, including execution instructions, and when a processor of an electronic device executes the execution instructions, the electronic device executes the method described in any one of the first aspects.

[0020] In a fourth aspect, the present application provides an electronic device, including a processor and a memory storing execution instructions, and when the processor executes the execution instructions stored in the memory, the processor executes the method described in any one of the first aspects.

[0021] As can be seen from the above technical solution, the method provided in this application can respond to the facial control instruction triggered by the user for the target page and obtain the user's facial image information; according to the user's facial image information, obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes in the user's facial image information; determine the input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows; determine the sliding interaction operation instruction according to the first feature point corresponding to the nose and the third feature point corresponding to the eyes; and complete the interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction. It can be seen that for people with physical disabilities, since it is difficult for them to operate the screen with their fingers, and the method provided in this application can achieve interacting with the target page only relying on facial movements, the operation convenience is greatly improved. In terms of interaction accuracy, by obtaining the feature points of the nose, eyebrows, and eyes on the face to determine the interaction instruction, compared with the traditional method, it can more precisely reflect the user's intention; for example, raised eyebrows and a stable nose position may represent a specific input instruction, and a specific relative movement between the eyes and the nose can accurately determine the sliding direction, thus effectively reducing misoperations. In terms of the interaction method, the design of a combination of various facial movements gives users the freedom to choose different operation methods according to their own habits. For example, when browsing the web, users can choose to use eye movements in combination with nose features to achieve fast sliding, or use the movements of the eyebrows and nose for precise input, significantly improving flexibility. From the perspective of applicable scenarios, because it only relies on facial movements and image information, in special environments where both hands are occupied or the screen cannot be touched, such as driving and underwater operations, users can still complete the interaction operation, greatly expanding the application scope. In terms of the technical implementation cost, since it is mainly based on existing facial image recognition technology and common device cameras, there is no need to add complex and expensive hardware, and only the software algorithm needs to be optimized, so the cost is relatively low, which also makes this technology easier to promote and popularize, benefiting more users.

[0022] The further effects of the above non-conventional preferred methods will be described in conjunction with specific embodiments below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the following will briefly introduce the drawings required for the description of the embodiments or the existing technical solutions. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0024] Figure 1 It is a schematic flowchart of an interaction method provided by an embodiment of the present application;

[0025] Figure 2 Schematic diagram of facial feature points provided by an embodiment of the present application;

[0026] Figure 3 Schematic structural diagram of an interaction device provided by an embodiment of the present application;

[0027] Figure 4 Schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0028] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0029] The following will describe in detail various non-restrictive implementation manners of the present application in conjunction with the drawings.

[0030] The inventor has found through research that for some special groups, such as people with physical disabilities, due to their limited physical functions, they cannot interact with the touch screen as flexibly as normal people; there are also the elderly, whose hand flexibility decreases with age, and there are certain difficulties in operating the touch screen. To meet the needs of this part of the special group, a non-contact operation method has emerged. At present, the traditional non-contact alternative operation methods mainly rely on voice recognition technology or specific hardware devices. Although voice recognition technology has achieved non-contact operation to a certain extent, it has many limitations. In a noisy environment, the accuracy of voice recognition will drop significantly, resulting in the inability to accurately convey operation instructions and affecting the user experience; moreover, for some users with speech impairments, the voice recognition operation method is simply not applicable. The operation method relying on specific hardware devices often requires users to purchase and carry relevant devices additionally, which not only increases the usage cost, but also reduces the usage convenience. At the same time, the compatibility of such devices may also have problems and cannot be perfectly adapted to all smart phones. Therefore, it is extremely urgent to develop a more stable, wider applicable range and convenient non-contact operation technology for smart phones.

[0031] To solve the above-mentioned existing technical problems, the present application provides an interaction method, which can be applied to a terminal device. Refer to Figure 1 , which shows an interaction method in an embodiment of the present application. In this embodiment, the method may include the following steps:

[0032] S101: Respond to the facial control instruction triggered by the user for the target page, and obtain the user's facial image information.

[0033] The terminal device can first detect the face control instruction triggered by the user for the target page. For example, the terminal detects the display of the target page or the screen lighting up. If the terminal device detects it, it can respond to the face control instruction triggered by the user for the target page and obtain the user's face image information. For example, after obtaining the user's consent, the user's face image information can be obtained through the front camera or face recognition device of the terminal device.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, user face image information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for the user to choose to authorize or reject.

[0035] S102: According to the user's face image information, obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes in the user's face image information.

[0036] In this embodiment, the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes in the user's face image information can be obtained first according to the user's face image information.

[0037] In one implementation, the user's face image information can be preprocessed first to obtain the preprocessed user's face image information to improve the detection accuracy of facial features; wherein, the preprocessing includes denoising processing, color balance, and enhancement processing. Then, the preprocessed user's face image information can be input into a trained facial feature extraction model (such as Haar cascade classifier, deep learning algorithm, convolutional neural network CNN) to obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes in the user's face image information. It can be understood that this application can use computer vision technology to extract the feature points of the user's face, especially the dynamic information of the eyes and eyebrows areas, that is, the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes; by tracking the changes of these feature points, the system can analyze the user's facial movements in real time.

[0038] S103: Determine the input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows.

[0039] In this embodiment, an input interaction operation instruction can be determined based on the first feature points corresponding to the nose and the second feature points corresponding to the eyebrows. According to the change in the distance between the first feature points corresponding to the nose and the second feature points corresponding to the eyebrows, for example, whether the distance between the first feature points corresponding to the nose and the second feature points corresponding to the eyebrows becomes longer or shorter, an input interaction operation instruction is determined. Among them, the input interaction operation instruction can be understood as the screen input interaction operation of an actual terminal device. For example, by mapping the eyebrow movements of the face to the input commands of the touch screen, the input actions of the user's fingers are simulated. For example, corresponding operations (such as "confirm" or "cancel" operations) are generated according to the raising or lowering of the eyebrows, thereby triggering operations such as confirmation and cancellation on the mobile phone screen.

[0040] As an example, multiple second feature points correspond to the eyebrows. For example, the 72nd - 87th points shown are all second feature points. For each second feature point corresponding to the eyebrows, the change in the distance between the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows is determined. That is, for the 72nd - 87th points, the change in the distance (i.e., whether the distance becomes farther or nearer) between the 72nd - 87th points and the first feature point corresponding to the nose (the 56th point) is calculated respectively. Figure 2 Then, an input interaction operation instruction can be determined according to the change in the distance between each second feature point corresponding to the eyebrows and the first feature point corresponding to the nose. If the number of second feature points with an increasing distance change from the first feature point corresponding to the nose is greater than the number of second feature points with a decreasing distance change from the first feature point corresponding to the nose, the input interaction operation instruction is determined as the first type of interaction instruction. For example, if the distance change between more than half of the second feature points among the 72nd - 87th points and the first feature point corresponding to the nose is a decrease, it can be determined that the input interaction operation instruction is the first type of interaction instruction (such as a confirm operation). If the number of second feature points with an increasing distance change from the first feature point corresponding to the nose is less than the number of second feature points with a decreasing distance change from the first feature point corresponding to the nose, the input interaction operation instruction is determined as the second type of interaction instruction. For example, if the distance change between more than half of the second feature points among the 72nd - 87th points and the first feature point corresponding to the nose is an increase, it can be determined that the input interaction operation instruction is the second type of interaction instruction (such as a cancel or return operation).

[0041] S104: Determine a sliding interaction operation instruction according to the first feature points corresponding to the nose and the third feature points corresponding to the eyes.

[0042] S104: Determine a sliding interaction operation instruction according to the first feature points corresponding to the nose and the third feature points corresponding to the eyes.

[0043] In this embodiment, a sliding interaction operation instruction can be determined based on the first feature points corresponding to the nose and the third feature points corresponding to the eyes. Also, based on the change in the distance between the first feature points corresponding to the nose and the third feature points corresponding to the eyes, for example, whether the distance between the first feature points corresponding to the nose and the third feature points corresponding to the eyes becomes longer or shorter, a sliding interaction operation instruction is determined. Among them, the sliding interaction operation instruction can be understood as the actual screen input sliding operation of the terminal device. For example, by mapping the eye movements of the face to the input commands of the touch screen, the sliding actions of the user's fingers are simulated. For example, corresponding operations (such as "swipe up", "swipe down", "swipe left", "swipe right") are generated according to the up, down, left, and right directions of the eyes, thereby triggering operations such as scrolling the mobile phone screen or switching pages.

[0044] As an example, multiple third feature points correspond to the eyes. For example, points 1, 2, 3, 7, 11, and 15 shown are all third feature points. First, for each third feature point corresponding to the eyes, the change in the distance between the first feature points corresponding to the nose and the third feature points corresponding to the eyes can be determined. That is, for points 1, 2, 3, 7, 11, and 15, the change in the distance (i.e., whether the distance becomes farther or closer) between points 1, 2, 3, 7, 11, and 15 and the first feature point corresponding to the nose (point 56) is calculated respectively. Figure 2 Then, based on the change in the distance between each third feature point corresponding to the eyes and the first feature point corresponding to the nose, a sliding interaction operation instruction can be determined. If the number of second feature points whose change in the linear distance from the first feature point corresponding to the nose is greater than the preset distance and the moving direction is the target direction is greater than the preset number, then the sliding interaction operation instruction is determined to be sliding in the target direction. For example, if the change in the linear distance between more than half of the third feature points among points 1, 2, 3, 7, 11, and 15 and the first feature point corresponding to the nose is greater than the preset distance (for example, 0.1 cm), and the moving directions of the third feature points whose linear distance change value is greater than the preset distance are all to the left, then the sliding interaction operation instruction is determined to be sliding in the left direction. Another example, if the change in the linear distance between more than half of the third feature points among points 1, 2, 3, 7, 11, and 15 and the first feature point corresponding to the nose is greater than the preset distance (for example, 0.1 cm), and the moving directions of the third feature points whose linear distance change value is greater than the preset distance are all upward, then the sliding interaction operation instruction is determined to be sliding in the upward direction.

[0045] Next, based on the change in the distance between each third feature point corresponding to the eyes and the first feature point corresponding to the nose, a sliding interaction operation instruction can be determined. If the number of second feature points whose change in the linear distance from the first feature point corresponding to the nose is greater than the preset distance and the moving direction is the target direction is greater than the preset number, then the sliding interaction operation instruction is determined to be sliding in the target direction. For example, if the change in the linear distance between more than half of the third feature points among points 1, 2, 3, 7, 11, and 15 and the first feature point corresponding to the nose is greater than the preset distance (for example, 0.1 cm), and the moving directions of the third feature points whose linear distance change value is greater than the preset distance are all to the left, then the sliding interaction operation instruction is determined to be sliding in the left direction. Another example, if the change in the linear distance between more than half of the third feature points among points 1, 2, 3, 7, 11, and 15 and the first feature point corresponding to the nose is greater than the preset distance (for example, 0.1 cm), and the moving directions of the third feature points whose linear distance change value is greater than the preset distance are all upward, then the sliding interaction operation instruction is determined to be sliding in the upward direction.

[0046] In one implementation, the speed and sensitivity of the sliding operation corresponding to the sliding interaction operation instruction are adjusted according to the facial action intensity of the user, where the facial action intensity of the user includes the blinking speed of the eyes and / or the raising and lowering speed of the eyebrows.

[0047] It can be understood that the facial action intensity can represent the transition effect of the sliding interaction operation. Taking Douyin as an example, if the blinking speed of the eyes is slow, the facial action intensity is low, and there may be a transition effect of short video switching. On the contrary, if the blinking speed of the eyes is fast, the facial action intensity is high, and it will quickly enter the next video.

[0048] It can be understood that the facial action intensity can also represent the response effect of the input interaction operation instruction. Taking the button on the page as an example, if the rising speed of the eyebrows is fast, the facial action intensity is high, and there may be a click button effect. On the contrary, if the blinking speed of the eyebrows is slow, the facial action intensity is low, and there may be a long-press button effect.

[0049] That is to say, the method of the present application supports adjusting the speed and sensitivity of sliding and input according to the facial action intensity of the user (such as rapid blinking of the eyes, violent raising and lowering of the eyebrows, etc.). Specifically, the user can customize the operation method suitable for himself according to his own usage habits and physical conditions.

[0050] S105: Complete the interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction.

[0051] After determining the input interaction operation instruction and the sliding interaction operation instruction, the interaction operation on the target page can be completed according to the input interaction operation instruction and the sliding interaction operation instruction. For example, click the button on the page, or switch videos, etc.

[0052] In one implementation, real-time interaction feedback information can be displayed on the screen of the terminal device, for example, visually prompting the user whether the current operation is successful, or guiding the user on how to adjust the facial action to obtain a better operation effect.

[0053] As can be seen from the above technical solutions, the method provided by this application can respond to the facial control instruction triggered by the user for the target page, and obtain the user's facial image information; according to the user's facial image information, obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes in the user's facial image information; determine the input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows; determine the sliding interaction operation instruction according to the first feature point corresponding to the nose and the third feature point corresponding to the eyes; and complete the interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction. It can be seen that for people with physical disabilities, since it is difficult for them to operate the screen with their fingers, and the method provided by this application can achieve interacting with the target page only relying on facial movements, the operation convenience is greatly improved. In terms of interaction accuracy, by obtaining the feature points of the nose, eyebrows and eyes on the face to determine the interaction instruction, compared with the traditional method, it can more finely reflect the user's intention; for example, raised eyebrows and a stable nose position may represent a specific input instruction, and a specific relative movement between the eyes and the nose can accurately determine the sliding direction, thus effectively reducing misoperations. In terms of the interaction method, the design of a combination of various facial movements gives the user the freedom to choose different operation methods according to their own habits. For example, when browsing the web, the user can choose to use eye movements in combination with nose features to achieve fast sliding, or use the movements of the eyebrows and nose for precise input, significantly improving flexibility. From the perspective of applicable scenarios, because it only relies on facial movements and image information, in special environments where the hands are occupied or the screen cannot be touched, such as driving and underwater operations, the user can still complete the interaction operation, greatly expanding the application scope. In terms of the technical implementation cost, since it is mainly based on the existing facial image recognition technology and the common device camera, there is no need to add complex and expensive hardware, and only the software algorithm needs to be optimized, so the cost is low, which also makes this technology easier to promote and popularize, enabling more users to benefit.

[0054] That is to say, the interaction method provided by this application has the following advantages:

[0055] Convenience: This method does not require the operation method of finger touching the screen, greatly improving the use convenience of users with physical disabilities or mobility impairments;

[0056] Universality: It supports the combination of various facial movements, can be adjusted according to the needs and facial features of different users, and adapts to the needs of different groups.

[0057] Efficiency: The technology of using facial movements to control the mobile phone can complete common mobile phone sliding operations through simple eye and eyebrow movements, and the operation response is rapid and the interaction is smooth.

[0058] Easy implementation: It can be achieved by means of the front camera and image recognition technology of existing terminals such as mobile phones and tablets, without the support of additional complex hardware, and has high implementability.

[0059] So far, this embodiment has realized the processing process of the interaction method in combination with a specific application scenario. Of course, it should be considered that the above scenario is only an exemplary scenario and does not limit the method provided in this application. The method provided in this application can be extended and applied to the processing process of other interaction methods with the same principle.

[0060] As Figure 3 shown, it is a specific embodiment of the interaction device described in this application. The device in this embodiment is an entity device for executing the method described in the above embodiment. Its technical solution is essentially the same as that of the above embodiment, and the corresponding descriptions in the above embodiment also apply to this embodiment. In this embodiment, the device includes:

[0061] The first unit 301 is used to respond to the face control instruction triggered by the user for the target page and obtain the user's face image information;

[0062] The second unit 302 is used to obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrows, and the third feature point corresponding to the eyes in the user's face image information according to the user's face image information;

[0063] The third unit 303 is used to determine the input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows;

[0064] The fourth unit 304 is used to determine the sliding interaction operation instruction according to the first feature point corresponding to the nose and the third feature point corresponding to the eyes;

[0065] The fifth unit 305 is used to complete the interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction.

[0066] Optionally, the third unit 303 is used for:

[0067] Determine the input interaction operation instruction according to the distance change between the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows.

[0068] Optionally, there are multiple second feature points corresponding to the eyebrows; the third unit 303 is used for:

[0069] For each second feature point corresponding to the eyebrows, determine the distance change between the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows;

[0070] Determine an input interaction operation instruction according to the distance change between each second feature point corresponding to the eyebrows and the first feature point corresponding to the nose.

[0071] Optionally, the third unit 303 is configured to:

[0072] If the number of second feature points with an increasing distance change from the first feature point corresponding to the nose is greater than the number of second feature points with a decreasing distance change from the first feature point corresponding to the nose, determine that the input interaction operation instruction is a first type of interaction instruction;

[0073] If the number of second feature points with an increasing distance change from the first feature point corresponding to the nose is less than the number of second feature points with a decreasing distance change from the first feature point corresponding to the nose, determine that the input interaction operation instruction is a second type of interaction instruction.

[0074] Optionally, the fourth unit 304 is configured to:

[0075] Determine a sliding interaction operation instruction according to the distance change between the first feature point corresponding to the nose and the third feature point corresponding to the eye region.

[0076] Optionally, there are multiple third feature points corresponding to the eye region; the fourth unit 304 is configured to:

[0077] For each third feature point corresponding to the eye region, determine the distance change between the first feature point corresponding to the nose and the third feature point corresponding to the eye region;

[0078] Determine a sliding interaction operation instruction according to the distance change between each third feature point corresponding to the eye region and the first feature point corresponding to the nose.

[0079] Optionally, the fourth unit 304 is configured to:

[0080] If the number of second feature points with a linear distance change value greater than a preset distance and a moving direction of the target direction from the first feature point corresponding to the nose is greater than a preset number, determine that the sliding interaction operation instruction is to slide in the target direction.

[0081] Optionally, the speed and sensitivity of the sliding operation corresponding to the sliding interaction operation instruction are adjusted according to the facial movement intensity of the user, where the facial movement intensity of the user includes the blinking speed of the eyes and / or the raising and lowering speed of the eyebrows.

[0082] Optionally, the second unit 302 is configured to:

[0083] Preprocess the user facial image information to obtain the preprocessed user facial image information; wherein, the preprocessing includes denoising, color balancing, and enhancement processing;

[0084] Input the preprocessed user facial image information into a trained facial feature extraction model to obtain a first feature point corresponding to the nose, a second feature point corresponding to the eyebrows, and a third feature point corresponding to the eyes in the user facial image information.

[0085] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0086] The processor, network interface, and memory can be interconnected through an internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 4 only a two-way arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0087] The memory is used to store executable instructions. Specifically, the executable instructions are computer programs that can be executed. The memory can include a memory and a non-volatile memory, and provide executable instructions and data to the processor.

[0088] In a possible implementation manner, the processor reads the corresponding executable instructions from the non-volatile memory into the memory and then runs them, or can also obtain the corresponding executable instructions from other devices to form an interaction device at the logical level. The processor executes the executable instructions stored in the memory to implement the interaction method provided in any embodiment of the present application through the executed executable instructions.

[0089] The above is as described in the present application Figure 1The method executed by the interaction device provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in software form. The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0090] The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0091] The embodiments of the present application also propose a readable medium. When the execution instructions stored in the readable storage medium are executed by the processor of the electronic device, the electronic device can execute the interaction method provided in any embodiment of the present application, and is specifically used to execute the method described for the above data query.

[0092] The electronic device described in each of the foregoing embodiments may be a computer.

[0093] Those skilled in the art should understand that the embodiments of the present application may be provided as a method or a computer program product. Therefore, the present application may adopt a completely hardware embodiment, a completely software embodiment, or a form combining software and hardware.

[0094] The embodiments in the present application are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.

[0095] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the said element.

[0096] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. An interaction method, characterized in that The method includes: In response to a face control instruction triggered by a user for a target page, obtaining user face image information; Based on the user face image information, obtaining a first feature point corresponding to the nose, a second feature point corresponding to the eyebrows, and a third feature point corresponding to the eyes in the user face image information; Based on the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows, determining an input interaction operation instruction; Based on the first feature point corresponding to the nose and the third feature point corresponding to the eyes, determining a sliding interaction operation instruction; Based on the input interaction operation instruction and the sliding interaction operation instruction, completing an interaction operation on the target page.

2. The method according to claim 1, wherein The determining the input interaction operation instruction based on the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows includes: Based on the distance change situation between the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows, determining the input interaction operation instruction.

3. The method according to claim 2, wherein Multiple second feature points correspond to the eyebrows; the determining the input interaction operation instruction based on the distance change situation between the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows includes: For each second feature point corresponding to the eyebrows, determining the distance change situation between the first feature point corresponding to the nose and the second feature point corresponding to the eyebrows; Based on the distance change situations between the respective second feature points corresponding to the eyebrows and the first feature point corresponding to the nose, determining the input interaction operation instruction.

4. The method according to claim 3, wherein, The determining the input interaction operation instruction based on the distance change situations between the respective second feature points corresponding to the eyebrows and the first feature point corresponding to the nose includes: If the number of second feature points with a distance increase situation between the first feature point corresponding to the nose is greater than the number of second feature points with a distance decrease situation between the first feature point corresponding to the nose, determining the input interaction operation instruction as a first type of interaction instruction; If the number of second feature points with a distance increase situation between the first feature point corresponding to the nose is less than the number of second feature points with a distance decrease situation between the first feature point corresponding to the nose, determining the input interaction operation instruction as a second type of interaction instruction.

5. The method according to claim 1, characterized in that, The determining the sliding interaction operation instruction based on the first feature point corresponding to the nose and the third feature point corresponding to the eyes includes: Based on the distance change situation between the first feature point corresponding to the nose and the third feature point corresponding to the eyes, determining the sliding interaction operation instruction.

6. The method according to claim 5, characterized in that, Multiple third feature points correspond to the eyes; the determining the sliding interaction operation instruction based on the distance change situation between the first feature point corresponding to the nose and the third feature point corresponding to the eyes includes: For each third feature point corresponding to the eyes, determining the distance change situation between the first feature point corresponding to the nose and the third feature point corresponding to the eyes; Based on the distance change situations between the respective third feature points corresponding to the eyes and the first feature point corresponding to the nose, determining the sliding interaction operation instruction.

7. The method according to claim 6, wherein Determining a sliding interaction operation instruction according to the distance change conditions between each third feature point corresponding to the eye part and the first feature point corresponding to the nose includes: If the number of second feature points with a linear distance change value greater than a preset distance and a moving direction being the target direction among the distance change conditions between the first feature points corresponding to the nose is greater than a preset number, determine that the sliding interaction operation instruction is to slide in the target direction.

8. The method according to claim 1, characterized in that The speed and sensitivity of the sliding operation corresponding to the sliding interaction operation instruction are adjusted according to the facial action intensity of the user, where the facial action intensity of the user includes the blinking speed of the eyes and / or the lifting and lowering speed of the eyebrows.

9. The method according to claim 1, wherein Obtaining the first feature point corresponding to the nose, the second feature point corresponding to the eyebrow, and the third feature point corresponding to the eye part in the user facial image information according to the user facial image information includes: Performing preprocessing on the user facial image information to obtain preprocessed user facial image information; where the preprocessing includes denoising processing, color balance, and enhancement processing; Inputting the preprocessed user facial image information into a trained facial feature extraction model to obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrow, and the third feature point corresponding to the eye part in the user facial image information.

10. An interactive device, characterized in that, The device includes: A first unit, configured to obtain user facial image information in response to a facial control instruction triggered by the user for a target page; A second unit, configured to obtain the first feature point corresponding to the nose, the second feature point corresponding to the eyebrow, and the third feature point corresponding to the eye part in the user facial image information according to the user facial image information; A third unit, configured to determine an input interaction operation instruction according to the first feature point corresponding to the nose and the second feature point corresponding to the eyebrow; A fourth unit, configured to determine a sliding interaction operation instruction according to the first feature point corresponding to the nose and the third feature point corresponding to the eye part; A fifth unit, configured to complete an interaction operation on the target page according to the input interaction operation instruction and the sliding interaction operation instruction.

Images