Equipment control method, model training method and device

Abstract

The invention provides an equipment control method and a model training method and device, and the method comprises the steps: obtaining one or more frames of face images, carrying out the eye behavior recognition based on any frame of face image, so as to obtain the eye behavior information of each face image, and determining a target control instruction based on the eye behavior information of one or more frames of face images, and carrying out equipment control based on the target control instruction. Therefore, according to the method, the device can be controlled through the eye behavior information, so that the requirement of a user for personalized control of the device can be met, the richness and convenience of device control are improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction technology, and in particular to a device control method, a model training method and a device. Background Art

[0002] At present, there are many types of device control methods, including voice control, mobile phone APP control, remote control, etc. Among them, voice control is the most popular control method for devices, which can be achieved through smart speakers or smart voice assistants; mobile phone APP control and remote control are both common device control methods. However, these control methods may not meet users' needs for personalized control. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of the present disclosure is to propose a device control method to control the device through eye behavior information, so as to meet the user's demand for personalized control of the device, improve the richness and convenience of device control, and improve the user experience.

[0005] The second objective of the present disclosure is to propose a model training method.

[0006] The third objective of the present disclosure is to provide a device control apparatus.

[0007] The fourth objective of the present disclosure is to provide a model training device.

[0008] A fifth objective of the present disclosure is to provide an electronic device.

[0009] A sixth objective of the present disclosure is to provide a computer-readable storage medium.

[0010] A seventh objective of the present disclosure is to provide a computer program product.

[0011] To achieve the above-mentioned purpose, the first embodiment of the present disclosure provides a device control method, including:

[0012] Acquire one or more frames of facial images;

[0013] Performing eye behavior recognition based on any frame of the facial image to obtain eye behavior information of each of the facial images;

[0014] Determining a target control instruction based on the eye behavior information of the one or more frames of facial images;

[0015] The device is controlled based on the target control instruction.

[0016] In one embodiment of the present disclosure, the eye behavior information includes gaze direction; and determining the target control instruction based on the eye behavior information of the one or more frames of facial images includes:

[0017] Combining the gaze directions of the multiple frames of facial images according to the acquisition order of the multiple frames of facial images to obtain a gaze direction sequence;

[0018] matching the gaze direction sequence with a preset standard sequence;

[0019] The target control instruction is determined according to the control instruction corresponding to the matched standard sequence.

[0020] In one embodiment of the present disclosure, matching the gaze direction sequence with a preset standard sequence includes:

[0021] Merging adjacent and identical gaze directions in the gaze direction sequence to obtain a merged sequence;

[0022] If there is at least one continuous subsequence in the combined sequence that is the same as a preset standard sequence, it is determined that the gaze direction sequence matches the standard sequence.

[0023] In one embodiment of the present disclosure, the controlling the device based on the control instruction includes:

[0024] Based on the control instruction, controlling the application program running on the wearable device;

[0025] The application is used to perform at least one of the following:

[0026] Control the operation of smart home devices;

[0027] Play or pause multimedia resource display;

[0028] The camera of the wearable device is controlled to capture the object being watched.

[0029] In one embodiment of the present disclosure, the performing eye behavior recognition based on any frame of the facial image to obtain the eye behavior information of each facial image includes:

[0030] Extracting an eye partial image for any frame of the facial image;

[0031] Extracting features from the eye partial image and the corresponding facial image respectively;

[0032] Classification is performed based on the features of the eye local image and the features of the corresponding facial image to obtain the eye behavior information.

[0033] In one embodiment of the present disclosure, extracting the eye partial image for any frame of the facial image includes:

[0034] Performing eye contour recognition on any frame of the facial image;

[0035] By using a minimum circumscribed frame surrounding the outer contour of the eye, a portion within the circumscribed frame is extracted as the extracted eye partial image.

[0036] In one embodiment of the present disclosure, the eye partial image includes a left eye partial image and a right eye partial image, and the feature extraction of the eye partial image and the corresponding facial image respectively includes:

[0037] Inputting the left eye partial image and the right eye partial image into corresponding first feature extraction networks for feature extraction, respectively, wherein the features extracted by the first feature extraction network include at least one of pupil position features and eye shape features;

[0038] The facial image is input into a second feature extraction network for feature extraction to obtain features of the facial image, wherein a convolution kernel of the second feature extraction network is larger than a convolution kernel of the first feature extraction network.

[0039] In one embodiment of the present disclosure, the classification based on the features of the eye partial image and the features of the facial image to which it belongs to obtain the eye behavior information includes:

[0040] Input the features extracted from the left eye partial image and the features extracted from the right eye partial image into a first fully connected layer for vector space mapping to obtain an output of the first fully connected layer;

[0041] Inputting the features of the facial image into a second fully connected layer for vector space mapping to obtain an output of the second fully connected layer;

[0042] Inputting the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer;

[0043] An activation function layer is used to activate the output of the third fully connected layer to classify the eye behavior information.

[0044] In one embodiment of the present disclosure, the first feature extraction network and the second feature extraction network each include at least one convolution block connected in sequence, and any of the convolution blocks includes a convolution layer that uses a convolution kernel to perform a convolution operation, and includes a pooling layer;

[0045] The pooling layer is used to use the most significant feature of the output of the convolution layer in the convolution block as the input of the next convolution block.

[0046] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes a model training method, including:

[0047] Get training samples;

[0048] The training samples are used to train the recognition model, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each of the facial images, so as to determine target control instructions for device control based on the eye behavior information of the one or more frames of facial images.

[0049] In one embodiment of the present disclosure, the training samples include sample images of facial and eye parts, and annotated eye behavior information;

[0050] The adopting the training sample to train the recognition model includes:

[0051] Inputting the sample image of the eye part and the sample image of the face into the recognition model for feature extraction, and performing classification based on the extracted features to obtain predicted behavior information output by the recognition model;

[0052] Based on the difference between the predicted behavior information and the labeled eye behavior information, a model parameter of the recognition model is adjusted.

[0053] In one embodiment of the present disclosure, the sample images of the eye parts include a left eye sample image and a right eye sample image; the step of inputting the sample images of the eye parts and the sample images of the face into the recognition model for feature extraction, and performing classification based on the extracted features to obtain the predicted behavior information output by the recognition model includes:

[0054] Inputting the left eye sample image and the right eye sample image into a first feature extraction network in the recognition model for feature extraction, wherein the features extracted by the first feature extraction network include at least one of a pupil position feature and an eye shape feature;

[0055] Inputting the sample image of the face into a second feature extraction network in the recognition model for feature extraction, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network;

[0056] Input the features extracted from the left eye sample image and the features extracted from the right eye sample image into a first fully connected layer for vector space mapping, and input the features of the facial sample image into a second fully connected layer for vector space mapping;

[0057] Inputting the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer;

[0058] An activation function layer is used to activate the output of the third fully connected layer to classify and obtain the predicted behavior information.

[0059] To achieve the above-mentioned purpose, a third aspect of the present disclosure provides a device control apparatus, including:

[0060] An acquisition module, used for acquiring one or more frames of facial images;

[0061] A recognition module, used for performing eye behavior recognition based on any frame of the facial image to obtain eye behavior information of each of the facial images;

[0062] A determination module, configured to determine a target control instruction based on the eye behavior information of the one or more frames of facial images;

[0063] A control module is used to control the device based on the target control instruction.

[0064] In one embodiment of the present disclosure, the eye behavior information includes a gaze direction; and the determination module is used to determine the target control instruction based on the eye behavior information of the one or more frames of facial images, including:

[0065] Combining the gaze directions of the multiple frames of facial images according to the acquisition order of the multiple frames of facial images to obtain a gaze direction sequence;

[0066] matching the gaze direction sequence with a preset standard sequence;

[0067] The target control instruction is determined according to the control instruction corresponding to the matched standard sequence.

[0068] In one embodiment of the present disclosure, when the determination module is used to match the gaze direction sequence with a preset standard sequence, it includes:

[0069] Merging adjacent and identical gaze directions in the gaze direction sequence to obtain a merged sequence;

[0070] If there is at least one continuous subsequence in the combined sequence that is the same as a preset standard sequence, it is determined that the gaze direction sequence matches the standard sequence.

[0071] In one embodiment of the present disclosure, when the control module is used to control the device based on the control instruction, it includes:

[0072] Based on the control instruction, controlling the application program running on the wearable device;

[0073] The application is used to perform at least one of the following:

[0074] Control the operation of smart home devices;

[0075] Play or pause multimedia resource display;

[0076] The camera of the wearable device is controlled to capture the object being watched.

[0077] In one embodiment of the present disclosure, the recognition module is used to perform eye behavior recognition based on any frame of the facial image to obtain eye behavior information of each facial image, including:

[0078] Extracting an eye partial image for any frame of the facial image;

[0079] Extracting features from the eye partial image and the corresponding facial image respectively;

[0080] Classification is performed based on the features of the eye local image and the features of the corresponding facial image to obtain the eye behavior information.

[0081] In one embodiment of the present disclosure, when the recognition module is used to extract the eye partial image for any frame of the facial image, it includes:

[0082] Performing eye contour recognition on any frame of the facial image;

[0083] By using a minimum circumscribed frame surrounding the outer contour of the eye, a portion within the circumscribed frame is extracted as the extracted eye partial image.

[0084] In one embodiment of the present disclosure, the eye partial image includes a left eye partial image and a right eye partial image, and the recognition module is used to extract features from the eye partial image and the corresponding facial image respectively, including:

[0085] Inputting the left eye partial image and the right eye partial image into corresponding first feature extraction networks for feature extraction, respectively, wherein the features extracted by the first feature extraction network include at least one of pupil position features and eye shape features;

[0086] The facial image is input into a second feature extraction network for feature extraction to obtain features of the facial image, wherein a convolution kernel of the second feature extraction network is larger than a convolution kernel of the first feature extraction network.

[0087] In one embodiment of the present disclosure, the recognition module is used to classify based on the features of the eye partial image and the features of the facial image to which it belongs to obtain the eye behavior information, including:

[0088] Input the features extracted from the left eye partial image and the features extracted from the right eye partial image into a first fully connected layer for vector space mapping to obtain an output of the first fully connected layer;

[0089] Inputting the features of the facial image into a second fully connected layer for vector space mapping to obtain an output of the second fully connected layer;

[0090] Inputting the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer;

[0091] An activation function layer is used to activate the output of the third fully connected layer to classify the eye behavior information.

[0092] In one embodiment of the present disclosure, the first feature extraction network and the second feature extraction network each include at least one convolution block connected in sequence, and any of the convolution blocks includes a convolution layer that uses a convolution kernel to perform a convolution operation, and includes a pooling layer;

[0093] The pooling layer is used to use the most significant feature of the output of the convolution layer in the convolution block as the input of the next convolution block.

[0094] To achieve the above-mentioned purpose, the fourth aspect of the present disclosure provides a model training device, including:

[0095] An acquisition module is used to acquire training samples;

[0096] A training module is used to train a recognition model using the training samples, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each of the facial images, so as to determine target control instructions for device control based on the eye behavior information of the one or more frames of facial images.

[0097] In one embodiment of the present disclosure, the training samples include sample images of facial and eye parts, and annotated eye behavior information;

[0098] The training module is used to train the recognition model using the training samples, including:

[0099] Inputting the sample image of the eye part and the sample image of the face into the recognition model for feature extraction, and performing classification based on the extracted features to obtain predicted behavior information output by the recognition model;

[0100] Based on the difference between the predicted behavior information and the labeled eye behavior information, a model parameter of the recognition model is adjusted.

[0101] In one embodiment of the present disclosure, the sample images of the eye parts include left eye sample images and right eye sample images; the training module is used to input the sample images of the eye parts and the sample images of the face into the recognition model for feature extraction, and classify based on the extracted features to obtain the predicted behavior information output by the recognition model, including:

[0102] Inputting the left eye sample image and the right eye sample image into a first feature extraction network in the recognition model for feature extraction, wherein the features extracted by the first feature extraction network include at least one of a pupil position feature and an eye shape feature;

[0103] Inputting the sample image of the face into a second feature extraction network in the recognition model for feature extraction, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network;

[0104] Input the features extracted from the left eye sample image and the features extracted from the right eye sample image into a first fully connected layer for vector space mapping, and input the features of the facial sample image into a second fully connected layer for vector space mapping;

[0105] Inputting the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer;

[0106] An activation function layer is used to activate the output of the third fully connected layer to classify and obtain the predicted behavior information.

[0107] To achieve the above-mentioned purpose, a fifth aspect of the present disclosure provides an electronic device, including:

[0108] at least one processor; and

[0109] a memory communicatively connected to the at least one processor; wherein,

[0110] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method proposed in the first aspect embodiment, or the method proposed in the second aspect embodiment.

[0111] To achieve the above objectives, the sixth aspect of the present disclosure proposes a computer-readable storage medium, characterized in that the computer instructions are used to enable the computer to execute the method proposed in the first aspect of the embodiment, or the method proposed in the second aspect of the embodiment.

[0112] To achieve the above objectives, the seventh aspect of the present disclosure proposes a computer program product. When the instruction processor in the computer program product executes, it executes the method proposed in the first aspect, or the method proposed in the second aspect.

[0113] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0114] In the embodiment of the present disclosure, one or more frames of facial images are first acquired, and eye behavior recognition is performed based on any frame of facial images to obtain eye behavior information of each facial image, and then a target control instruction is determined based on the eye behavior information of one or more frames of facial images, and the device is controlled based on the target control instruction. Thus, the method can control the device through the eye behavior information, which can meet the user's demand for personalized control of the device, improve the richness and convenience of device control, and improve the user experience.

[0115] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0117] Figure 1 is a flow chart of a device control method according to an embodiment of the present disclosure;

[0118] Figure 2 is a schematic diagram of a control scenario of a smart home device according to an embodiment of the present disclosure;

[0119] Figure 3 According to an embodiment of the present disclosure Figure 1 Flowchart of the execution process of step S102 in the flowchart;

[0120] Figure 4 is a structural diagram of a recognition model according to an embodiment of the present disclosure;

[0121] Figure 5 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0122] Figure 6is a schematic diagram of obtaining prediction results according to an embodiment of the present disclosure;

[0123] Figure 7 is a block diagram of a device control apparatus according to an embodiment of the present disclosure;

[0124] Figure 8 It is a block diagram of a model training device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0125] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0126] The device control method, training method and apparatus of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0127] Before introducing the device control method of the embodiment of the present disclosure, the eye tracker technology is first introduced.

[0128] Eye tracker technology is a technology used to measure eye movements. It can study visual processing and cognitive processes by recording the movement trajectory and reaction time of the eye in visual tasks. The main principle of eye tracker technology is to detect eye movements through technologies such as infrared or electrodes, and then transmit the data to a computer for analysis and processing. The eye tracker technology solution includes two aspects: hardware and software. In terms of hardware, the eye tracker is usually composed of an infrared camera, a light source, glasses or a head-mounted device to capture eye movement data; in terms of software, the eye tracker is usually equipped with data acquisition, analysis and visualization tools to process and present eye movement data.

[0129] Currently, eye tracking is performed using devices such as eye trackers, and eye intention recognition and prediction is performed using technologies such as deep learning. However, eye trackers have the following defects:

[0130] Limited accuracy: The accuracy of the eye tracker is affected by many factors, such as eyeball size, visual acuity, pupil size, head movement, etc., all of which may lead to measurement errors;

[0131] Limited scope of application: Eye trackers are usually only suitable for static or semi-static environments, such as laboratories and offices. In dynamic environments, such as outdoors and sports venues, the measurement accuracy of eye trackers will be greatly affected by factors such as head movement and environmental interference;

[0132] High equipment cost: The equipment cost of eye trackers is relatively high and requires professional equipment and technical support;

[0133] Complex data processing: The data collected by the eye tracker needs to be processed and analyzed in a complex manner, which requires professional software and technical support;

[0134] Poor subject experience: When using an eye tracker for experiments, subjects need to wear the device and keep their heads relatively fixed, which may affect the subjects' comfort and experience;

[0135] In poor lighting conditions, the recognition rate is limited: Pupil recognition here usually depends on the obvious color difference between the pupil and the white of the eye, thereby directly locking the position of the pupil. This method is prone to misidentification in dim light conditions.

[0136] To this end, the present disclosure proposes a device control method based on smart glasses, which can accurately obtain eye behavior information and control the device through eye behavior information, thereby meeting the user's demand for personalized control of the device, improving the richness and convenience of device control, and improving the user experience.

[0137] Figure 1 is a flow chart of a device control method according to an embodiment of the present disclosure.

[0138] like Figure 1 The device control method of the embodiment of the present disclosure includes:

[0139] S101, obtaining one or more frames of facial images.

[0140] For example, one or more frames of facial images may be captured by a camera of a wearable device such as smart glasses.

[0141] S102, performing eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image.

[0142] For example, a behavior recognition algorithm (such as a trained recognition model) can be used to perform eye behavior recognition on any frame of facial image to obtain eye behavior information of each facial image at least including the gaze direction; wherein the eye behavior information can include left eye behavior information and / or right eye behavior information, which is not specifically limited in the present invention. For ease of description, the following description is based on an example in which the eye behavior information includes left eye behavior information and right eye behavior information.

[0143] S103, determining a target control instruction based on the eye behavior information of one or more frames of facial images.

[0144] When the eye behavior information includes gaze direction, the gaze directions of multiple frames of facial images are combined according to the acquisition order of the multiple frames of facial images to obtain a gaze direction sequence, and then the gaze direction sequence is matched with a preset standard sequence, and the target control instruction is determined according to the control instruction corresponding to the matched standard sequence.

[0145] In the process of matching the gaze direction sequence with the preset standard sequence, adjacent and identical gaze directions in the gaze direction sequence are first merged to obtain a merged sequence, and then all continuous subsequences in the merged sequence are compared with the preset standard sequence. If there is at least one continuous subsequence in the merged sequence that is the same as the preset standard sequence, it is determined that the gaze direction sequence matches the standard sequence.

[0146] It should be noted that before determining the target control instruction based on the control instruction corresponding to the matching standard sequence, the correspondence between the standard sequence and the control instruction can be set in advance, so that after obtaining the matching standard sequence, the target control instruction corresponding to the matching standard sequence can be determined according to the preset correspondence.

[0147] S104, performing device control based on the target control instruction.

[0148] In step S104, based on the target control instruction, the application program running on the wearable device is controlled;

[0149] The application is used to perform at least one of the following:

[0150] Control the operation of smart home devices;

[0151] Play or pause multimedia resource display;

[0152] Control the wearable device's camera to capture the object it is looking at.

[0153] For example, in the smart home device control scenario:

[0154] like Figure 2 As shown, after the camera collects the video stream, the eye behavior is identified to obtain the eye behavior information, and the target control instruction is determined based on the eye behavior information, and the target control instruction is sent to the control software. The control software controls (such as turning on or off) the smart home devices (such as smart lights, smart fans, smart air conditioners, etc.) according to the target control instruction, thereby meeting the user's demand for personalized control, improving the richness and convenience of device control, and improving the convenience and comfort of family life.

[0155] In the multimedia resource display scenario:

[0156] After the target control command is determined through the smart glasses, the smart home devices (such as smart TVs, smart projectors) are controlled (such as playing or pausing) according to the target control command, thereby meeting the user's demand for personalized control, improving the richness and convenience of device control, and improving the user experience.

[0157] In the retail scenario:

[0158] After the target control instructions are determined through the smart glasses, the camera of the wearable device is controlled to collect the object of gaze, so that the user's eye intention in the store can be tracked through the smart glasses, and the user's attention to and purchase intention of different products can be understood, thereby optimizing product display and promotion strategies and increasing sales.

[0159] In this scenario, a user enters a store and puts on smart glasses. Smart glasses can understand the user's attention and purchase intention for different products by tracking the user's eye behavior in real time, mainly eyeball intention. By analyzing this data, the store can optimize store display and promotion strategies to increase sales. For example, if smart glasses find that the user pays a lot of attention to a certain product, but the display location of the product is not prominent enough, the store can put it in a more prominent position to increase sales.

[0160] Therefore, the device control method disclosed in the present invention has the following advantages:

[0161] Improve user experience: By tracking user eye intentions, smart glasses can better understand user needs and interests, thereby providing more personalized services and recommendations and improving user experience;

[0162] Improve work efficiency: Smart glasses can identify users’ work priorities and focus by tracking their eye intentions, thereby providing more accurate information and guidance to help users complete their work more efficiently.

[0163] Reduce safety risks: Smart glasses can identify the user's concentration level by tracking the user's eye intention, thereby reminding the user to pay attention to safety and avoid accidents;

[0164] Improve marketing effectiveness: Smart glasses can understand users’ shopping preferences and interests by tracking their eye intentions, thereby providing more accurate advertising and recommendations and improving marketing effectiveness.

[0165] In order to enable those skilled in the art to more clearly understand the process of step S102, Figure 3 is a flowchart of the execution process of step S102 according to an embodiment of the present disclosure, such as Figure 3 As shown, the execution process of step S102 includes:

[0166] S301, extracting an eye partial image from any frame of facial image.

[0167] In this step, for any frame of facial image, eye contour recognition is first performed. During the eye contour recognition process, the minimum bounding box including the eye contour is used as the eye contour. After the minimum bounding box of the eye contour is recognized, the part within the minimum bounding box is extracted as the extracted eye partial image.

[0168] S302, extracting features from the eye local image and the corresponding facial image respectively.

[0169] The eye partial image may include a left eye partial image and a right eye partial image.

[0170] In this embodiment, if Figure 4 As shown, in the process of extracting features from the partial image of the eye, the partial image of the left eye and the partial image of the right eye are respectively input into the corresponding first feature extraction network for feature extraction to obtain the features of the partial image of the eye, wherein the features of the partial image of the eye include the features of the partial image of the left eye and the features of the partial image of the right eye; wherein the features extracted by the first feature extraction network include at least one of pupil position features and eye shape features. In other words, the pupil position and / or eye shape features of the partial image of the left eye, and the pupil position and / or eye shape features of the partial image of the right eye can be obtained. It should be noted that, since the matrix sizes corresponding to the input left eye layout image and the partial image of the right eye are different, the convolution kernel sizes used can also be different.

[0171] In the process of extracting features from the facial image, the facial image is input into the second feature extraction network for feature extraction to obtain features of the facial image, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network.

[0172] The first feature extraction network and the second feature extraction network each include at least one convolution block connected in sequence, any convolution block includes a convolution layer that uses a convolution kernel to perform a convolution operation, and includes a pooling layer; wherein the pooling layer is used to use the most significant feature output by the convolution layer in the convolution block as the input of the next convolution block. For example, the first feature extraction network includes: a first convolution block CONV-E1 (convolution layer E1 and pooling layer E1), a second convolution block CONV-E2 (convolution layer E2 and pooling layer E2), and a third convolution block CONV-E3 (convolution layer E3 and pooling layer E3); the second feature extraction network includes: a first convolution block CONV-F1 (convolution layer F1 and pooling layer F1), a second convolution block CONV-F2 (convolution layer F2 and pooling layer F2), and a third convolution block CONV-F3 (convolution layer F3 and pooling layer F3).

[0173] S303: Classify the eye local image based on the features of the corresponding facial image to obtain eye behavior information.

[0174] In this step, continue to refer to Figure 4 First, the features extracted from the local image of the left eye and the features extracted from the local image of the right eye are input into the first fully connected layer FC-E1 for vector space mapping to obtain the output of the first fully connected layer FC-E1. At the same time, the features of the facial image are input into the second fully connected layer FC-F1 for vector space mapping to obtain the output of the second fully connected layer FC-F1. Then, the output of the first fully connected layer FC-E1 and the output of the second fully connected layer FC-F1 are input into the third fully connected layer FC for vector space mapping to obtain the output of the third fully connected layer FC. Finally, the activation function layer Softmax is used to activate the output of the third fully connected layer to classify the eye behavior information. For example, -1 can be output to the left, 0 can be output in the middle, and 1 can be output to the right.

[0175] In summary, one or more frames of facial images are first acquired, and eye behavior recognition is performed based on any frame of facial images to obtain eye behavior information of each facial image, and then the target control instruction is determined based on the eye behavior information of one or more frames of facial images, and the device is controlled based on the target control instruction. Thus, the method can accurately obtain eye behavior information and control the device through the eye behavior information, which can meet the user's demand for personalized control of the device, improve the richness and convenience of device control, and improve the user experience.

[0176] Figure 5 It is a flowchart of a model training method according to an embodiment of the present disclosure.

[0177] like Figure 5 As shown, the model training method of the embodiment of the present disclosure includes:

[0178] S501, obtaining training samples.

[0179] In this step, the training samples can use face data in the face recognition database. After obtaining the face data, first collect sample images of the face and eye parts, make them into RGB data, and then use manual annotation to combine some open source data sets to generate a set number of samples of eyeballs turning left and right.

[0180] For example, Figure 6 As shown, the training samples include sample images of facial and eye parts, as well as annotated eye behavior information.

[0181] S502, using training samples to train the recognition model, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image, so as to determine the target control instruction for device control based on the eye behavior information of one or more frames of facial images.

[0182] In this step, the sample images of the eye part and the sample images of the face are first input into the recognition model for feature extraction, and classification is performed based on the extracted features to obtain the predicted behavior information output by the recognition model. Figure 6 When the sample images of the local part of the eye include the sample images of the left eye and the sample images of the right eye, the sample images of the left eye and the sample images of the right eye are input into the first feature extraction network in the recognition model for feature extraction, and the features extracted by the first feature extraction network include at least one of the pupil position feature and the eye shape feature. At the same time, the sample images of the face are input into the second feature extraction network in the recognition model for feature extraction, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network. Then, the features extracted from the sample images of the left eye and the features extracted from the sample images of the right eye are input into the first fully connected layer for vector space mapping, and the features of the sample images of the face are input into the second fully connected layer for vector space mapping, and the outputs of the first fully connected layer and the outputs of the second fully connected layer are input into the third fully connected layer for vector space mapping to obtain the outputs of the third fully connected layer. Finally, the outputs of the third fully connected layer are activated by using the activation function layer to classify and obtain the predicted behavior information, i.e., the predicted result.

[0183] Then, based on the difference between the predicted behavior information and the labeled eye behavior information, that is, the loss value, the model parameters of the recognition model are adjusted so that the difference between the predicted behavior information and the labeled eye behavior information meets the set requirements, thereby improving the recognition accuracy of the recognition model and obtaining eye behavior information more accurately.

[0184] The model training method of the embodiment of the present disclosure obtains training samples, uses the training samples to train the recognition model, and uses the trained recognition model to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image, and determines the target control instruction for device control based on the eye behavior information of one or more frames of facial image to achieve control of the device. Thus, the method can train the recognition model and obtain eye behavior information using the trained recognition model, so as to control the device according to the eye behavior information, which can meet the user's demand for personalized control of the device, improve the richness and convenience of device control, and improve user experience.

[0185] In order to implement the above embodiment, the present disclosure also proposes a device control apparatus 700 .

[0186] Figure 7 is a block diagram of a device control apparatus 700 according to an embodiment of the present disclosure.

[0187] like Figure 7 As shown, the device control apparatus 700 of the embodiment of the present disclosure includes:

[0188] An acquisition module 710 is used to acquire one or more frames of facial images;

[0189] The recognition module 720 is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image;

[0190] A determination module 730, configured to determine a target control instruction based on eye behavior information of one or more frames of facial images;

[0191] The control module 740 is used to control the device based on the target control instruction.

[0192] In one embodiment of the present disclosure, the eye behavior information includes gaze direction; the determination module 730 is used to determine the target control instruction based on the eye behavior information of one or more frames of facial images, including:

[0193] Combining the gaze directions of the multiple facial image frames according to the acquisition order of the multiple facial image frames to obtain a gaze direction sequence;

[0194] Matching the gaze direction sequence with a preset standard sequence;

[0195] The target control instruction is determined according to the control instruction corresponding to the matched standard sequence.

[0196] In one embodiment of the present disclosure, when the determination module 730 is used to match the gaze direction sequence with a preset standard sequence, it includes:

[0197] Merge adjacent and identical gaze directions in the gaze direction sequence to obtain a merged sequence;

[0198] If there is at least one continuous subsequence in the merged sequence that is the same as the preset standard sequence, it is determined that the gaze direction sequence matches the standard sequence.

[0199] In one embodiment of the present disclosure, the control module 740 is used to control the device based on the control instruction, including:

[0200] Based on the control instructions, control the application program running on the wearable device;

[0201] The application is used to perform at least one of the following:

[0202] Control the operation of smart home devices;

[0203] Play or pause multimedia resource display;

[0204] Control the wearable device's camera to capture the object it is looking at.

[0205] In one embodiment of the present disclosure, the recognition module 720 is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image, including:

[0206] Extracting the eye partial image for any frame of facial image;

[0207] Extract features from the eye local image and the facial image to which it belongs respectively;

[0208] Classification is performed based on the features of the local eye image and the features of the facial image to which it belongs to obtain eye behavior information.

[0209] In one embodiment of the present disclosure, when the recognition module 720 is used to extract the eye partial image for any frame of facial image, it includes:

[0210] Recognize the outer contour of the eyes for any frame of facial image;

[0211] By using the minimum circumscribed frame surrounding the outer contour of the eye, the portion within the circumscribed frame is extracted as the extracted eye partial image.

[0212] In one embodiment of the present disclosure, the eye partial image includes a left eye partial image and a right eye partial image, and the recognition module 720 is used to extract features from the eye partial image and the corresponding facial image respectively, including:

[0213] Inputting the left eye partial image and the right eye partial image into the corresponding first feature extraction network for feature extraction, wherein the features extracted by the first feature extraction network include at least one of pupil position features and eye shape features;

[0214] The facial image is input into a second feature extraction network for feature extraction to obtain features of the facial image, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network.

[0215] In one embodiment of the present disclosure, the recognition module 720 is used to classify based on the features of the eye partial image and the features of the facial image to which it belongs to obtain the eye behavior information, including:

[0216] Input the features extracted from the left eye partial image and the features extracted from the right eye partial image into the first fully connected layer for vector space mapping to obtain the output of the first fully connected layer;

[0217] Input the features of the facial image into the second fully connected layer for vector space mapping to obtain the output of the second fully connected layer;

[0218] Input the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer;

[0219] The activation function layer is used to activate the output of the third fully connected layer to classify the eye behavior information.

[0220] In one embodiment of the present disclosure, the first feature extraction network and the second feature extraction network each include at least one convolution block connected in sequence, and any convolution block includes a convolution layer that uses a convolution kernel to perform a convolution operation, and includes a pooling layer;

[0221] Among them, the pooling layer is used to use the most significant features of the convolutional layer output in the convolutional block as the input of the next convolutional block.

[0222] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0223] In this embodiment, the acquisition module acquires one or more frames of facial images, the recognition module performs eye behavior recognition based on any frame of facial images to obtain eye behavior information of each facial image, and the determination module determines the target control instruction based on the eye behavior information of one or more frames of facial images, so that the control module controls the device based on the target control instruction. Thus, the device controls the device through the eye behavior information, which can meet the user's demand for personalized control of the device, improve the richness and convenience of device control, and improve the user experience.

[0224] In order to implement the above embodiment, the present disclosure also proposes a model training device 800.

[0225] Figure 8 is a block diagram of a model training device 800 according to an embodiment of the present disclosure.

[0226] like Figure 8 As shown, the model training device 800 of the embodiment of the present disclosure includes:

[0227] An acquisition module 810 is used to acquire training samples;

[0228] The training module 820 is used to train the recognition model using training samples, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image, so as to determine target control instructions for device control based on the eye behavior information of one or more frames of facial images.

[0229] In one embodiment of the present disclosure, the training samples include sample images of facial and eye parts, and annotated eye behavior information;

[0230] The training module 820 is used to train the recognition model using training samples, including:

[0231] Inputting the sample images of the eye parts and the sample images of the face into the recognition model for feature extraction, and performing classification based on the extracted features to obtain the predicted behavior information output by the recognition model;

[0232] Based on the difference between the predicted behavior information and the labeled eye behavior information, model parameters of the recognition model are adjusted.

[0233] In one embodiment of the present disclosure, the sample images of the eye parts include left eye sample images and right eye sample images; the training module 820 is used to input the sample images of the eye parts and the sample images of the face into the recognition model for feature extraction, and classify based on the extracted features to obtain the predicted behavior information output by the recognition model, including:

[0234] Inputting the left eye sample image and the right eye sample image into a first feature extraction network in the recognition model for feature extraction, wherein the features extracted by the first feature extraction network include at least one of a pupil position feature and an eye shape feature;

[0235] Inputting the sample image of the face into the second feature extraction network in the recognition model for feature extraction, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network;

[0236] Input the features extracted from the left eye sample image and the features extracted from the right eye sample image into the first fully connected layer for vector space mapping, and input the features of the facial sample image into the second fully connected layer for vector space mapping;

[0237] Input the output of the first fully connected layer and the output of the second fully connected layer to the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer;

[0238] The activation function layer is used to activate the output of the third fully connected layer to classify and obtain the predicted behavior information.

[0239] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0240] In this embodiment, the training sample is obtained by the acquisition module, and the training sample is used by the training module to train the recognition model, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each facial image, and to determine the target control instruction for device control based on the eye behavior information of one or more frames of facial image. Thus, the device can train the recognition model and obtain eye behavior information using the trained recognition model, so as to control the device according to the eye behavior information, which can meet the user's demand for personalized control of the device, improve the richness and convenience of device control, and improve the user experience.

[0241] In order to implement the above embodiments, the present disclosure also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the aforementioned device control method, or the aforementioned model training method.

[0242] In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium, when computer instructions are used to enable a computer to execute the aforementioned device control method, or the aforementioned model training method.

[0243] In order to implement the above embodiments, the present disclosure also proposes a computer program product, which, when an instruction processor in the computer program product is executed, executes the aforementioned device control method, or the aforementioned model training method.

[0244] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0245] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0246] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0247] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0248] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0249] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0250] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0251] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A device control method, characterized in that: include: Acquire one or more frames of facial images; Performing eye behavior recognition based on any frame of the facial image to obtain eye behavior information of each of the facial images; Determining a target control instruction based on the eye behavior information of the one or more frames of facial images; The device is controlled based on the target control instruction.

2. The method according to claim 1, characterized in that The eye behavior information includes gaze direction; the determining the target control instruction based on the eye behavior information of the one or more frames of facial images includes: Combining the gaze directions of the multiple frames of facial images according to the acquisition order of the multiple frames of facial images to obtain a gaze direction sequence; matching the gaze direction sequence with a preset standard sequence; The target control instruction is determined according to the control instruction corresponding to the matched standard sequence.

3. The method according to claim 2, characterized in that The matching of the gaze direction sequence with a preset standard sequence includes: Merging adjacent and identical gaze directions in the gaze direction sequence to obtain a merged sequence; If there is at least one continuous subsequence in the combined sequence that is the same as a preset standard sequence, it is determined that the gaze direction sequence matches the standard sequence.

4. The method according to claim 1, characterized in that: The performing device control based on the target control instruction includes: Based on the target control instruction, controlling the application program running on the wearable device; The application is used to perform at least one of the following: Control the operation of smart home devices; Play or pause multimedia resource display; The camera of the wearable device is controlled to capture the object being watched.

5. The method according to any one of claims 1 to 4, characterized in that: The performing eye behavior recognition based on any frame of the facial image to obtain eye behavior information of each facial image includes: Extracting an eye partial image for any frame of the facial image; Extracting features from the eye partial image and the corresponding facial image respectively; Classification is performed based on the features of the eye local image and the features of the corresponding facial image to obtain the eye behavior information.

6. The method according to claim 5, characterized in that The step of extracting the eye partial image from the facial image of any frame comprises: Performing eye contour recognition on any frame of the facial image; By using a minimum circumscribed frame surrounding the outer contour of the eye, a portion within the circumscribed frame is extracted as the extracted eye partial image.

7. The method according to claim 5, characterized in that The eye partial image includes a left eye partial image and a right eye partial image, and the feature extraction of the eye partial image and the corresponding facial image respectively includes: Inputting the left eye partial image and the right eye partial image into corresponding first feature extraction networks for feature extraction, respectively, wherein the features extracted by the first feature extraction network include at least one of pupil position features and eye shape features; The facial image is input into a second feature extraction network for feature extraction to obtain features of the facial image, wherein a convolution kernel of the second feature extraction network is larger than a convolution kernel of the first feature extraction network.

8. The method according to claim 7, characterized in that The classifying based on the features of the eye local image and the features of the corresponding facial image to obtain the eye behavior information includes: Input the features extracted from the left eye partial image and the features extracted from the right eye partial image into a first fully connected layer for vector space mapping to obtain an output of the first fully connected layer; Inputting the features of the facial image into a second fully connected layer for vector space mapping to obtain an output of the second fully connected layer; Inputting the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer; An activation function layer is used to activate the output of the third fully connected layer to classify the eye behavior information.

9. The method according to claim 7, characterized in that: The first feature extraction network and the second feature extraction network each include at least one convolution block connected in sequence, and any of the convolution blocks includes a convolution layer that uses a convolution kernel to perform a convolution operation, and includes a pooling layer; The pooling layer is used to use the most significant feature of the output of the convolution layer in the convolution block as the input of the next convolution block.

10. A model training method, characterized in that: include: Get training samples; The training samples are used to train the recognition model, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each of the facial images, so as to determine target control instructions for device control based on the eye behavior information of the one or more frames of facial images.

11. The method according to claim 10, characterized in that The training samples include sample images of facial and eye parts, and annotated eye behavior information; The adopting the training sample to train the recognition model includes: Inputting the sample image of the eye part and the sample image of the face into the recognition model for feature extraction, and performing classification based on the extracted features to obtain predicted behavior information output by the recognition model; Based on the difference between the predicted behavior information and the labeled eye behavior information, a model parameter of the recognition model is adjusted.

12. The method according to claim 11, characterized in that The sample images of the local parts of the eyes include a left eye sample image and a right eye sample image; the sample images of the local parts of the eyes and the sample images of the face are input into the recognition model for feature extraction, and classification is performed based on the extracted features to obtain the predicted behavior information output by the recognition model, including: Inputting the left eye sample image and the right eye sample image into a first feature extraction network in the recognition model for feature extraction, wherein the features extracted by the first feature extraction network include at least one of a pupil position feature and an eye shape feature; Inputting the sample image of the face into a second feature extraction network in the recognition model for feature extraction, wherein the convolution kernel of the second feature extraction network is larger than the convolution kernel of the first feature extraction network; Input the features extracted from the left eye sample image and the features extracted from the right eye sample image into a first fully connected layer for vector space mapping, and input the features of the facial sample image into a second fully connected layer for vector space mapping; Inputting the output of the first fully connected layer and the output of the second fully connected layer into the third fully connected layer for vector space mapping to obtain the output of the third fully connected layer; An activation function layer is used to activate the output of the third fully connected layer to classify and obtain the predicted behavior information.

13. A device control device, characterized in that: include: An acquisition module, used for acquiring one or more frames of facial images; A recognition module, used for performing eye behavior recognition based on any frame of the facial image to obtain eye behavior information of each of the facial images; A determination module, configured to determine a target control instruction based on the eye behavior information of the one or more frames of facial images; A control module is used to control the device based on the target control instruction.

14. A model training device, characterized in that: include: An acquisition module is used to acquire training samples; A training module is used to train a recognition model using the training samples, wherein the trained recognition model is used to perform eye behavior recognition based on any frame of facial image to obtain eye behavior information of each of the facial images, so as to determine target control instructions for device control based on the eye behavior information of the one or more frames of facial images.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9, or the method of any one of claims 10-12.

16. A computer-readable storage medium, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9, or the method according to any one of claims 10 to 12.

Images