Blink detection method, electronic device and storage medium

Abstract

The embodiments of the present application provide a blink detection method, an electronic device, and a storage medium, which relate to the field of data processing technology. The method includes: identifying the eye area of ​​the target object in each image contained in the target image sequence; detecting and obtaining a multimodal result reflecting whether a complete blink event has occurred from multiple dimensions based on the eye area in each image, wherein the complete blink event is: the target object blinks within the image acquisition time period corresponding to the target image sequence; fusing each single-modal result in the multimodal result to obtain a fusion result; and determining whether the target object blinks based on the fusion result. Applying the solution provided by the embodiments of the present application can improve the accuracy of blink detection.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a blink detection method, an electronic device and a storage medium. Background Art

[0002] Blink detection is used to detect the blinking behavior of an object and is an important detection item in object recognition.

[0003] In the existing blink detection scheme, the electronic device for detection can obtain multiple continuous images of the object captured by the image acquisition device, detect whether there are closed-eye images of the object when the eyes are closed in the multiple images, and when it is detected that the multiple images contain at least one closed-eye image, it is determined that the object blinked during the acquisition time period of the multiple images. However, the blinking process can be considered as the process of closing and opening the eyes. The scheme of judging whether the object blinks only by detecting whether there are closed-eye images in multiple images is not consistent with the blinking process. Therefore, the accuracy of blink detection using the existing blink detection scheme is low. Summary of the invention

[0004] In view of this, the present application provides a blink detection method, an electronic device and a storage medium to improve the accuracy of blink detection.

[0005] In a first aspect, an embodiment of the present application provides a blink detection method, the method comprising:

[0006] Identify the eye region of the target object in each image included in the target image sequence;

[0007] According to the eye regions in each image, a multimodal result is detected to reflect whether a complete blink event occurs from multiple dimensions, wherein the complete blink event is: the target object blinks within the image acquisition time period corresponding to the target image sequence;

[0008] fusing the single modality results in the multimodality results to obtain a fusion result;

[0009] According to the fusion result, it is determined whether the target object blinks.

[0010] As can be seen from the above, when the scheme provided by the embodiment of the present application is used for blink detection, according to the eye area of ​​the target object in each image contained in the target image sequence, a multimodal result reflecting whether a complete blink event occurs in multiple dimensions is detected, wherein a complete blink event refers to a blink of the target object in the image acquisition time period corresponding to the target image sequence, that is, the target object closes and opens its eyes in the image acquisition time period. Therefore, each single-modal result in the multimodal result actually reflects from one dimension whether the eyes of the target object are closed and opened in the image acquisition time period, thereby fusing each single-modal result in the multimodal result, and determining whether the target object blinks according to the fusion result. This detection method is consistent with the actual blinking process. Moreover, the fusion result is obtained by fusing multiple single-modal results, which comprehensively reflects whether a complete blink event occurs in multiple different dimensions. Therefore, according to the fusion result, it is possible to accurately determine whether the target object blinks. In view of this, the blink detection scheme provided by the embodiment of the present application can improve the accuracy of blink detection.

[0011] In one embodiment of the present application, each single-modal result in the multi-modal result includes a plurality of sub-results;

[0012] The fusing of the single-modality results in the multi-modality results to obtain a fusion result includes:

[0013] For each single-modal result in the multi-modal result, each sub-result included in the single-modal result is merged to obtain an intermediate result;

[0014] The intermediate results are fused to obtain a fusion result.

[0015] It can be seen from the above that when the scheme provided in the embodiment of the present application is used for blink detection, the sub-results contained in each single-modal result are fused to obtain an intermediate result. Each intermediate result can comprehensively reflect whether a complete blink event has occurred from multiple situations in one dimension. The intermediate results corresponding to each single-modal result are then fused, which can be understood as fusing multiple dimensions. The fused result obtained by the fusion can comprehensively reflect whether a complete blink event has occurred from more situations in multiple dimensions. Therefore, based on the fusion result, it is possible to more accurately determine whether the target object has blinked.

[0016] In one embodiment of the present application, the fusion of the sub-results included in the single-modality result to obtain the intermediate result includes:

[0017] Normalizing each sub-result included in the single-modal result;

[0018] Multiply each normalized sub-result by the preset result label value;

[0019] Accumulate the multiplication results;

[0020] Obtain intermediate results based on the accumulated results.

[0021] In this scheme, each result label value can be understood as a value representing a result, that is, a value representing a situation, and each sub-result in the unimodal result can reflect whether a complete blink event will occur in one dimension. Each normalized sub-result is multiplied by each structural label value, and each multiplication result is accumulated. It is possible to integrate multiple situations in one dimension, and obtain an intermediate result based on the accumulated result. The intermediate result can comprehensively reflect whether a complete blink event occurs from multiple situations in one dimension. In this way, multiple intermediate results are integrated to obtain a fused result, which can comprehensively reflect whether a complete blink event occurs from more situations in multiple dimensions. Therefore, based on the fused result, it is possible to more accurately determine whether the target object blinks.

[0022] In one embodiment of the present application, obtaining an intermediate result according to the accumulated result includes:

[0023] The accumulated result is subtracted from one to obtain an intermediate result.

[0024] By adopting this solution, the intermediate result can be accurately obtained, thereby improving the accuracy of blink detection.

[0025] In one embodiment of the present application, obtaining an intermediate result according to the accumulated result includes:

[0026] The total number of images included in the target image sequence is subtracted from the accumulated result to obtain an intermediate result.

[0027] By adopting this solution, the intermediate result can be accurately obtained, thereby improving the accuracy of blink detection.

[0028] In one embodiment of the present application, obtaining an intermediate result according to the accumulated result includes:

[0029] The accumulated result is divided by the total number of images to obtain an intermediate result.

[0030] By adopting this solution, the intermediate result can be accurately obtained, thereby improving the accuracy of blink detection.

[0031] In one embodiment of the present application, obtaining an intermediate result according to the accumulated result includes:

[0032] The accumulated result is subtracted from the total number of images, and the subtraction result is divided by the total number of images to obtain an intermediate result.

[0033] By adopting this solution, the intermediate result can be accurately obtained, thereby improving the accuracy of blink detection.

[0034] In one embodiment of the present application, obtaining an intermediate result according to the accumulated result includes:

[0035] The accumulated result is determined as an intermediate result.

[0036] By adopting this solution, the intermediate result can be accurately obtained, thereby improving the accuracy of blink detection.

[0037] In one embodiment of the present application, the multimodal result includes at least two of the following single-modal results:

[0038] A first single-modal result reflecting the occurrence of a complete blink event from the dimension of whether the target object wears glasses;

[0039] A second single modality result reflecting the occurrence of a complete blink event from the dimension of the number of abnormal images contained in the target image sequence;

[0040] A third single-modality result reflecting the occurrence of a complete blink event, which is a dimension of a position of a start blink image of the target object in the target image sequence;

[0041] A fourth single-modal result reflecting the occurrence of a complete blink event from the dimension of the position of the ending blink image of the target object in the target image sequence;

[0042] A fifth single modality result reflecting the occurrence of a complete blink event from the dimension of the number of blink images included in the target image sequence;

[0043] From the dimension of blink stage events, a sixth unimodal result reflecting the occurrence of complete blink events is provided, wherein blink stage events include: events in which the target object blinks completely within the image acquisition time period, events in which the target object blinks incompletely, and events in which the target object does not blink.

[0044] Each of the above six single-modal results can accurately reflect whether a complete blink event occurs from various different dimensions, and the above multimodal result includes at least two of the above six single-modal results. In this way, the single-modal results in the multimodal result are fused to obtain a fusion result. The fusion result can comprehensively reflect whether a complete blink event occurs from multiple different dimensions. Therefore, based on the fusion result, it is possible to accurately determine whether the target object blinks while comprehensively considering multiple dimensions, thereby improving the accuracy of blink detection.

[0045] In one embodiment of the present application, the method further includes:

[0046] According to each image included in the target image sequence, detecting the posture of the target object at the time of acquisition of each image, and obtaining posture information corresponding to each image;

[0047] The detection of the eye regions in each image to obtain a multimodal result reflecting the occurrence of a complete blink event from multiple dimensions includes:

[0048] According to the eye regions in each image and the posture information corresponding to each image, a multimodal result reflecting the occurrence of a complete blink event from multiple dimensions is detected.

[0049] From the above, it can be seen that when the solution provided in the embodiment of the present application is applied to perform blink detection, when the multimodal result is detected, not only the eye area of ​​each image is detected, but also the posture information of the target object corresponding to each image is detected. This can increase the information on which the detection is based, improve the accuracy of the multimodal results, overcome the interference of the detection due to the deformation of the eye area of ​​the target object in postures such as lowering or raising the head (such as the eyelid distance becomes smaller when lowering the head), thereby improving the accuracy of blink detection.

[0050] In one embodiment of the present application, determining whether the target object blinks according to the fusion result includes:

[0051] Acquire a fusion result corresponding to an auxiliary image sequence adjacent to the target image sequence;

[0052] Whether the target object blinks is determined according to a fusion result corresponding to the target image sequence and a fusion result corresponding to the auxiliary image sequence.

[0053] In this solution, after obtaining the fusion results corresponding to the target image sequence and the fusion results corresponding to the auxiliary image sequence, it is possible to determine whether the target object has blinked based on the two fusion results. In this way, the fusion results corresponding to multiple image sequences are considered when determining whether the target object has blinked, and the information considered is richer, so it is possible to more accurately determine whether the target object has blinked. In addition, compared with determining whether the target object has blinked based only on the fusion results corresponding to the target image sequence, determining based on the fusion results corresponding to multiple image sequences can reduce the false alarm rate, which can improve the user experience in scenarios where users use electronic devices or drive vehicles.

[0054] In one embodiment of the present application, the target image sequence and the auxiliary image sequence satisfy at least one of the following conditions:

[0055] The starting image in each image sequence is an image acquired sequentially by the image acquisition device;

[0056] The images included in each set of image sequences are: images continuously acquired by the image acquisition device;

[0057] Each image sequence contains the same number of images.

[0058] If the target image sequence and the auxiliary image sequence satisfy that the starting image in each image sequence is an image sequentially captured by the image capture device, the electronic device can quickly determine the auxiliary image sequence, thereby improving the efficiency of blink detection.

[0059] If the target image sequence and the auxiliary image sequence satisfy that the images contained in each group of image sequences are: images continuously captured by the image acquisition device, then the electronic device can improve the accuracy of blink detection by adopting the blink detection solution provided in the embodiment of the present application.

[0060] If the target image sequence and the auxiliary image sequence satisfy the requirement that the number of images contained in each image sequence is the same, the electronic device adopts the blink detection scheme provided in the embodiment of the present application to make the fusion result corresponding to each image sequence more standardized. In this way, determining whether the target object blinks based on the fusion results corresponding to multiple image sequences can improve the reliability of blink detection.

[0061] In one embodiment of the present application, determining whether the target object blinks according to the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence includes:

[0062] Calculating the variance of the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence and the sum of the fusion results;

[0063] If the calculated variance is greater than a first preset threshold, the sum of the fusion results is greater than a second preset threshold, the fusion result corresponding to the target image sequence is greater than a third preset threshold, and the distance between the target image sequence and the image sequence corresponding to the target object's last blink is greater than a preset distance, it is determined that the target object blinked within the image acquisition time period corresponding to the target image sequence.

[0064] In this solution, when the variance is greater than the first preset threshold, the sum of the fusion results is greater than the second preset threshold, the fusion result corresponding to the target image sequence is greater than the third preset threshold, and the above distance is greater than the preset distance, it can be reliably proved that the target object blinked during the image acquisition time period corresponding to the target image sequence, that is, a complete blink event occurred. It can be seen that the use of this implementation method can accurately determine whether the target object blinks, thereby improving the accuracy of blink detection.

[0065] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0066] one or more processors and memory;

[0067] The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in the first aspect.

[0068] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer program, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect.

[0069] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on an electronic device, the electronic device executes the method described in the first aspect.

[0070] In the fifth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processors are used to call computer instructions so that the electronic device inputs data into the chip system and executes the method described in the first aspect to perform blink detection.

[0071] The beneficial effects of the solutions provided by the embodiments in the second to fifth aspects can refer to the beneficial effects of the solutions provided by the embodiments in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0073] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0074] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a flow chart of a first blink detection method provided in an embodiment of the present application;

[0076] Figure 4a A schematic diagram of a process for obtaining multimodal results provided in an embodiment of the present application;

[0077] Figure 4b A schematic diagram of the structure of a detection model provided in an embodiment of the present application;

[0078] Figure 4c A schematic diagram of the structure of the first network layer provided in an embodiment of the present application;

[0079] Figure 4d A schematic diagram of the structure of the second network layer provided in the embodiment of the present application;

[0080] Figure 4e A schematic diagram of the structure of the third network layer provided in the embodiment of the present application;

[0081] Figure 4f A schematic diagram of a process of converting a network layer provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of the distribution of parameter values ​​of a target parameter provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of a result fusion process provided in an embodiment of the present application;

[0084] Figure 7 A schematic diagram of a flow chart of a second blink detection method provided in an embodiment of the present application;

[0085] Figure 8 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0087] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first instruction and the second instruction are to distinguish different user instructions, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0088] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0089] The solution provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, smart watches, and wearable electronic devices.

[0090] A possible hardware structure and software structure of the electronic device described above are described below.

[0091] For example, Figure 1 The structure diagram of the electronic device 1000 is shown. The electronic device 1000 may include a processor 1010, a display screen 1020, a camera 1030, an internal memory 1040, a Subscriber Identification Module (SIM) card interface 1050, a Universal Serial Bus (USB) interface 1060, a charging management module 1070, a battery management module 1071, a battery 1072, a sensor module 1080, a mobile communication module 1090, a wireless communication module 1100, an antenna 1 and an antenna 2, etc. The sensor module 1080 may include a pressure sensor 1080A, a touch sensor 1080B, etc.

[0092] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0093] The processor 1010 may include one or more processing units, for example: the processor 1010 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 1000 may also include one or more processors 1010. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be provided in the processor 1010 for storing instructions and data. Exemplarily, the memory in the processor 1010 may be a cache memory. The memory may store instructions or data that have just been used or circulated by the processor 1010.

[0094] In some embodiments, the processor 1010 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 1060 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 1060 can be used to connect a charger to charge the electronic device 1000, and can also be used to transmit data between the electronic device 1000 and a peripheral device. The USB interface 1060 can also be used to connect headphones to play audio through the headphones.

[0095] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0096] The wireless communication function of the electronic device 1000 can be implemented through the antenna 1, the antenna 2, the mobile communication module 1090, the wireless communication module 1100, the modem processor, the baseband processor, and the like.

[0097] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0098] The electronic device 1000 implements the display function through a GPU, a display screen 1020, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1020 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1010 may include one or more GPUs, which execute program instructions to generate or change display information.

[0099] The display screen 1020 is used to display pictures, videos, etc. The display screen 1020 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1000 may include one or more display screens 1020.

[0100] In some embodiments of the present application, when the display panel is made of materials such as OLED, AMOLED, FLED, etc., the above Figure 1 The display screen 1020 in the embodiment can be bent. Here, the display screen 1020 can be bent, which means that the display screen can be bent to any angle at any position and can be kept at the angle. For example, the display screen 1020 can be folded in half from the middle to the left or right. It can also be folded in half from the middle to the top or bottom.

[0101] The display screen 1020 of the electronic device 1000 may be a flexible screen. Compared with a traditional screen, the flexible screen has the characteristics of strong flexibility and bendability, and can provide users with a new interaction method based on the bendable characteristics. For an electronic device equipped with a foldable display screen, the foldable display screen on the electronic device can be switched between a small screen in a folded state and a large screen in an unfolded state at any time. Therefore, the user can also use the split screen function on the electronic device equipped with a foldable display screen.

[0102] The display screen 1020 displays the desktop of the operating system deployed on the electronic device, and the desktop displays application icons of various application programs installed on the electronic device.

[0103] The electronic device 1000 can realize the shooting function through the ISP, the camera 1030, the video codec, the GPU, the display screen 1020 and the application processor, wherein the camera 1030 includes a front camera and a rear camera.

[0104] The ISP is used to process the data fed back by the camera 1030. For example, when shooting, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be set in the camera 1030.

[0105] The camera 1030 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red green blue (RGB), YUV or other format. In some embodiments, the electronic device 1000 may include 1 or N cameras 1030, where N is a positive integer greater than 1.

[0106] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 1000 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0107] Video codecs are used to compress or decompress digital videos. The electronic device 1000 may support one or more video codecs. Thus, the electronic device 1000 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0108] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 1000, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0109] The internal memory 1040 can be used to store one or more computer programs, which include instructions. The processor 1010 can execute the above instructions stored in the internal memory 1040, so that the electronic device 1000 performs the blink detection method provided in some embodiments of the present application, as well as various applications and data processing. The internal memory 1040 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 1000. In addition, the internal memory 1040 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (Universal Flash Storage, UFS), etc. In some embodiments, the processor 1010 can execute the blink detection method provided in the embodiment of the present application, as well as other applications and data processing by running instructions stored in the internal memory 1040 and / or instructions stored in a memory provided in the processor 1010.

[0110] The internal memory 1040 can be used to store the related programs of the blink detection method provided in the embodiment of the present application, and the processor 1010 can be used to call the related programs of the blink detection method stored in the internal memory 1040 to execute the blink detection method of the embodiment of the present application.

[0111] Specifically, when the user is using the electronic device 1000, the camera 1030 can capture an image of the user, the processor 1010 can obtain the image captured by the camera 1030, and execute the blink detection method of the embodiment of the present application to detect whether the user blinks through the obtained image.

[0112] The sensor module 1080 may include a pressure sensor 1080A, a touch sensor 1080B, and the like.

[0113] The pressure sensor 1080A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 1080A can be set on the display screen 1020. There are many types of pressure sensors 1080A, for example, they can be resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When the force acts on the pressure sensor 1080A, the capacitance between the electrodes changes, and the electronic device 1000 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 1020, the electronic device 1000 detects the touch operation according to the pressure sensor 1080A. The electronic device 1000 can also calculate the position of the touch according to the detection signal of the pressure sensor 1080A, and determine the application or the control in the application to which the touch operation is applied based on the position of the touch. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed; when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0114] Touch sensor 1080B is also called a touch device. Touch sensor 1080B can be set on display screen 1020, and touch sensor 1080B and display screen 1020 form a touch screen, which is also called a touch screen. Touch sensor 1080B is used to detect touch operations acting on or near it. Touch sensor 1080B can pass the detected touch operation to the application processor to determine the touch event type, which may include click type, long press type, sliding type, etc. Visual output related to the touch operation can be provided through display screen 1020. In some embodiments, different touch event types may correspond to different operation instructions. In other embodiments, touch sensor 1080B may also be set on the surface of electronic device 1000 and be set at a different position from display screen 1020.

[0115] Figure 2 The software structure diagram of the electronic device to which the embodiment of the present application is applicable is shown in FIG. The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

[0116] The layered architecture divides the software system of the electronic device into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into five layers, namely, application layer (applications), application framework layer (application framework), system library, hardware abstraction layer (HAL) and kernel layer (kernel).

[0117] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. Figure 2 As shown, the application package may include application programs such as settings, gallery, instant messaging software, video, etc. It can be understood that the port of each of the above application programs can be used to receive data.

[0118] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, and a Dynamic Host Configuration Protocol (DHCP) module.

[0119] Among them, the window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0120] Content providers are used to store and retrieve data and make it accessible to applications. The data may include video, pictures, audio, etc.

[0121] The view system includes visual controls, such as controls for displaying pictures, controls for displaying application icons, controls for displaying text, controls for displaying buttons, etc.

[0122] The resource manager provides various resources for applications, such as localized strings, application icons, images, layout files, video files, and so on.

[0123] The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and disappear automatically after a short stay without user interaction.

[0124] The system library can include multiple functional modules, such as surface manager, 3D graphics processing library, 2D graphics engine and file library. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0125] The hardware abstraction layer may include multiple library modules, such as a hard disk library module and a display library module. The electronic device system may load the corresponding library module for the device hardware, thereby realizing the purpose of the process accessing the device hardware. For example, the electronic device system may load the corresponding library module for the hard disk, thereby realizing the process accessing the system configuration file stored in the hard disk.

[0126] The kernel layer is a layer between hardware and software. The kernel layer is deployed with an operating system for driving the hardware to make the hardware work. The kernel layer at least includes a hard disk drive, a display drive, a camera drive, a sensor drive, and a motor drive, etc., which are not limited in the embodiments of the present application. It can be understood that the hard disk drive, the display drive, the camera drive, the sensor drive, and the motor drive, etc. can all be regarded as a driver node. Each of the above-mentioned driver nodes includes an interface that can be used to receive data.

[0127] Among them, in the solution provided by the embodiment of the present application, the operating system deployed in the kernel layer of the electronic device can execute the blink detection solution provided by the embodiment of the present application.

[0128] The following describes various application scenarios of the blink detection solution provided in the embodiments of the present application.

[0129] In the first application scenario, when a user uses an electronic device, the electronic device can detect whether the user blinks and perform various operations based on the detection results, so that the user can achieve various functions by blinking. For example, after the user turns on the camera, the user can control the camera to take an image by blinking, and for another example, the user can blink twice quickly to take a screenshot, etc.

[0130] In the second application scenario, when the user is driving the vehicle, the vehicle can detect in real time whether the user blinks, count the number and frequency of the user's blinks, etc., and determine whether the user is driving fatigued based on the counted information.

[0131] In the third application scenario, during the eye movement test, the testing device focuses on the movement of the user's eyeballs. When the user blinks, the user's eyes will be blocked, which will affect the test results of the eye movement test. Therefore, during the eye movement test, it is necessary to detect whether the user blinks to avoid the impact of the user's blinking on the eye movement test.

[0132] The blink detection method, electronic device and storage medium provided in the embodiments of the present application are described in detail below.

[0133] In one embodiment of the present application, see Figure 3 , a flow chart of a first blink detection method is provided. In this embodiment, the method includes the following steps S301-S304.

[0134] Step S301: identifying the eye region of the target object in each image included in the target image sequence.

[0135] The images included in the target image sequence may be multiple images collected by an image acquisition device, and the images included in the target image sequence may be continuous images among the images collected by the image acquisition device, or images extracted from every other image or multiple images.

[0136] Specifically, the target image sequence includes multiple images of the target object. For each image, the electronic device can use a pre-trained object recognition model or an object recognition algorithm to recognize the eye region of the target object in the image.

[0137] If the image acquisition device only captures the image of one eye of the target object when capturing an image, then when the electronic device identifies the eye area of ​​the target object in the image, it can identify the single eye area of ​​the target object; if the image acquisition device captures the image of both eyes of the target object, then when the electronic device identifies the eye area of ​​the target object in the image, it can identify the binocular area of ​​the target object.

[0138] Since the eye region of the target object in the image is often small, during the recognition process, the facial region of the target object in the image may be recognized first, and then the eye region included in the facial region of the target object may be recognized.

[0139] When an image contains image contents of multiple objects, when identifying the facial region of a target object in the image, the facial regions of each object in the image may be identified, and any of the following three implementation methods may be used to determine the facial region of the same target object among the identified facial regions:

[0140] In a first implementation, the facial region of the same target object may be determined based on the size of the facial region of each object, for example, the largest facial region included in each image may be determined as the facial region of the same target object.

[0141] In the second implementation, the facial region of the target object can be determined according to the location of the facial region of each object. For example, the facial region whose center is closest to the center of the image contained in each image is determined to be the facial region of the same target object.

[0142] In a third implementation manner, a facial region that is closest in position to a facial region of a target object in an adjacent image may be determined as the facial region of the same target object.

[0143] In addition, for each image in the target image sequence, after the facial area of ​​the same target object in each image is identified, the facial area of ​​the same target object in each image can be subjected to facial alignment processing, and then the eye area of ​​the target object can be identified in each area after the facial alignment processing.

[0144] In one embodiment of the present application, before identifying the eye region, the electronic device may perform image preprocessing such as image filtering, image denoising, image enhancement, etc. on each image in the target image sequence, thereby identifying the eye region based on each image after the image preprocessing.

[0145] For example, the image is pre-processed before recognizing the face region of the target object in the image, before recognizing the eye region included in the face region of the target object, or after the face alignment process.

[0146] Step S302: Based on the eye regions in each image, a multimodal result is detected to obtain a multimodal result reflecting whether a complete blink event occurs from multiple dimensions.

[0147] The complete blink event is: the target object blinks within the image acquisition time period corresponding to the target image sequence.

[0148] The image acquisition time period corresponding to the above target image sequence is: the time period between the acquisition time of the earliest acquired image and the acquisition time of the latest acquired image in the target image sequence.

[0149] The above multimodal result includes multiple unimodal results, each of which corresponds to a dimension and reflects whether a complete blink event occurs in the dimension.

[0150] In one embodiment of the present application, the multimodal result includes at least two of the following single-modal results:

[0151] The first single-modal result reflecting the occurrence of a complete blink event from the dimension of whether the target object wears glasses;

[0152] A second single-modality result reflecting the occurrence of a complete blink event from the dimension of the number of abnormal images contained in the target image sequence;

[0153] The dimension of the position of the start blink image of the target object starting to blink in the target image sequence, reflecting the third single modality result of the occurrence of the complete blink event;

[0154] A fourth single-modality result reflecting the occurrence of a complete blink event, based on the dimension of the position of the ending blink image of the target subject in the target image sequence;

[0155] A fifth single modality result reflecting the occurrence of a complete blink event from the dimension of the number of blink images contained in the target image sequence;

[0156] From the dimension of blink stage events, a sixth unimodal result is provided that reflects the occurrence of complete blink events, wherein blink stage events include: events in which the target object blinks completely within the image acquisition time period, events in which the target object blinks incompletely, and events in which the target object does not blink.

[0157] For the introduction of the above six single-mode results, please refer to the subsequent embodiments, which will not be described in detail here.

[0158] Specifically, after identifying the eye area in each image, the electronic device can detect and obtain multiple single-modal results in multiple dimensions according to the eye area in each image, thereby obtaining a multi-modal result including multiple single-modal results.

[0159] For example, for the above-mentioned first unimodal result, the electronic device can detect whether the target object is wearing glasses based on the eye area in each image to obtain the confidence that the target object is wearing glasses, and detect whether the target object is not wearing glasses to obtain the confidence that the target object is not wearing glasses. In this way, a first unimodal result including these two confidences can be obtained, or a first unimodal result including either of the two confidences can be obtained.

[0160] For the detection process of other single-modal results, please refer to the subsequent introduction of the six single-modal results, which will not be described in detail here.

[0161] Step S303: Fusing the single-modality results in the multi-modality results to obtain a fusion result.

[0162] Each unimodal result included in the multimodal result reflects whether a complete blink event occurs from one dimension. By fusing the unimodal results in the multimodal result, the dimensions corresponding to each unimodal result can be comprehensively considered. The fused result obtained by fusion can comprehensively reflect whether a complete blink event occurs.

[0163] The specific implementation method of fusing the single-modal results in the multi-modal results can be found in the subsequent embodiments, which will not be described in detail here.

[0164] Step S304: Determine whether the target object blinks according to the fusion result.

[0165] Specifically, when fusing the unimodal results in the multimodal results, the various confidence levels contained in the unimodal results can be fused to obtain a final confidence level that comprehensively reflects whether a complete blinking event occurred. In this way, based on the fusion result, if the confidence level contained in the fusion result is high, it can be determined that the target object blinked during the above-mentioned image acquisition time period.

[0166] For example, if the confidence level included in the fusion result is greater than a preset confidence level threshold, it can be determined that the target object blinked during the above image acquisition time period.

[0167] In addition, the electronic device can also combine the fusion results corresponding to the target image sequence and the fusion results corresponding to other image sequences to jointly determine whether the target object blinks. This determination method can be specifically referred to in subsequent embodiments and will not be described in detail here.

[0168] As can be seen from the above, when the scheme provided by the embodiment of the present application is used for blink detection, according to the eye area of ​​the target object in each image contained in the target image sequence, a multimodal result reflecting whether a complete blink event occurs in multiple dimensions is detected, wherein a complete blink event refers to a blink of the target object in the image acquisition time period corresponding to the target image sequence, that is, the target object closes and opens its eyes in the image acquisition time period. Therefore, each single-modal result in the multimodal result actually reflects from one dimension whether the eyes of the target object are closed and opened in the image acquisition time period, thereby fusing each single-modal result in the multimodal result, and determining whether the target object blinks according to the fusion result. This detection method is consistent with the actual blinking process. Moreover, the fusion result is obtained by fusing multiple single-modal results, which comprehensively reflects whether a complete blink event occurs in multiple different dimensions. Therefore, according to the fusion result, it is possible to accurately determine whether the target object blinks. In view of this, the blink detection scheme provided by the embodiment of the present application can improve the accuracy of blink detection.

[0169] The first to sixth single-modal results mentioned in the above step S302 are introduced below.

[0170] Each unimodal result from the first unimodal result to the sixth unimodal result corresponds to a dimension, and there are multiple situations under each dimension. Therefore, each unimodal result includes multiple sub-results corresponding to the multiple situations under one dimension, and each sub-result can be expressed by the confidence level of the corresponding situation.

[0171] 1. First single-mode results

[0172] The first unimodal result corresponds to the dimension of whether the target object wears glasses.

[0173] When the target object wears glasses, the glasses may reflect light, which may affect the eye area and interfere with blink detection. Therefore, when detecting whether the target object blinks during the image acquisition time period, that is, when judging whether a complete blink event occurs, the dimension of whether the target object wears glasses can be considered to avoid interference when the target object wears glasses.

[0174] In the dimension of whether the target object wears glasses, there are two situations: the target object wears glasses and the target object does not wear glasses. For these two situations, the electronic device can respectively detect and obtain the confidences corresponding to the two situations based on the eye areas in each image, that is, detect the confidence that the target object wears glasses, and detect the confidence that the target object does not wear glasses, thereby obtaining a first unimodal result that includes the confidences corresponding to the two situations.

[0175] Two implementation methods of obtaining the first single-modality result according to the eye region detection in each image are described below.

[0176] In the first implementation method, the electronic device can use the eye area in each image as input data of a pre-trained first classification model, and input it into the first classification model. The first classification model detects the confidence level of the target object wearing glasses and the confidence level of not wearing glasses based on the eye area in each image, and outputs the two confidence levels, so that the electronic device can obtain a first unimodal result including the two confidence levels.

[0177] In a second implementation, the electronic device may use a prediction algorithm, function, etc. to predict the confidence level of the target object wearing glasses and the confidence level of the target object not wearing glasses based on the eye area in each image, thereby obtaining a first unimodal result including these two confidence levels.

[0178] 2. Second single mode results

[0179] The second single-modality result corresponds to the dimension of the number of abnormal images contained in the target image sequence.

[0180] The above-mentioned abnormal images may include: images in which the facial area of ​​the target object is too small, images in which the eye area of ​​the target object is too small, images in which the facial area of ​​the target object does not exist, images in which the eye area of ​​the target object does not exist, images with low image quality, etc.

[0181] For example, the above-mentioned image with a facial region that is too small may be an image with a facial region area that is smaller than a preset first area threshold, or an image with a facial region width or height that is smaller than a preset first size threshold.

[0182] The image with too small eye region may be an image with an eye region area smaller than a preset second area threshold, or an image with an eye region width or height smaller than a preset second size threshold.

[0183] The image with lower image quality may be an image whose parameter value of image quality parameter is less than a preset parameter threshold.

[0184] During the blink detection process, the greater the number of abnormal images contained in the target image sequence, the less reliable the blink detection result is, and the less it can reflect whether a complete blink event has occurred. The smaller the number of abnormal images, the more reliable the blink detection result is, and the more it can reflect whether a complete blink event has occurred.

[0185] In the dimension of the number of abnormal images included in the target image sequence, the number of abnormal images may be any integer in the range of zero to the total number N of images included in the target image sequence. Therefore, there are N+1 situations in this dimension. The electronic device can detect and obtain a second unimodal result containing sub-results corresponding to the N+1 situations based on the eye area in each image.

[0186] For example, if the target image sequence contains three images, there are four situations: the target image sequence contains no abnormal images, contains one abnormal image, contains two abnormal images, and contains three abnormal images. The electronic device can detect the confidence that the target image sequence does not contain abnormal images, the confidence that it contains one abnormal image, the confidence that it contains two abnormal images, and the confidence that it contains three abnormal images based on the eye areas in each image, thereby obtaining a second unimodal result containing these four confidences.

[0187] Two implementation methods of obtaining the second single-modality result according to the eye region detection in each image are described below.

[0188] In the first implementation method, the electronic device can use the eye area in each image as input data of a pre-trained second classification model, and input it into the second classification model. The second classification model detects the confidence of various situations based on the eye area in each image and outputs each confidence, so that the electronic device can obtain a second unimodal result containing each confidence.

[0189] In a second implementation, the electronic device may use a prediction algorithm, function, etc. to predict the confidence of various situations based on the eye area in each image, thereby obtaining a second unimodal result including the confidence of various situations.

[0190] 3. The third single mode results

[0191] The third single-modality result corresponds to the dimension of the position of the start blink image at which the target object starts blinking in the target image sequence.

[0192] In the target image sequence, if there is a starting blink image, and the earlier the position of the starting blink image is, the higher the possibility that the target object blinked during the above-mentioned image acquisition time period, and the more it can reflect the occurrence of a complete blink event; the later the position of the starting blink image is, the lower the possibility that the target object blinked during the above-mentioned image acquisition time period.

[0193] For example, if the second to last image in the target image sequence is the starting blink image, the target object may not have opened its eyes at the end of the above-mentioned image acquisition time period. In this way, the above-mentioned image acquisition time period does not include the complete blinking action of the target object, which also reflects that the above-mentioned complete blinking event did not occur.

[0194] In the dimension of the position of the starting blink image at which the target object begins to blink in the target image sequence, the position of the starting blink image may be any position of the N images contained in the target image sequence. Therefore, there are N situations in this dimension. The electronic device can detect and obtain a third unimodal result containing word results corresponding to the N situations based on the eye areas in each image.

[0195] For example, if the target image sequence contains three images, there are three situations that the starting blink image is the first image, the second image, and the third image in the target image sequence. The electronic device can detect the confidence that the starting blink image is the first image in the target image sequence, the confidence that the starting blink image is the second image in the target image sequence, and the confidence that the starting blink image is the third image in the target image sequence based on the eye areas in each image, thereby obtaining a third unimodal result containing these three confidences.

[0196] Two implementation methods of obtaining the third single-modality result according to the eye region detection in each image are described below.

[0197] In the first implementation method, the electronic device can use the eye area in each image as input data of a pre-trained third classification model, and input it into the third classification model. The third classification model detects the confidence of various situations based on the eye area in each image and outputs each confidence. The electronic device can then obtain a third unimodal result containing each confidence.

[0198] In the second implementation, the electronic device may use a prediction algorithm, function, etc. to predict the confidence of various situations based on the eye area in each image, thereby obtaining a third unimodal result including the confidence of various situations.

[0199] 4. Fourth single mode results

[0200] The fourth unimodal result corresponds to the dimension of the position of the end-blink image of the target object ending the blink in the target image sequence.

[0201] This dimension is opposite to the dimension of the position of the starting blink image where the target object begins to blink in the target image sequence. In the target image sequence, if there is an ending blink image, and the later the position of the ending blink image is, the higher the possibility that the target object blinked during the above-mentioned image acquisition time period, and the more it can reflect the occurrence of a complete blink event; the earlier the position of the ending blink image is, the lower the possibility that the target object blinked during the above-mentioned image acquisition time period.

[0202] For example, if the second image in the target image sequence is a terminated blink image, the target object may have closed its eyes at the beginning of the above image acquisition time period. In this way, the above image acquisition time period does not include the complete blink action of the target object, which also reflects that the above complete blink event did not occur.

[0203] In the dimension of the position of the termination blink image of the target object ending the blink in the target image sequence, the position of the termination blink image may be any position of the N images contained in the target image sequence. Therefore, there are N situations in this dimension. The electronic device can detect and obtain a third unimodal result containing word results corresponding to the N situations based on the eye areas in each image.

[0204] For example, if the target image sequence contains three images, there are three situations that the terminal blink image is the first image, the second image, and the third image in the target image sequence. The electronic device can detect the confidence that the terminal blink image is the first image in the target image sequence, the confidence that the terminal blink image is the second image in the target image sequence, and the confidence that the terminal blink image is the third image in the target image sequence based on the eye areas in each image, thereby obtaining a fourth unimodal result containing these three confidences.

[0205] Two implementation methods of obtaining the fourth single-modality result according to the eye region detection in each image are described below.

[0206] In the first implementation method, the electronic device can use the eye area in each image as input data of a pre-trained fourth classification model, and input it into the fourth classification model. The fourth classification model detects the confidence of various situations based on the eye area in each image and outputs each confidence. The electronic device can then obtain a fourth unimodal result containing each confidence.

[0207] In a second implementation, the electronic device may use a prediction algorithm, function, etc. to predict the confidence of various situations based on the eye area in each image, thereby obtaining a fourth unimodal result including the confidence of various situations.

[0208] 5. Fifth single mode results

[0209] The fifth unimodal result corresponds to the dimension of the number of blink images contained in the target image sequence.

[0210] Blink images include images of the target object's eyes when they are fully closed and partially closed. The greater the number of blink images, the more likely the target object is to blink, and the more likely a complete blink event is to occur. The fewer the number of blink images, the more difficult it is to explain that the target object is blinking, and the less likely a complete blink event is to occur.

[0211] In addition, the number of blink images contained in the target image sequence can also reflect the individual blinking differences. If the number of blink images is large, it means that the target object blinks slowly. If the number of blink images is small, it means that the target object blinks quickly.

[0212] Similar to the dimension of the number of abnormal images contained in the aforementioned target image sequence, under the dimension of the number of blink images contained in the target image sequence, the number of blink images may be any integer in the range of zero to the total number N of images contained in the target image sequence. Therefore, there are N+1 situations under this dimension. The electronic device can detect and obtain a fifth unimodal result containing sub-results corresponding to the N+1 situations based on the eye areas in each image.

[0213] For example, if the target image sequence contains three images, there are four situations: the target image sequence does not contain a blink image, contains one blink image, contains two blink images, and contains three blink images. The electronic device can detect, based on the eye areas in each image, the confidence that the target image sequence does not contain a blink image, the confidence that it contains one blink image, the confidence that it contains two blink images, and the confidence that it contains three blink images, thereby obtaining a fifth unimodal result containing these four confidences.

[0214] Two implementation methods of obtaining the fifth single-modality result according to the eye region detection in each image are described below.

[0215] In the first implementation method, the electronic device can use the eye area in each image as input data of a pre-trained fifth classification model, and input it into the fifth classification model. The fifth classification model detects the confidence of various situations based on the eye area in each image and outputs each confidence. The electronic device can then obtain a fifth unimodal result containing each confidence.

[0216] In the second implementation, the electronic device may use a prediction algorithm, function, etc. to predict the confidence of various situations based on the eye area in each image, thereby obtaining a fifth unimodal result including the confidence of various situations.

[0217] 6. Sixth single-mode results

[0218] The sixth unimodal result corresponds to the blink phase event.

[0219] Blinking stage events include: events in which the target object blinks completely, events in which the target object blinks incompletely, and events in which the target object does not blink during the image acquisition period.

[0220] An incomplete blink of the target object means that the eyes of the target object are already closed at the start time of the above image acquisition time period, or the eyes of the target object are not fully opened at the end time. If the eyes of the target object are already closed at the start time, the above start blink image can be considered as the first image of the target image sequence, and if the eyes of the target object are not fully opened at the end time, the above end blink image can be considered as the last image of the target image sequence.

[0221] For the above three events, the electronic device detects the confidence of the occurrence of the three events based on the eye area in each image, and obtains a sixth unimodal result including the confidence corresponding to the three events.

[0222] Two implementation methods of obtaining the second single-modality result according to the eye region detection in each image are described below.

[0223] In the first implementation method, the electronic device can use the eye area in each image as input data of a pre-trained prediction model and input it into the prediction model. The prediction model predicts the confidence of the occurrence of the above three events based on the eye area in each image and outputs each confidence. The electronic device can then obtain a sixth unimodal result containing each confidence.

[0224] In the second implementation, the electronic device may use prediction algorithms, functions, etc. to predict the confidence of the occurrence of the above three events based on the eye area in each image, thereby obtaining a sixth unimodal result including the confidence of various situations.

[0225] Each of the above six single-modal results can accurately reflect whether a complete blink event occurs from various different dimensions, and the above multimodal result includes at least two of the above six single-modal results. In this way, the single-modal results in the multimodal result are fused to obtain a fusion result. The fusion result can comprehensively reflect whether a complete blink event occurs from multiple different dimensions. Therefore, based on the fusion result, it is possible to accurately determine whether the target object blinks while comprehensively considering multiple dimensions, thereby improving the accuracy of blink detection.

[0226] Two implementation methods 1 and 2 for obtaining multimodal results are described below.

[0227] Implementation 1:

[0228] Among the above-mentioned first classification model, second classification model, third classification model, fourth classification model, fifth classification model and prediction model, some or all of the models can be the same model, so that after the electronic device recognizes the eye area in each image, the eye area in each image can be used as input data and input into the same model to obtain multiple single-modal results output by the model.

[0229] See also Figure 4a , showing a schematic diagram of the process of obtaining multimodal results. Figure 4a In the present invention, the detection model may be a pre-trained model for detecting and obtaining multimodal results. The electronic device may The input data is input into the detection model to obtain a model output result, which may include at least two of the above six single-modal results.

[0230] The above N represents the total number of images contained in the target image sequence, and W and H represent the width and height of the eye area respectively. If the electronic device recognizes the area of ​​the two eyes of the target object, that is, the binocular area, the above input data is data.

[0231] See also Figure 4b , shows a schematic diagram of the structure of a detection model, which includes a Stem layer, multiple Stage layers, multiple downsampling layers, a pooling layer and a fully connected layer, wherein the specific structure of the Stem layer is as follows Figure 4c As shown, the specific structure of the downsampling layer is as follows Figure 4d As shown, the specific structure of the Stage layer is as follows Figure 4e shown.

[0232] When the electronic device takes the eye area in each image as input data and inputs it into the detection model, the Stem layer performs structural convolution processing on the input data and outputs a first processing result. The first Stage layer performs deep convolution processing on the first processing result and outputs a second processing result. The first downsampling layer performs downsampling processing on the second processing result to obtain a third processing result. The second Stage layer performs deep convolution processing on the third processing result and outputs a fourth processing result. The second downsampling layer performs downsampling processing on the fourth processing result and outputs a fifth processing result. The third Stage layer performs deep convolution processing on the fifth processing result and outputs a sixth processing result. The third downsampling layer performs downsampling processing on the sixth processing result and outputs a seventh processing result. The fourth Stage layer performs deep convolution processing on the seventh processing result and outputs an eighth processing result. The pooling layer performs pooling processing on the eighth processing result and outputs a ninth processing result. The fully connected layer performs fully connected processing on the ninth processing result and outputs a tenth processing result. The tenth processing result is the output result of the detection model.

[0233] The Stem layer, downsampling layer, and Stage layer are introduced below.

[0234] exist Figure 4c In the Stem layer, there are two types of convolutional layers and activation layers. The convolution kernel size in one of the convolutional layers is , the stride is 2, this convolutional layer can be called The convolution layer of another convolution layer has a convolution kernel size of , this convolutional layer can be called The FFN layer represents a feed-forward neural network layer constructed by multiple fully connected layers.

[0235] When the Stem layer performs structural convolution on the data to be processed, the data to be processed goes through a The convolutional layer, activation layer, and another The convolution layer performs convolution, activation, and reconvolution to obtain the first intermediate result as the output result of the FFN layer. The convolutional layer, activation layer, and another The convolution layer performs convolution, activation, and reconvolution again to obtain the second intermediate result. Finally, the Stem layer adds the first intermediate result and the second intermediate result to obtain the output result of this network layer.

[0236] exist Figure 4d In the example, the downsampling layer includes the RepViTBlock layer, Deep convolutional layers, The convolutional layer and FFN layer, where The convolution kernel size used in the depth convolution layer is , the stride is 2.

[0237] When the downsampling layer performs structural convolution on the data to be processed, the RepViTBlock layer first performs lightweight convolution on the data to be processed to obtain the third intermediate result, and then passes through Deep convolutional layers, The convolution layer and FFN layer perform deep convolution, convolution and multiple full connection processes on the third intermediate result in turn to obtain the output result of this network layer.

[0238] exist Figure 4e In the example, the Stage layer includes the RepViTSEBlock layer and the RepViTBlock layer, where the RepViTSEBlock layer includes Deep convolutional layers, Deep convolutional layer, SE layer and FFN layer, RepViTBlock layer contains Deep convolutional layers, Deep convolutional layers and FFN layers.

[0239] When the Stage layer performs structural convolution on the data to be processed, the RepViTSEBlock layer processes the data to be processed, specifically: The convolution process and The convolution process is performed, and the two processing results and the data to be processed are added to obtain the fourth intermediate result. The SE layer compresses and weights the fourth intermediate result to obtain the fifth intermediate result. The FFN layer performs multiple full connection processes on the fifth intermediate result to obtain the output result of the RepViTSEBlock layer. After that, the RepViTBlock layer processes the output result again, specifically: respectively performing The convolution process and The convolution processing is performed, and the two processing results and the output result are added to obtain a sixth intermediate result. The FFN layer then performs multiple full connection processing on the sixth intermediate result to obtain the output result of the RepViTBlock layer.

[0240] In one embodiment of the present application, the above-mentioned detection model can be constructed with a lightweight repvits model as a template to achieve a lightweight model effect, which is beneficial to the deployment of the detection model.

[0241] In addition, in the above Figure 4b Based on the model shown, the Stem layer contains The stride in the convolutional layer can also be set to 1 to widen the network, so that under the premise of achieving downsampling, the number of channels of data processed by each network layer in the model is increased, the overall amount of data information remains unchanged, image information loss is prevented, and the purpose of extracting small area features is achieved. In addition, the RepViTSEBlock layer and the RepViTBlock layer contain Deep convolutional layers, The deep convolutional layer can be transformed into one by using the reparameterization technique. Deep convolutional layers, such as Figure 4f As shown in the figure, this can not only ensure that the overall reasoning effect of the model remains unchanged, but also reduce the number of parameters of the overall model parameters, further realize the lightweight model, and is more conducive to the deployment of the detection model.

[0242] Implementation 2:

[0243] For each image in the target image sequence, a target parameter representing the degree of eye closure of the target object can be detected according to the eye region of the image. The higher the parameter value, the greater the degree of eye closure of the target object.

[0244] like Figure 5 As shown, a parameter value distribution diagram of the target parameter corresponding to each image in the target image sequence is shown. Among them, the horizontal axis is the image sequence number arranged according to the acquisition time, and the vertical axis is the parameter value of the target parameter. The parameter values ​​corresponding to the first to fifth images and the parameter values ​​corresponding to the eighth to tenth images are 0, indicating that the target object's eyes are not closed at the acquisition time of each image. The parameter value corresponding to the sixth image is the largest, indicating that the target object's eyes are completely closed at the acquisition time corresponding to the sixth image. The parameter value corresponding to the seventh image is between the maximum parameter value and zero, indicating that the target object's eyes are not completely closed at the acquisition time corresponding to the seventh image.

[0245] After detecting and obtaining the target parameters corresponding to each image, the electronic device can obtain the third to sixth single-modal results mentioned above according to the distribution of the target parameters.

[0246] The specific implementation method of fusing the single-modality results in the multi-modality results mentioned in the above step S303 is described below.

[0247] In one embodiment of the present application, each single-modality result in the multi-modality result includes multiple sub-results.

[0248] From the above introduction to the six unimodal results, it can be seen that each unimodal result corresponds to a dimension, and there are multiple situations in each dimension. When the electronic device performs detection based on the eye area in each image, it can detect the confidence levels corresponding to various situations in various dimensions, which are contained in the sub-results of the unimodal results corresponding to each dimension.

[0249] For example, the first unimodal result may include the confidence that the target object wears glasses and the confidence that the target object does not wear glasses, so the first unimodal result may include two sub-results, each sub-result including one confidence.

[0250] Other unimodal results are similar to the first unimodal result and also include multiple sub-results, each of which corresponds to the confidence level in one case.

[0251] In view of this, when fusing the unimodal results in the multimodal results, for each unimodal result in the multimodal results, the sub-results contained in the unimodal result can be fused to obtain an intermediate result; and then the intermediate results obtained can be fused to obtain a fused result.

[0252] The intermediate results corresponding to each of the above single-modal results can be regarded as prior information used for blink detection from one dimension. The fusion result combines the intermediate results and can be regarded as the fusion of multiple prior information, which is information used for blink detection from multiple dimensions.

[0253] Two implementation methods for fusing the sub-results contained in a single-modal result are described below.

[0254] In the first implementation, each sub-result included in the single-modal result can be normalized, each normalized sub-result can be multiplied by a preset result label value, each multiplication result can be accumulated, and an intermediate result can be obtained based on the accumulated result.

[0255] The above result label values ​​are pre-set numerical values, and each result label value is used to represent a situation. For example, 0 represents the situation that the user wears glasses, 1 represents the situation that the user does not wear glasses, 1 represents a complete blink event, 0.5 represents an event in which the target object does not blink completely, and 0 represents an event in which the target object does not blink.

[0256] Each sub-result contained in the unimodal result also corresponds to a situation. After normalizing the sub-results contained in the unimodal result, the confidence in the normalized sub-results corresponding to the same situation can be multiplied by the result label value, and the multiplication results can be accumulated to obtain the intermediate result based on the accumulated results.

[0257] The method of obtaining the intermediate result according to the accumulated result can be found in the subsequent embodiments and will not be described in detail here.

[0258] In this implementation, each result label value can be understood as a value representing a result, that is, a value representing a situation, and each sub-result in the unimodal result can reflect whether a complete blink event will occur in one dimension. Each normalized sub-result is multiplied by each structural label value, and each multiplication result is accumulated. This can fuse multiple situations in one dimension, and obtain an intermediate result based on the accumulated result. The intermediate result can comprehensively reflect whether a complete blink event occurs from multiple situations in one dimension. In this way, multiple intermediate results are fused to obtain a fused result, which can comprehensively reflect whether a complete blink event occurs from more situations in multiple dimensions. Therefore, based on the fused result, it is possible to more accurately determine whether the target object blinks.

[0259] In the second implementation, the average value or median of each sub-result included in the single-modal result may be calculated, and the calculated result may be used as an intermediate result.

[0260] After obtaining the intermediate results corresponding to each single modality result by using any of the above two implementation methods, the intermediate results can be multiplied or the average value of the intermediate results can be calculated to obtain a calculation result as the fusion result.

[0261] It can be seen from the above that when the scheme provided in the embodiment of the present application is used for blink detection, the sub-results contained in each single-modal result are fused to obtain an intermediate result. Each intermediate result can comprehensively reflect whether a complete blink event has occurred from multiple situations in one dimension. The intermediate results corresponding to each single-modal result are then fused, which can be understood as fusing multiple dimensions. The fused result obtained by the fusion can comprehensively reflect whether a complete blink event has occurred from more situations in multiple dimensions. Therefore, based on the fusion result, it is possible to more accurately determine whether the target object has blinked.

[0262] In one embodiment of the present application, when obtaining the fusion result, a Bayesian fusion method can be used to fuse the sub-results in each single-modal result to obtain an intermediate result, and then fuse the intermediate results to obtain a fusion result.

[0263] The following describes five implementation methods of obtaining intermediate results based on the accumulated results mentioned in the above embodiments.

[0264] In the first implementation, the intermediate result can be obtained by subtracting one from the accumulated result.

[0265] In the second implementation, the total number of images included in the target image sequence minus the accumulated result may be used to obtain an intermediate result.

[0266] In the third implementation, the accumulated result may be divided by the total number of images to obtain an intermediate result.

[0267] In a fourth implementation, the total number of images may be subtracted from the accumulated result, and the subtraction result may be divided by the total number of images to obtain an intermediate result.

[0268] In the fifth implementation, the accumulated result may be directly determined as the intermediate result.

[0269] By adopting any of the above five implementation methods, the intermediate results can be accurately obtained, and then the intermediate results are fused to obtain a fusion result. According to the fusion result, it can be accurately determined whether the target object blinks.

[0270] In one embodiment of the present application, there may be multiple unimodal results included in the multimodal results. For different unimodal results, different methods among the above five implementation methods may be used to obtain intermediate results, or the same method may be used to obtain the intermediate results.

[0271] See also Figure 6 , shows a result fusion process, which includes six branch processes, corresponding to the six single-modal results mentioned above.

[0272] In the first branch process, each sub-result included in the first single-modal result is normalized (step S601), each sub-result after normalization is multiplied by the preset result label value, each multiplication result is accumulated (step S602), and then the accumulated result is subtracted from one to obtain the first intermediate result (step S603).

[0273] Among them, each result label value in this branch process includes a preset label value of the target object wearing glasses and a result label value of the target object not wearing glasses.

[0274] In the second branch process, each sub-result included in the second single-modal result is normalized (step S604), each sub-result after normalization is multiplied by the preset result label value, each multiplication result is accumulated (step S605), and then the accumulated result is subtracted from one to obtain a second intermediate result (step S606).

[0275] Among them, each result label value in this branch process includes the label value of the situation corresponding to each preset number of abnormal images.

[0276] In the third branch process, each sub-result contained in the third single-modal result is normalized (step S607), each sub-result after normalization is multiplied by the preset result label value, each multiplication result is accumulated (step S608), and then the accumulated result is divided by the total number of images to obtain a third intermediate result (step S609).

[0277] Among them, each result label value in this branch process includes a preset label value corresponding to each image position.

[0278] In the fourth branch process, each sub-result included in the fourth single-modal result is normalized (step S610), and each sub-result after normalization is multiplied by the preset result label value, and each multiplication result is accumulated (step S611), and then the accumulated result is subtracted from the total number of images, and the subtraction result is divided by the total number of images to obtain a fourth intermediate result (step S612).

[0279] Among them, each result label value in this branch process includes a preset label value corresponding to each image position.

[0280] In the fifth branch process, each sub-result contained in the fifth single-modal result is normalized (step S613), each sub-result after normalization is multiplied by the preset result label value, each multiplication result is accumulated (step S614), and then the accumulated result is divided by the total number of images to obtain the fifth intermediate result (step S615).

[0281] Among them, each result label value in this branch process includes a label value corresponding to a preset number of blink images.

[0282] In the sixth branch process, each sub-result included in the sixth single-modal result is normalized (step S616), each sub-result after normalization is multiplied by the preset result label value, each multiplication result is accumulated (step S617), and the accumulated result is directly determined as the sixth intermediate result (step S618).

[0283] Among them, each result label value in this branch process includes label values ​​corresponding to three preset events.

[0284] After obtaining the first to sixth intermediate results, the first to sixth intermediate results may be fused to obtain a fused result (step S619 ).

[0285] When a multimodal result is obtained during detection, in addition to detecting based on the eye region in the image, detection may also be performed in combination with the eye region and other information.

[0286] In one embodiment of the present application, the electronic device can detect the posture of the target object at the time of acquisition of each image based on each image included in the target image sequence, obtain the posture information corresponding to each image, and when a multimodal result is obtained through detection, based on the eye area in each image and the posture information corresponding to each image, obtain a multimodal result reflecting the occurrence of a complete blink event from multiple dimensions.

[0287] Specifically, for each image in the target image sequence, multiple key points of various parts of the face of the target object in the image may be detected, and the posture information of the target object may be obtained based on the detected key points of various parts.

[0288] For example, the key points of the two eyes, the tip of the nose and the two corners of the mouth of the target object in the image can be detected to obtain five key points, and based on these five key points, the PNP five-point detection method is used to detect the posture information of the target object.

[0289] After the electronic device detects and obtains the posture information corresponding to each image, it can jointly detect and obtain the single-modal results corresponding to each dimension based on the eye area in each image and the posture information corresponding to the image, thereby obtaining a multimodal result corresponding to each single-modal result.

[0290] If the aforementioned Figure 4a The detection model shown in FIG. 1 can detect the eye area in each image and the posture information corresponding to the image, and the size of The input data is input into the detection model to obtain the model output result.

[0291] From the above, it can be seen that when the solution provided in the embodiment of the present application is applied to perform blink detection, when the multimodal results are detected, not only the eye area of ​​each image is detected, but also the posture information of the target object corresponding to each image is detected. This can increase the information on which the detection is based, improve the accuracy of the multimodal results, overcome the interference of the detection due to the deformation of the eye area in the posture of the target object lowering or raising the head, thereby improving the accuracy of blink detection.

[0292] According to the fusion result, the method provided in the following embodiment may be used to determine whether the target object blinks.

[0293] In one embodiment of the present application, the electronic device may obtain a fusion result corresponding to an auxiliary image sequence adjacent to the target image sequence, and determine whether the target object blinks according to the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence.

[0294] The relationship between the target image sequence and the auxiliary image sequence is introduced below.

[0295] The image acquisition device can continuously acquire images of the target object, and the electronic device can acquire multiple continuous images from the images acquired by the image acquisition device as an image sequence, and reacquire the multiple continuous images in a sliding manner as another image sequence.

[0296] For example, if the image sequence needs to include 5 images, when the image acquisition device has acquired 5 images, the electronic device can regard these 5 images as an image sequence, and each time the image acquisition device subsequently acquires an image, the electronic device can regard the 5 images most recently acquired by the image acquisition device as an image sequence. For example, the first to fifth images acquired by the image acquisition device are the first image sequence, the second to sixth images are the second image sequence, and the third to seventh images are the third image sequence.

[0297] When the electronic device acquires multiple images to form an image sequence, it can also extract an image every other image or multiple images. When the number of extracted images reaches a required number, such as a preset number, the extracted images are regarded as an image sequence.

[0298] In one embodiment of the present application, the target image sequence and the auxiliary image sequence meet at least one of the following three conditions:

[0299] The first condition: the starting image in each image sequence is an image acquired sequentially by the image acquisition device;

[0300] The second condition: the images contained in each set of image sequences are: images continuously acquired by the image acquisition device;

[0301] The third condition: the number of images contained in each image sequence is the same.

[0302] If the target image sequence and the auxiliary image sequence meet the first condition, then when determining the auxiliary image sequence, the electronic device can quickly determine that multiple image sequences with the shortest distance between the starting image in the sequence and the starting image in the target image sequence are multiple auxiliary image sequences, thereby quickly obtaining the fusion results corresponding to the auxiliary image sequences. It can be seen that if the target image sequence and the auxiliary image sequence meet the first condition, it helps to speed up the efficiency of blink detection.

[0303] If the target image sequence and the auxiliary image sequence meet the second condition, that is, the images contained in each image sequence are continuously acquired images, the information loss caused by the interval selection of images in the sequence can be avoided, and the accuracy of the fusion result corresponding to the image sequence can be improved, thereby improving the accuracy of blink detection.

[0304] If the target image sequence and the auxiliary image sequence meet the third condition, each image sequence contains the same number of images. Therefore, the fusion results corresponding to each image sequence are obtained based on the same number of images, which can make the fusion results corresponding to each image sequence more standardized. In this way, determining whether the target object blinks based on the fusion results corresponding to multiple image sequences can improve the reliability of blink detection.

[0305] The method of obtaining the fusion result corresponding to the auxiliary image sequence can be referred to the aforementioned embodiment, which will not be described again here.

[0306] After obtaining the fusion results corresponding to the target image sequence and the fusion results corresponding to the auxiliary image sequence, it is possible to determine whether the target object has blinked based on the two fusion results. In this way, the fusion results corresponding to multiple image sequences are considered when determining whether the target object has blinked, and the information considered is richer, so it is possible to more accurately determine whether the target object has blinked. In addition, compared with determining whether the target object has blinked based only on the fusion results corresponding to the target image sequence, determining based on the fusion results corresponding to multiple image sequences can reduce the false alarm rate, which can improve the user experience in scenarios where users use electronic devices or drive vehicles.

[0307] Two implementation methods of determining whether the target object blinks according to the above two fusion results are described below.

[0308] In the first implementation method, the variance of the fusion results corresponding to the target image sequence and the fusion results corresponding to the auxiliary image sequence and the sum of the fusion results are calculated; if the calculated variance is greater than a first preset threshold, the sum of the fusion results is greater than a second preset threshold, the fusion result corresponding to the target image sequence is greater than a third preset threshold, and the distance between the target image sequence and the image sequence corresponding to the target object's last blink is greater than a preset distance, it is determined that the target object blinked during the image acquisition time period corresponding to the target image sequence.

[0309] The first preset threshold, the second preset threshold, the third preset threshold and the preset distance are all preset parameters.

[0310] The distance between the target image sequence and the image sequence corresponding to the target object's last blink may be represented by the distance between the starting images in the two image sequences, the distance between the ending images, or the distance between images at the same sorting position in the sequences.

[0311] Specifically, the variance of the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence reflects the difference between the two fusion results. If the variance is greater than the first preset threshold, it means that the difference between the two fusion results is large. The large difference between the two fusion results means that one of the fusion results reflects that the target object blinks, and the other fusion result reflects that the target object does not blink. If the variance is not greater than the first preset threshold, it means that the two fusion results may not be very different, and the two fusion results may both reflect that the target object blinks, or may both reflect that the target object does not blink.

[0312] The sum of the fusion results of the fusion results corresponding to the target image sequence and the fusion results corresponding to the auxiliary image sequence represents whether the fusion results are normal values. If the sum of the fusion results is greater than the second preset threshold, it means that both fusion results are normal values ​​and can be used for blink detection; if the sum of the fusion results is not greater than the second preset threshold, it means that there may be abnormal fusion results in the two fusion results, and the reliability of using these two fusion results for blink detection is low.

[0313] If the fusion result corresponding to the target image sequence is greater than the third preset threshold, it means that from the perspective of the information contained in the target image sequence itself, the target object is more likely to blink during the image acquisition time period corresponding to the target image sequence; if the fusion result corresponding to the target image sequence is not greater than the third preset threshold, it means that from the perspective of the information contained in the target image sequence itself, the target object is less likely to blink during the image acquisition time period corresponding to the target image sequence.

[0314] If the distance between the target image sequence and the image sequence corresponding to the target object's last blink is not greater than the preset distance, it means that the complete blink events within the image acquisition time period corresponding to the two image sequences are the same blink event; if the above distance is greater than the preset distance, it means that the complete blink events within the image acquisition time period corresponding to the two image sequences are two independent blink events.

[0315] When the variance is greater than the first preset threshold, the sum of the fusion results is greater than the second preset threshold, the fusion result corresponding to the target image sequence is greater than the third preset threshold, and the above distance is greater than the preset distance, it can be reliably proved that the target object blinked during the image acquisition time period corresponding to the target image sequence, that is, a complete blink event occurred. It can be seen that the use of this implementation method can accurately determine whether the target object blinks, thereby improving the accuracy of blink detection.

[0316] In the second implementation method, according to the fusion results corresponding to the target image sequence and the fusion results corresponding to the auxiliary image sequence, a peak detection algorithm can be used to detect whether the fusion result corresponding to the target image sequence is at a peak position in the distribution of fusion results corresponding to each image sequence. If so, it is determined that the target object blinked during the image acquisition time period corresponding to the target image sequence, that is, a complete blink event occurred.

[0317] In one embodiment of the present application, see Figure 7 , a flow chart of a second blink detection method is provided. In this embodiment, the method includes the following steps S701-S709.

[0318] Step S701: performing facial recognition on each image included in the target image sequence.

[0319] Step S702: When each image contains image contents of multiple objects, determine the facial region of the target object from the facial regions contained in each image.

[0320] Step S703: performing facial alignment processing on the facial area of ​​the target object contained in each image.

[0321] Step S704: performing image preprocessing on the aligned image.

[0322] Step S705: Determine the eye region of the target object in the facial region of the target object contained in each preprocessed image.

[0323] For the description of the above steps S701 to S705, please refer to the above explanation of step S301, which will not be repeated here.

[0324] Step S706: According to each image included in the target image sequence, the posture of the target object at the time of acquisition of each image is detected to obtain posture information corresponding to each image.

[0325] Step S707: Based on the eye regions and posture information corresponding to each image, a multimodal result reflecting the occurrence of a complete blink event from multiple dimensions is detected.

[0326] The description of the above steps S706 and S707 can be found in the above embodiments and will not be repeated here.

[0327] Step S708: Fusing the single-modality results in the multi-modality results to obtain a fusion result.

[0328] For the description of this step, please refer to the aforementioned step S303, which will not be repeated here.

[0329] Step S709: obtaining a fusion result corresponding to an auxiliary image sequence adjacent to the target image sequence, and determining whether the target object blinks according to the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence.

[0330] The description of this step can be found in the above-mentioned embodiment and will not be repeated here.

[0331] In a specific implementation, the present application further provides a computer storage medium, wherein the computer storage medium may store a program, wherein when the program is executed, the device where the computer readable storage medium is located is controlled to execute some or all of the steps in the above embodiment. The above storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0332] In a specific implementation, an embodiment of the present application further provides a computer program product, which includes executable instructions. When the executable instructions are executed on an electronic device, the electronic device executes part or all of the steps in the above method embodiment.

[0333] In a specific implementation, an embodiment of the present application further provides an electronic device, including: one or more processors and a memory;

[0334] The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the blink detection method provided in the embodiment of the present application.

[0335] like Figure 8 As shown, the present application also provides a chip system, which is applied to electronic devices. The chip system includes one or more processors 801. The processor 801 is used to call computer instructions so that the electronic device inputs the data to be processed into the chip system. The chip system performs processing based on the blink detection method provided in the embodiment of the present application.

[0336] In a possible implementation, the chip system also includes input and output interfaces for inputting and outputting data.

[0337] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0338] Program code may be applied to input instructions to perform the functions described herein and to generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0339] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0340] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories, random access memories, erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0341] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0342] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0343] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0344] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A blink detection method, characterized in that: The method comprises: Identify the eye region of the target object in each image included in the target image sequence; According to the eye area in each image, a multimodal result is detected to reflect whether a complete blinking event occurs from multiple dimensions, wherein the complete blinking event is: the target object blinks within the image acquisition time period corresponding to the target image sequence, each single-modal result in the multimodal result includes multiple sub-results, each single-modal result corresponds to a dimension, and there are multiple situations under each dimension. The sub-results included in the single-modal result corresponding to each dimension are: confidence levels corresponding to various situations in various dimensions; fusing the single modality results in the multimodality results to obtain a fusion result; Determining whether the target object blinks according to the fusion result; The fusing of the single-modality results in the multi-modality results to obtain a fusion result includes: For each single-modal result in the multi-modal result, each sub-result included in the single-modal result is merged to obtain an intermediate result; The intermediate results are fused to obtain a fusion result, which comprehensively reflects whether a complete blink event occurs; The sub-results included in the single-modality result are fused to obtain an intermediate result, including: Normalizing each sub-result included in the single-modal result; Multiply each normalized sub-result by the preset result label value; Accumulate the multiplication results; Obtaining intermediate results based on the accumulated results; The multimodal result includes at least two of the following single-modal results: A first single-modal result reflecting the occurrence of a complete blink event from the dimension of whether the target object wears glasses; A second single modality result reflecting the occurrence of a complete blink event from the dimension of the number of abnormal images contained in the target image sequence; A third single-modality result reflecting the occurrence of a complete blink event, which is a dimension of a position of a start blink image of the target object in the target image sequence; A fourth single-modal result reflecting the occurrence of a complete blink event from the dimension of the position of the ending blink image of the target object in the target image sequence; A fifth single modality result reflecting the occurrence of a complete blink event from the dimension of the number of blink images included in the target image sequence; From the dimension of blink stage events, a sixth unimodal result reflecting the occurrence of complete blink events is provided, wherein blink stage events include: events in which the target object blinks completely within the image acquisition time period, events in which the target object blinks incompletely, and events in which the target object does not blink.

2. The method according to claim 1, characterized in that The step of obtaining an intermediate result according to the accumulated result includes: Subtract the accumulated result from one to obtain an intermediate result; and / or Subtract the accumulated result from the total number of images included in the target image sequence to obtain an intermediate result; and / or Dividing the accumulated result by the total number of images to obtain an intermediate result; and / or Subtracting the accumulated result from the total number of images, and dividing the subtraction result by the total number of images to obtain an intermediate result; and / or The accumulated result is determined as an intermediate result.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: According to each image included in the target image sequence, detecting the posture of the target object at the time of acquisition of each image, and obtaining posture information corresponding to each image; The detection of the eye regions in each image to obtain a multimodal result reflecting the occurrence of a complete blink event from multiple dimensions includes: According to the eye regions in each image and the posture information corresponding to each image, a multimodal result reflecting the occurrence of a complete blink event from multiple dimensions is detected.

4. The method according to claim 1 or 2, characterized in that: Determining whether the target object blinks according to the fusion result includes: Acquire a fusion result corresponding to an auxiliary image sequence adjacent to the target image sequence; Whether the target object blinks is determined according to a fusion result corresponding to the target image sequence and a fusion result corresponding to the auxiliary image sequence.

5. The method according to claim 4, characterized in that The target image sequence and the auxiliary image sequence satisfy at least one of the following conditions: The starting image in each image sequence is an image acquired sequentially by the image acquisition device; The images included in each set of image sequences are: images continuously acquired by the image acquisition device; Each image sequence contains the same number of images.

6. The method according to claim 4, characterized in that The determining whether the target object blinks according to the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence includes: Calculating the variance of the fusion result corresponding to the target image sequence and the fusion result corresponding to the auxiliary image sequence and the sum of the fusion results; If the calculated variance is greater than a first preset threshold, the sum of the fusion results is greater than a second preset threshold, the fusion result corresponding to the target image sequence is greater than a third preset threshold, and the distance between the target image sequence and the image sequence corresponding to the target object's last blink is greater than a preset distance, it is determined that the target object blinked within the image acquisition time period corresponding to the target image sequence.

7. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when being executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 6.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device inputs data into the chip system and executes the method described in any one of claims 1 to 6 to perform blink detection.

Images