A head action-based visual stimulus reaction monitoring method and intelligent terminal

Abstract

This invention discloses a method for monitoring visual stimulus responses based on head movements. The method includes: acquiring image or video data containing head images; inputting the image or video data into a head detection network model for training to obtain a head detection bounding box; inputting the head detection bounding box into a head keypoint localization model for training to obtain the coordinates of head keypoints; and obtaining the direction of the visual stimulus and the visual stimulus response based on the coordinates of the head keypoints. This invention combines a simple test task to apply stimuli to subjects, monitors head movement parameters, and assesses the subjects' cognitive state based on the head movement parameters. By using a deep learning-based head feature point localization and tracking method, it reduces hardware requirements and increases the versatility of visual stimulus response monitoring.

Description

Technical Field

[0001] This invention relates to the field of visual stimulus response monitoring based on head movements and a smart terminal. Background Technology

[0002] Visual stimulus response monitoring has wide applications in fields such as psychology, neuroscience, and infant and toddler research. In clinical research, responses to visual stimuli can be used to diagnose ophthalmic diseases and brain and neurological disorders, such as autism and Parkinson's disease. It can also provide early warning signs of some diseases, such as Alzheimer's and Parkinson's. Specific parameter data allows for the objective quantification of disease or rehabilitation progress research.

[0003] Currently, monitoring user responses to visual stimuli received in front of computers or mobile devices primarily employs eye-tracking methods. This involves analyzing the user's cognitive function by monitoring eye movements during reading or task completion. However, eye trackers used for eye analysis are expensive and inconvenient to carry. Alternatively, methods using a single built-in camera can partially replace eye trackers, but these require high clarity and resolution in the captured images or videos. Furthermore, they are unsuitable for poor lighting conditions such as backlighting or strong light, or when the eyes are not fully captured in the image. Additionally, the elderly population, who have higher needs for cognitive function assessment, generally have a smaller eye opening range than younger people, all of which make single-camera-based eye tracking difficult to implement and thus hinder the monitoring of visual stimulus responses.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and intelligent terminal for monitoring visual stimulus response based on head movements, in order to address the above-mentioned deficiencies of the prior art. The aim is to solve the problems of high implementation cost of eye movement detection and difficulty in implementing eye movement tracking based on a single camera, which makes it difficult to monitor visual stimulus response.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a method for monitoring visual stimulus response based on head movements, wherein the method includes:

[0008] Acquire image or video data containing head images;

[0009] The image or video data is input into a head detection network model for training to obtain head detection bounding boxes.

[0010] The head detection bounding box is input into the head key point localization model for training to obtain the coordinates of the head key points.

[0011] Based on the coordinates of the key points on the head, the direction of the visual stimulus and the visual stimulus response are obtained.

[0012] In one implementation, the step of inputting the image or video data into a head detection network model for training to obtain a head detection bounding box includes:

[0013] The head image data is input into a head detection network model for training to obtain detection results;

[0014] The detection results are subjected to nonmaximum suppression post-processing to obtain the head detection rectangle.

[0015] In one implementation, the head detection network model is a MobileFaceNet model; wherein the MobileFaceNet model includes a first separable convolutional module and a second separable convolutional module; the first separable convolutional module includes a 5×5 convolutional kernel for shallow layers of the network; the second separable convolutional module includes two 5×5 convolutional kernels for deep layers of the network.

[0016] In one implementation, the step of inputting the head detection bounding box into a head keypoint localization model for training to obtain the head keypoint position coordinates includes:

[0017] The head image within the head detection bounding box is input into the MobileMeshNet network model; wherein, the MobileMeshNet network model includes 2D convolutional layers and residual modules; the residual module includes two 3×3 2D convolutional layers; the kernel size of the 2D convolutional layers is 3×3;

[0018] The head image is trained using a downsampled residual module to obtain the dependencies between key point features;

[0019] Based on the dependencies between the key point features, and by using a convolutional layer-residual module-convolutional layer header to decode the head image, the coordinates of the head key points are obtained.

[0020] In one implementation, obtaining the visual stimulus direction and visual stimulus response based on the coordinates of the head key points includes:

[0021] Based on the coordinates of the key points on the head, the head position, the coordinates of each feature point on the head, and the relative depth estimate of each feature point on the head are obtained.

[0022] Based on the head position, the coordinates of each feature point on the head, and the relative depth estimation of each feature point on the head, the head motion parameters are obtained.

[0023] Based on the head movement parameters, the direction of the visual stimulus and the visual stimulus response are obtained.

[0024] In one implementation, the head motion parameters include head orientation, head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, head movement acceleration, and reaction time.

[0025] In one implementation, obtaining the visual stimulus direction and visual stimulus response based on the head movement parameters includes: obtaining the visual stimulus response based on the head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, and head movement acceleration.

[0026] Based on the head orientation, the direction of the visual stimulus;

[0027] The visual stimulus response is obtained based on the head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, and head movement acceleration.

[0028] Secondly, embodiments of the present invention also provide a visual stimulus response monitoring device based on head movements, wherein the device includes:

[0029] The image acquisition module is used to acquire image or video data containing head images;

[0030] The head detection rectangle acquisition module is used to input the image or video data into the head detection network model for training to obtain the head detection rectangle.

[0031] The head key point position coordinate acquisition module is used to input the head detection rectangle into the head key point localization model for training, and obtain the head key point position coordinates.

[0032] The visual stimulus response acquisition module is used to obtain the visual stimulus response based on the head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, and head movement acceleration.

[0033] Thirdly, embodiments of the present invention also provide a smart terminal, wherein the smart terminal includes a memory, a processor, and a head-movement-based visual stimulus response monitoring program stored in the memory and executable on the processor, wherein when the processor executes the head-movement-based visual stimulus response monitoring program, it implements the steps of the head-movement-based visual stimulus response monitoring method as described in any of the above claims.

[0034] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a visual stimulus response monitoring program based on head movements, and when the visual stimulus response monitoring program based on head movements is executed by a processor, it implements the steps of the visual stimulus response monitoring method based on head movements as described in any of the above claims.

[0035] Beneficial Effects: Compared with existing technologies, this invention provides a method and intelligent terminal for monitoring visual stimulus responses based on head movements. First, image or video data containing head images is acquired, enabling monitoring of head movements across cross-platform devices and networks using their built-in cameras, with low hardware requirements. Then, the image or video data is input into a head detection network model for training, resulting in head detection bounding boxes. This allows for intelligent detection of the bounding boxes containing head images using a neural network model. Next, the head detection bounding boxes are input into a head keypoint localization model for training, obtaining the coordinates of head keypoints. This deep learning-based head feature point localization and tracking method reduces environmental requirements and increases the versatility of visual stimulus response monitoring. Finally, based on the coordinates of the head keypoints, the direction and response to the visual stimulus are obtained. This can be combined with simple test tasks to monitor head movement parameters, and the cognitive state of the subject can be assessed based on these parameters. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the visual stimulus response monitoring method based on head movements provided in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the module for separating convolution provided in an embodiment of the present invention.

[0039] Figure 3This is a schematic diagram of the MobileFaceNet network structure provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the MobileMeshNet network structure provided in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of the visual stimulus response monitoring device based on head movements provided in an embodiment of the present invention.

[0042] Figure 6 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0044] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0046] Visual stimulus response monitoring has wide applications in fields such as psychology, neuroscience, and infant and toddler research. In clinical research, responses to visual stimuli can be used to diagnose ophthalmic diseases and brain and neurological disorders, such as autism and Parkinson's disease. It can also provide early warning signs of some diseases, such as Alzheimer's and Parkinson's. Currently, eye tracking is the most commonly used method for monitoring user visual information. However, eye trackers are expensive, and in some extreme cases, the difficulty in capturing eye characteristics can lead to inaccurate identification of visual stimulus responses.

[0047] Currently, the correlation between head movements and eye movements has been verified. Frank et al. designed a reading experiment using infrared pupil and head trackers to monitor head and eye movements, demonstrating that the head and eye movement system is a highly coupled yet extremely flexible system. Therefore, the head movement monitoring proposed in this invention can effectively serve as an alternative to eye tracking, enabling the monitoring of visual stimulus responses, facilitating a series of human-computer interactions, and achieving the assessment of psychological and cognitive states. This invention can be implemented on cross-platform devices and, based on simple tests using some cognitive psychology methods, analyzes the head movement behavior of subjects when stimuli are actively provided. In various application scenarios, it is not only cost-effective but also easier to operate.

[0048] Exemplary methods

[0049] This embodiment provides a method for monitoring visual stimulus responses based on head movements. For example... Figure 1 As shown, the method includes the following steps:

[0050] Step S100: Obtain image or video data containing head images;

[0051] Specifically, this embodiment utilizes a camera to acquire images or videos containing head images. The camera can be a network camera with remote control and transmission capabilities, a local camera device, or a medical image acquisition device. The image should contain a complete head image, and each frame containing a complete head image is extracted from the video data as source material. If the head image is missing more than a loss threshold, the image or the video frame should be discarded. In this embodiment, the loss threshold is set to 70%.

[0052] Step S200: Input the image or video data into the head detection network model for training to obtain head detection bounding boxes;

[0053] Specifically, in this embodiment, we will perform head detection on the acquired image or video data. A neural network monitoring model is trained on the input image or video data to intelligently extract head detection bounding boxes. Each head detection bounding box is a rectangular image containing a complete head image. These bounding boxes can then be used to further locate and analyze the head image.

[0054] In one implementation, step S200 of this embodiment includes the following steps:

[0055] Step S201: Input the head image data into the head detection network model for training to obtain the detection result; the head detection network model is the MobileFaceNet model; wherein, the MobileFaceNet model includes a first separable convolutional module and a second separable convolutional module; the first separable convolutional module includes a 5×5 convolutional kernel for the shallow layers of the network; the second separable convolutional module includes two 5×5 convolutional kernels for the deep layers of the network.

[0056] Step S202: Perform nonmaximum suppression post-processing on the detection results to obtain the head detection rectangle.

[0057] Specifically, this embodiment proposes a head detection solution called MobileFaceNet, such as... Figure 2 As shown, it is a lightweight and high-performance head detector based on MobileNet, specifically tailored for mobile inference. Its ultra-real-time performance allows it to be applied to any real-time viewfinder that requires accurate regions of interest (ROIs) of the head as input for the next head keypoint estimation task. Figure 2 In the diagram, the left side shows a single-layer FaceBlock, while the right side shows a double-layer DoubleFaceBlock with a rising-falling configuration. We use 5×5 convolutional kernels to increase the network's receptive field size while reducing network depth, thus optimizing real-time inference speed.

[0058] FaceBlock is suitable for shallow layers of the network, while DoubleFaceblock is suitable for deep layers. We configure the optimized depthwise separating convolutional module in the SSD detection network. On the other hand, compared to the SSD detection model, this embodiment uses two feature dimensions (16*16, 8*8) for anchor learning, resulting in a lighter model, fewer candidate boxes, and less computation on the GPU. We replace the original two 2×2, 4×4, and 8×8 feature maps with six 8×8 feature maps to improve the default anchor mechanism. The specific network architecture is as follows: Figure 3 As shown. In Figure 3 In this model, the backbone network is composed of stacked depthwise separable convolutional modules FaceBlock and DoubleFaceBlock, and candidate boxes are proposed by the last two outputs.

[0059] Step S300: Input the head detection rectangle into the head key point localization model for training to obtain the coordinates of the head key points.

[0060] Specifically, this embodiment employs a deep learning model, namely a head keypoint localization model, to infer the 3D head surface. It only requires the head detection bounding box as input, without the need for a dedicated depth sensor. The head keypoint localization model is bundled with the head transformation module to establish a metric 3D space, and the head transformation within this space is estimated using the head marker screen position.

[0061] In one implementation, step S300 of this embodiment includes the following steps:

[0062] Step S301: Input the head image in the head detection rectangle into the MobileMeshNet network model; wherein, the MobileMeshNet network model includes 2D convolutional layers and residual modules; the residual module includes two 3×3 2D convolutional layers; the kernel size of the 2D convolutional layers is 3×3;

[0063] Step S302: Train the head image using a downsampled residual module to obtain the dependencies between key point features;

[0064] Step S303: Based on the dependencies between the key point features, and using the head of the convolutional layer-residual module-convolutional layer to decode the head image, the coordinates of the head key points are obtained.

[0065] Specifically, this embodiment uses a deep learning model to infer the 3D head surface, requiring only the head captured by the camera as input, without the need for a dedicated depth sensor. MobileMeshNet is bundled with a head transformation module to establish a metric 3D space, and the head's position on the screen is used to estimate the head transformation within this space. Specifically, the head detected by the head detection model is used as input, and a 3D head keypoint model approximating the 3D surface is predicted through regression. MobileMeshNet uses more downsampling at the network input, dedicating most of the computation to its shallow layers, allowing the neuron receptive fields to begin covering a large area of ​​the input image relatively early, thus better capturing the positional information and relative positions of different keypoints, converting them to coordinates only in the last few layers of the network.

[0066] Specifically, in Figure 4 In MobileMeshNet, an asymmetric network structure is adopted. The backbone network uses more downsampled residual modules to capture the dependencies between keypoint features, and then decodes the head keypoint coordinates through a convolutional layer-residual module-convolutional layer head. The model input is the head image in the head detection bounding box of MobileFaceNet, and the output is the head keypoint coordinates. Specifically, the kernel size of the 2D convolutional layer (Convolutional2DLayer, Conv2D) is 3×3, the stride is set to 1, and the padding size is set to 1. The residual module is a standard residual module, consisting of two stacked 3×3 Conv2D convolutional layers, to achieve residual learning of y = F(x) + . MaxPooling2DLayer (MaxPool2D) is used on the connections, with a pooling size of 2×2.

[0067] Step S400: Based on the coordinates of the key points on the head, obtain the direction of the visual stimulus and the response to the visual stimulus.

[0068] Specifically, in this embodiment, a simple test task is combined to apply stimuli to the subject and monitor the head image of the subject when responding to the stimulus. This allows the coordinates of key head points to be obtained. By calculating the changes in the coordinates of key head points, the parameter values ​​representing head movement can be inferred, thereby obtaining the subject's cognitive state, namely the direction of visual stimulus and the response to visual stimulus.

[0069] For example, if the subject's head key point coordinates move from (x1, y1, z1) to (x2, y2, z2) in 0.01 seconds during the first test, compared to the subject's head key point coordinates moving from (x1, y1, z1) to (x2, y2, z2) in 0.1 seconds during the second test, it can be inferred that the two visual stimuli are in the same direction, but the visual stimulus responses are different, with the visual stimulus response being stronger in the first test.

[0070] In one implementation, step S400 of this embodiment includes the following steps:

[0071] Step S401: Based on the coordinates of the head key points, obtain the head position, the coordinates of each feature point of the head, and the relative depth estimate of each feature point of the head;

[0072] Step S402: Based on the head position, the coordinates of each feature point on the head, and the relative depth estimation of each feature point on the head, obtain the head motion parameters; the head motion parameters include head orientation, head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, head movement acceleration, and reaction time.

[0073] Step S403: Based on the head movement parameters, obtain the visual stimulus direction and visual stimulus response.

[0074] Specifically, when a visual stimulus occurs, the eyes follow the target's movement, simultaneously causing the head to shift in a specific direction. The trajectory of the head movement can be determined by the head's orientation, direction of rotation, speed of rotation, direction of movement, speed of movement, trajectory area, acceleration, and reaction time. By observing the trajectory of the head movement, the subject's current eye-tracking status can be determined. Thus, even in environments without eye trackers, monitoring head movements can reveal the subject's visual stimulus direction and response.

[0075] Specifically, this embodiment determines the head shape estimation by using the coordinates of key head points. By using these key point coordinates, the head position, the coordinates of each head feature point, and the relative depth estimation of each feature point are accurately determined. Based on these parameters, head movements are analyzed, including the current head orientation, head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, area of ​​the head movement trajectory, head movement acceleration, and reaction time. The specific calculation method is as follows:

[0076] Head orientation

[0077] The head movement speed is

[0078] Head movement acceleration is

[0079] The direction of head movement is

[0080] The direction of head rotation is

[0081] The head rotation speed is

[0082] in, Let x be the x-coordinate of the vector from the center of the head to the tip of the nose. Let x0 be the x-coordinate of the head's center point, and y0 and z0 be the y-coordinate and vertical coordinates, respectively. Let k0 be the slope from the center of the head to the tip of the nose on the xoy plane, k1 be the original state, and Δt be the time interval.

[0083] Specifically, even when image or video quality is poor and eye movements cannot be accurately tracked, the method in this embodiment can still acquire the head movement state, thereby maintaining high accuracy, high robustness, and low hardware requirements. It eliminates the need for infrared or high-definition cameras required in previous methods, and only requires analysis of the subject's head movement behavior to detect the direction of visual stimuli and the response to visual stimuli.

[0084] In one implementation, step S403 of this embodiment includes the following steps:

[0085] Step S4031: Determine the direction of the visual stimulus based on the head orientation;

[0086] Step S4032: Obtain the visual stimulus response based on the head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, and head movement acceleration.

[0087] Specifically, when a visual stimulus occurs to the subject, the head orientation and the direction of gaze are in the same direction, so the direction of the visual stimulus can be inferred from the head orientation. Simultaneously, the direction and speed of head rotation reflect how quickly the subject's head rotates when facing the visual stimulus, reflecting the speed of the subject's local head reaction. The direction, speed, trajectory, area, and acceleration of head movement reflect the subject's body movement trajectory and speed, and can also infer the subject's response to changes in visual stimuli.

[0088] It should be noted that, in order to enable the method in this embodiment to run on cross-platform devices and networks, this embodiment uses TensorFlow.js to convert the deep learning model into a JSON file of the model structure and a BIN file of the model weights. This supports the online running of the model on cross-platform devices and networks such as browsers, allowing eye-tracking analysis to migrate to the scalable platform of web browsers so that it can run on mobile terminals, and actively promoting the development of online and real-time head tracking.

[0089] Exemplary device

[0090] like Figure 5 As shown in the illustration, this embodiment also provides a visual stimulus response monitoring device based on head movements, the device comprising:

[0091] Image acquisition module 10 is used to acquire image or video data containing head images;

[0092] The head detection rectangle acquisition module 20 is used to input the image or video data into the head detection network model for training to obtain the head detection rectangle.

[0093] The head key point position coordinate acquisition module 30 is used to input the head detection rectangle into the head key point localization model for training to obtain the head key point position coordinates.

[0094] The visual stimulus response acquisition module 40 is used to obtain the visual stimulus direction and visual stimulus response based on the coordinates of the key points on the head.

[0095] In one implementation, the head detection rectangle acquisition module 20 includes:

[0096] The first training acquisition unit is used to input the head image data into the head detection network model for training to obtain the detection result; the head detection network model is the MobileFaceNet model; wherein, the MobileFaceNet model includes a first separable convolutional module and a second separable convolutional module; the first separable convolutional module includes a 5×5 convolutional kernel for the shallow layers of the network; the second separable convolutional module includes two 5×5 convolutional kernels for the deep layers of the network.

[0097] The head detection bounding box acquisition unit is used to perform nonmaximum suppression post-processing on the detection results to obtain the head detection bounding box.

[0098] In one implementation, the head key point position coordinate acquisition module 30 includes:

[0099] The second training acquisition unit is used to input the head image in the head detection rectangle into the MobileMeshNet network model; wherein, the MobileMeshNet network model includes 2D convolutional layers and residual modules; the residual module includes two 3×3 2D convolutional layers; the kernel size of the 2D convolutional layers is 3×3;

[0100] The dependency acquisition unit is used to train the head image using a downsampled residual module to obtain the dependencies between key point features;

[0101] The head key point location coordinate acquisition unit is used to decode the head image based on the dependency relationship between the key point features and using the head of the convolutional layer-residual module-convolutional layer to obtain the head key point location coordinates.

[0102] In one implementation, the visual stimulus response acquisition module 40 includes:

[0103] The head coordinate acquisition unit is used to obtain the head position, the coordinates of each feature point of the head, and the relative depth estimate of each feature point of the head based on the coordinates of the head key points.

[0104] The head motion parameter acquisition unit is used to obtain head motion parameters based on the head position, the coordinates of each feature point of the head, and the relative depth estimation of each feature point of the head. The head motion parameters include head orientation, head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, head movement acceleration, and reaction time.

[0105] The visual stimulus direction and visual stimulus response acquisition unit is used to obtain the visual stimulus direction and visual stimulus response based on the head movement parameters.

[0106] In one implementation, the visual stimulus direction and visual stimulus response acquisition unit of this embodiment includes:

[0107] A visual stimulus direction acquisition subunit is used to determine the visual stimulus direction based on the head orientation.

[0108] The visual stimulus response acquisition subunit is used to obtain the visual stimulus response based on the head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, and head movement acceleration.

[0109] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 6 As shown, the smart terminal includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for monitoring visual stimulus responses based on head movements. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed within the smart terminal to detect the operating temperature of internal devices.

[0110] Those skilled in the art will understand that Figure 6 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0111] In one embodiment, a smart terminal is provided, comprising a memory, a processor, and a head-movement-based visual stimulus response monitoring program stored in the memory and executable on the processor. When the processor executes the head-movement-based visual stimulus response monitoring program, it implements the following operation instructions:

[0112] Acquire image or video data containing head images;

[0113] The image or video data is input into a head detection network model for training to obtain head detection bounding boxes.

[0114] The head detection bounding box is input into the head key point localization model for training to obtain the coordinates of the head key points.

[0115] Based on the coordinates of the key points on the head, the direction of the visual stimulus and the visual stimulus response are obtained.

[0116] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, operational databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operating data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0117] In summary, this invention discloses a method for monitoring visual stimulus responses based on head movements. The method includes: acquiring image or video data containing head images; inputting the image or video data into a head detection network model for training to obtain a head detection bounding box; inputting the head detection bounding box into a head keypoint localization model for training to obtain the coordinates of head keypoints; and obtaining the direction of the visual stimulus and the visual stimulus response based on the coordinates of the head keypoints. This invention combines a simple test task to apply stimuli to the subject, monitors head movement parameters, and assesses the subject's cognitive state based on the head movement parameters. By using a deep learning-based head feature point localization and tracking method, it reduces environmental requirements and increases the versatility of visual stimulus response monitoring.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring visual stimulus responses based on head movements, characterized in that, The method includes: Acquire image or video data containing head images; The image or video data is input into a head detection network model for training to obtain head detection bounding boxes. The head detection bounding box is input into the head key point localization model for training to obtain the coordinates of the head key points. Based on the coordinates of the key points on the head, the direction of the visual stimulus and the visual stimulus response are obtained; The step of inputting the image or video data into a head detection network model for training to obtain a head detection bounding box includes: The head image data is input into a head detection network model for training to obtain detection results. The head detection network model is a MobileFaceNet model. The MobileFaceNet model includes a first separable convolutional module and a second separable convolutional module. The first separable convolutional module includes a 5×5 convolutional kernel for shallow layers of the network; the second separable convolutional module includes two 5×5 convolutional kernels for deeper layers of the network. The MobileFaceNet model uses feature maps with two feature dimensions (16×16 and 8×8) for candidate box learning and six 8×8 feature maps for extracting relevant features from the candidate boxes. The detection results are post-processed with nonmaximum suppression to obtain the head detection bounding box; The step of inputting the head detection bounding box into the head key point localization model for training to obtain the head key point position coordinates includes: The head image within the head detection bounding box is input into the MobileMeshNet network model; wherein, the MobileMeshNet network model includes 2D convolutional layers and residual modules; the residual module includes two 3×3 2D convolutional layers; the kernel size of the 2D convolutional layers is 3×3; the MobileMeshNet network model adopts an asymmetric network structure, and uses max pooling with a pooling size of 2×2 in the connections; The head image is trained using a downsampled residual module to obtain the dependencies between key point features; Based on the dependencies between the key point features, and by using a convolutional layer-residual module-convolutional layer header to decode the head image, the coordinates of the head key points are obtained.

2. The method for monitoring visual stimulus response based on head movements according to claim 1, characterized in that, The process of obtaining the direction of visual stimulation and the visual stimulus response based on the coordinates of the key head points includes: Based on the coordinates of the key points on the head, the head position, the coordinates of each feature point on the head, and the relative depth estimate of each feature point on the head are obtained. Based on the head position, the coordinates of each feature point on the head, and the relative depth estimation of each feature point on the head, the head motion parameters are obtained. Based on the head movement parameters, the direction of the visual stimulus and the visual stimulus response are obtained.

3. The method for monitoring visual stimulus response based on head movements according to claim 2, characterized in that, The head movement parameters include head orientation, head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, head movement acceleration, and reaction time.

4. The method for monitoring visual stimulus response based on head movements according to claim 3, characterized in that, The process of obtaining the visual stimulus direction and visual stimulus response based on the head movement parameters includes: Based on the head orientation, the direction of the visual stimulus; The visual stimulus response is obtained based on the head rotation direction, head rotation speed, head movement direction, head movement speed, head movement trajectory, head movement trajectory area, and head movement acceleration.

5. A visual stimulus response monitoring device based on head movements, characterized in that, The apparatus, used to implement the head-movement-based visual stimulus response monitoring method as described in any one of claims 1-4, comprises: The image acquisition module is used to acquire image or video data containing head images; The head detection rectangle acquisition module is used to input the image or video data into the head detection network model for training to obtain the head detection rectangle. The head key point position coordinate acquisition module is used to input the head detection rectangle into the head key point localization model for training, and obtain the head key point position coordinates. The visual stimulus response acquisition module is used to obtain the visual stimulus direction and visual stimulus response based on the coordinates of the key points on the head.

6. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a head-movement-based visual stimulus response monitoring program stored in the memory and executable on the processor. When the processor executes the head-movement-based visual stimulus response monitoring program, it implements the steps of the head-movement-based visual stimulus response monitoring method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a visual stimulus response monitoring program based on head movements, which, when executed by a processor, implements the steps of the visual stimulus response monitoring method based on head movements as described in any one of claims 1-4.

Images