A method and system for recording graphics drawing process information

Abstract

The present invention discloses a method and system for recording information of a graphic drawing process, and relates to the technical field of recording graphic drawing processes. The method includes: obtaining process information of a subject drawing a graphic and a facial video of the subject; determining the gaze features of the subject during the drawing process based on the facial video of the subject; extracting features from the process information to obtain the graphic features and process features of the subject; splicing the graphic features, process features, and gaze features to obtain the graphic drawing features of the subject; inputting the graphic drawing features into a trained neural network model to predict the cognitive impairment results of the subject. Compared with traditional tests, the present invention can completely preserve information during the graphic drawing process, providing a strong basis for doctors' diagnoses; in large-scale assessment scenarios such as community screening, it can provide technical support for testing large-scale populations using scales containing complex graphic drawing.

Description

Technical Field

[0001] The present invention relates to the technical field of recording graphic drawing process, and in particular to a method and system for recording graphic drawing process information. Background Art

[0002] A wealth of useful information can be obtained from the drawing of complex graphs. For example, various scales, including those used clinically, often feature items with complex graphs. These scales are widely popular with both physicians and test subjects due to their scientific rationality and ease of use. With the help of scales, physicians can assess the patient's condition and status, assisting with further diagnosis and treatment. This is particularly true when diagnosing brain and mental illnesses, which often lack organic lesions in the early and middle stages. In these cases, the results of the scales serve as an important basis for the physician's diagnosis.

[0003] Moreover, with the help of the scale, people can also conduct self-examinations before seeking medical treatment and have a general understanding of their own conditions. At present, the scoring and judgment of complex graphic drawing mainly rely on professional doctors. However, relying solely on the judgment of doctors also has certain limitations. For example, the judgment of doctors will be affected by subjective factors to a certain extent. Moreover, based on the current situation of relatively scarce doctor resources, in application scenarios such as community screening, assessments involving complex graphics are difficult to implement widely. And more importantly, in traditional tests, the information of the test subject during the graphic drawing process is difficult to retain. What is left is only the final drawing, and a considerable amount of information that can be used for auxiliary diagnosis is lost.

[0004] Therefore, there are currently some practical problems in complex graphic drawing tests: first, there is a high degree of dependence on doctors, which makes it difficult to widely implement complex graphic-based evaluations in application scenarios such as community screening; more importantly, in the traditional complex graphic drawing test process, apart from direct observation by doctors, there are no more ways to collect information about the graphic drawing process. What is left is only the final drawn graphic, and a considerable amount of information that can be used for auxiliary diagnosis is lost. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for recording information of the graphics drawing process. In hospital scenarios, compared with traditional tests, it can completely preserve the information in the graphics drawing process, providing a strong basis for the doctor's diagnosis; in large-scale evaluation scenarios such as community screening, it can provide technical support for testing large-scale populations using scales involving complex graphics drawing.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for recording graphics drawing process information, the method comprising:

[0008] Obtaining process information of the subject drawing the figure and a facial video of the subject; the process information includes the starting position of each stroke, the stopping position of each stroke, the starting time of each stroke, the stopping time of each stroke, and the length of the trajectory of each stroke;

[0009] Determining the gaze characteristics of the subject during the drawing process based on the facial video of the subject; the gaze characteristics include the number of gaze focus switches, the frequency of gaze focus switches, the drawing time, and the time of observing the sample drawing;

[0010] Extracting features from the process information to obtain graphic features and process features of the subject;

[0011] splicing the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the subject;

[0012] The graphic drawing features are input into a trained neural network model to predict the cognitive impairment result of the subject; the cognitive impairment result includes whether cognitive impairment exists and the level of cognitive impairment.

[0013] Optionally, the training process of the neural network model includes:

[0014] Obtaining process information of the tester when drawing the graphics and the tester's facial video;

[0015] determining, based on the tester's facial video, gaze features during the tester's drawing process;

[0016] Performing feature extraction on the process information to obtain graphic features and process features of the tester;

[0017] splicing the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the tester;

[0018] Determining the cognitive impairment result of the test subject according to the test subject's score on the scale;

[0019] The neural network model is trained with the test subject's graphic drawing features as input and the test subject's cognitive impairment results as output to obtain a trained neural network model.

[0020] Optionally, determining the gaze features of the subject during the drawing process based on the facial video of the subject specifically includes:

[0021] Extracting a single frame image from the facial video of the subject;

[0022] Applying the YOLO target detection algorithm to the single-frame image to obtain an eye picture of the subject with eyes open;

[0023] Applying a convolutional neural network to analyze the eye image of the open eyes to obtain eye positioning features;

[0024] Determining the current behavior of the subject according to the eye positioning feature; the current behavior includes looking at a sample picture and drawing;

[0025] Recording the time when the subject looks at the sample picture and the time when the subject draws the picture;

[0026] The gaze feature is determined according to the time of viewing the sample image and the time of drawing the image.

[0027] Optionally, extracting features from the process information to obtain graphic features and process features of the subject specifically includes:

[0028] Divide the process information into segments according to set time lengths, count the strokes, thinking time, and drawing time in each time segment, and obtain initial process features;

[0029] Extracting initial graphic features based on the graphic drawn by the subject; the graphic features include graphic area, graphic position, graphic drawing time, graphic thinking time and stroke length;

[0030] Dimensionality reduction is performed on the initial process features and the initial graphic features respectively to obtain graphic features and process features of the subject.

[0031] A system for recording graphics drawing process information, applied to the above-mentioned method for recording graphics drawing process information, the system comprising:

[0032] An acquisition module is used to obtain process information of the subject drawing the figure and the subject's facial video; the process information includes the starting position of each stroke, the stopping position of each stroke, the starting time of each stroke, the stopping time of each stroke and the length of each stroke;

[0033] A gaze feature determination module is used to determine the gaze features of the subject during the drawing process based on the facial video of the subject; the gaze features include the number of gaze focus switches, the frequency of gaze focus switches, the drawing time, and the time of observing the sample drawing;

[0034] A feature extraction module is used to extract features from the process information to obtain graphic features and process features of the subject;

[0035] a splicing module, configured to splice the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the subject;

[0036] The prediction module is used to input the graphic drawing features into the trained neural network model to predict the cognitive impairment results of the subject; the cognitive impairment results include whether cognitive impairment exists and the level of cognitive impairment.

[0037] Optionally, the system further includes a training module; the training module includes:

[0038] The acquisition submodule is used to obtain the process information of the tester when drawing the figure and the tester's facial video;

[0039] A gaze feature determination submodule, configured to determine the gaze features of the tester during the drawing process based on the tester's facial video;

[0040] A feature extraction submodule, configured to extract features from the process information to obtain graphic features and process features of the tester;

[0041] a splicing submodule, configured to splice the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the tester;

[0042] A result determination submodule, for determining the cognitive impairment result of the test subject according to the test subject's score on the scale;

[0043] The training submodule is used to train the neural network model with the test subject's graphic drawing features as input and the test subject's cognitive impairment results as output to obtain a trained neural network model.

[0044] Optionally, the gaze feature determination module includes:

[0045] A single-frame image extraction submodule is used to extract a single-frame image from the facial video of the subject;

[0046] an eye image determination submodule, configured to apply a YOLO target detection algorithm to the single-frame image to obtain an eye image of the subject with their eyes open;

[0047] an analysis submodule, configured to analyze the open eye image using a convolutional neural network to obtain eye positioning features;

[0048] A behavior determination submodule is used to determine the current behavior of the subject according to the eye positioning feature; the current behavior includes looking at sample pictures and drawing;

[0049] A recording submodule, used to record the time when the subject looks at the sample picture and the time when the subject draws the picture;

[0050] The gaze feature determination submodule is used to determine the gaze feature according to the time of viewing the sample image and the time of drawing the image.

[0051] Optionally, the feature extraction module includes:

[0052] The segmentation submodule is used to segment the process information according to the set time length, count the strokes, thinking time and drawing time in each time period, and obtain the initial process characteristics;

[0053] An initial graphic feature extraction submodule is used to extract initial graphic features based on the graphic drawn by the subject; the graphic features include graphic area, graphic position, graphic drawing time, graphic thinking time and stroke length;

[0054] The dimensionality reduction submodule is used to perform dimensionality reduction on the initial process features and the initial graphic features respectively to obtain the graphic features and process features of the subject.

[0055] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0056] The present invention uses a tablet computer and a smart stylus as testing tools. When the tester draws a complex graphic, it not only records the graphic drawn, but also records all the process information when the drawer draws the graphic, including the position of each stroke, the time and length of each stroke, and the time spent thinking when drawing the graphic. In addition, with the help of the camera on the tablet, the gaze position of the testee when drawing the graphic can also be recorded, so as to better evaluate the testee's visual space and spatial imagination ability. Therefore, when this method is applied in hospital scenarios, compared with traditional testing methods, it can better preserve the information in the graphic drawing process and provide analysis suggestions to provide effective assistance to doctors when making diagnoses. When this method is applied to large-scale evaluation scenarios such as community screening, it can provide technical support for the use of complex graphic drawing tests in large-scale populations. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A flowchart of a method for recording graphics drawing process information provided by the present invention;

[0059] Figure 2 A workflow diagram for recording graphic drawing process information provided by the present invention;

[0060] Figure 3A schematic diagram of the implementation process of the method for recording information about the graphic drawing process;

[0061] Figure 4 Schematic diagram of boundary points;

[0062] Figure 5 Schematic diagram of a specific implementation of the video analysis method;

[0063] Figure 6 Flowchart for human eye focus detection;

[0064] Figure 7 This is a schematic diagram of the drawing time and the sample drawing viewing time;

[0065] Figure 8 Draw a workflow diagram for recording process information by functional area;

[0066] Figure 9 Schematic diagram of the implementation process of the method for recording graphically drawn process information by functional area;

[0067] Figure 10 Draw interface diagrams for complex graphics;

[0068] Figure 11 Draw a diagram of the process for the pentagon;

[0069] Figure 12 Draw a process diagram for the clock;

[0070] Figure 13 A diagram of the complex graphical results drawn for the test subject;

[0071] Figure 14 This is a module diagram of a system for recording graphics drawing process information provided by the present invention.

[0072] Explanation of symbols:

[0073] 1-Acquisition module, 2-Gaze ​​feature determination module, 3-Feature extraction module, 4-Splicing module, 5-Prediction module. DETAILED DESCRIPTION

[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0075] The purpose of the present invention is to provide a method and system for recording information of the graphics drawing process. In hospital scenarios, compared with traditional tests, it can completely preserve the information in the graphics drawing process, providing a strong basis for the doctor's diagnosis; in large-scale evaluation scenarios such as community screening, it can provide technical support for testing large-scale populations using scales involving complex graphics drawing.

[0076] The hardware facilities of the present invention are mainly composed of a tablet computer with a camera and a stylus.

[0077] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] like Figure 1 、 Figure 2 and Figure 3 As shown, the present invention provides a method for recording graphics drawing process information, the method comprising:

[0079] Step S1: Obtain process information of the subject drawing a graphic and a facial video of the subject; the process information includes the starting position of each stroke, the stopping position of each stroke, the starting time of each stroke, the stopping time of each stroke, and the trajectory length of each stroke.

[0080] In actual applications, the test subjects use a tablet computer and a stylus to complete complex graphic drawing tasks. During drawing, the system records process information in real time, including the starting and ending positions, start and end times, trajectory length, etc. of each stroke; the camera simultaneously records the test subject's facial video data during the drawing process; the process information also includes information that can be directly and indirectly obtained by drawing on the tablet with a stylus.

[0081] Specifically, the starting and ending position information of each stroke of the stylus is represented by coordinate points, where (x0, y0) represents the starting coordinates and (x1, y1) represents the ending coordinates.

[0082] Where x0 and x1 represent the number of horizontal pixels from the horizontal coordinates of the starting point and the end point to the origin (the origin is set at the upper left corner of the screen); y0 and y1 represent the number of vertical pixels from the vertical coordinates of the starting point and the end point to the origin (the origin is in the upper left corner of the screen).

[0083] Based on the start and end position information, the length of each stroke can be obtained. Based on the high sampling rate of the stylus, with the help of the idea of ​​differentiation, the n points sampled during the handwriting are recorded, and each two adjacent points are connected into a line segment. These end-to-end line segments are used to approximate the curved handwriting. Then the length L of this handwriting can also be obtained by the line segment l between these adjacent sampling points. i Approximate by the sum.

[0084] in,

[0085] The length of this stroke can be obtained, that is: L = l1 + l2 + ... + l n-1 (2)

[0086] Based on the start and end positions and stroke distance, we can calculate:

[0087] (1) The longest and shortest stroke lengths.

[0088] After the drawing is completed, all stroke lengths are calculated according to formula (1) and formula (2), where the longest stroke length is recorded as the longest stroke length and the shortest stroke length is recorded as the shortest stroke length.

[0089] (2) Average stroke length

[0090] Sum up the lengths of all strokes and divide by the total number of strokes N to get the average stroke length As shown in formula (3).

[0091]

[0092] (3) Minimum circle area and minimum rectangle area

[0093] From all the collected points (x0,y0),(x1,y1),...,(x N ,y N ) in the coordinates of the graph, and then add the corresponding y to find the right and left boundary points of the graph. Similarly, the upper and lower boundary points of the graph can also be found. According to these four boundary points (x up ,y up ),(x down ,y down ),(x right ,y right ),(x left ,y left ), we can get the minimum circle of the entire figure (any three points that are not on the same line can determine a circle, in this case, we only need to ensure that the fourth point is inside the circle) and the minimum rectangle, such as Figure 4 As shown in the figure, the solid points are boundary points, and the dashed lines correspond to the smallest circle and the smallest rectangle, respectively. Based on the boundary points, substituting them into the area formulas for circles and rectangles, we can find the areas of the smallest circle and the smallest rectangle.

[0094] (4) Graphic center coordinates

[0095] According to the four boundary points (x up ,y up ),(x down ,ydown ),(x right ,y right ),(x left ,y left ), we can get the coordinates of the center of the graphic,

[0096] In addition, the start and end time information of each stroke sampled by the stylus is represented by t0 and t1, respectively. Among them, t0 and t1 are the time when the stylus starts to touch the screen and the time when the stylus leaves the screen after drawing this continuous stroke. The first time the stylus touches the screen is recorded as time 0, and the unit is ms (milliseconds); based on the start and end time information of each stroke, time-related features, that is, process features, can be obtained. Specifically, they include:

[0097] (1) Number of strokes

[0098] The stylus's "touch and release" action is captured and recorded, which reflects the number of strokes used in the drawing process. From this, the total number of strokes and the number of strokes per minute can be calculated.

[0099] (2) Stroke drawing time

[0100] The drawing time t of one stroke j The time t when the stroke passes the last point jn and the time t of the first point of the stroke j1 Assume that the stroke has n sampling points, then t j =t jn -t j1 .

[0101] (3) Thinking (delay) time

[0102] Thinking time between two adjacent strokes tt i =t i+1,1 -t i,n , we can use the time t corresponding to the first point of the next stroke i+1,1 Subtract the time t corresponding to the last point of the previous stroke i,n Then tt i =t i+1,1 -t i,n .

[0103] It is also possible to obtain the longest thinking time and the shortest thinking time by screening all the thinking times, average all the thinking times to obtain the average thinking time, and sum up all the thinking times to obtain the total thinking time.

[0104] (4) Drawing speed

[0105] According to the position and time of drawing strokes during the graphic drawing process, The characteristics of the drawing speed can be obtained; the characteristics of the drawing speed include the fastest stroke speed, the slowest stroke speed, the average drawing speed, and the longest stroke drawing speed.

[0106] Step S2: Determine the gaze characteristics of the subject during the drawing process based on the facial video of the subject; the gaze characteristics include the number of gaze focus switches, the frequency of gaze focus switches, the drawing time, and the time for observing the sample picture; specifically, the parameters obtained and used include but are not limited to the drawing time, the shortest single drawing time, the longest single drawing time, the average drawing time per time, the total sample picture viewing time, the longest single sample picture viewing time, the shortest single sample picture viewing time, the average sample picture viewing time per time, the number of switches, etc.

[0107] In practical applications, such as Figure 5 and Figure 6 As shown, the video captured by the camera is analyzed by the video processing module. Through this module, relevant parameters such as the time the subject gazes at the sample image, the number of times the gaze switches between the sample image and the drawing area, the longest single viewing time of the sample image, and the longest single drawing time can be analyzed. The process of analysis by the video processing module is the specific step of S2. If the subject blinks while drawing a complex image (which is almost inevitable), it is difficult to obtain the eye positioning characteristics at this time, and the gaze state at this time is defined as the same as the previous moment. (If the previous moment was also in the eyes-closed state, the gaze state is postponed until the previous detectable eye-opening moment is found).

[0108] S2 specifically includes:

[0109] Step S21: extracting a single frame image from the facial video of the subject.

[0110] Step S22: Apply the YOLO target detection algorithm to the single-frame image to obtain an eye picture of the subject with open eyes; in addition, applying the YOLO target detection algorithm to the single-frame image can also obtain an eye picture of the subject with closed eyes; when the eye picture with open eyes is obtained based on the eye picture, it is determined whether the subject is looking at a sample picture or drawing at this time; when the eye picture with closed eyes is obtained based on the eye picture, it is considered that the behavior of the subject at this time is consistent with the behavior at the previous moment.

[0111] Step S23: applying a convolutional neural network to analyze the eye image with the eyes open to obtain eyeball positioning features.

[0112] In practical applications, these include but are not limited to methods such as using YOLO to analyze videos to capture eye positions, using CNN to analyze eyes in videos to determine eye gaze positions (or visual focus), and automatically obtaining changes in eye gaze positions (including the number of gaze position changes, the time interval between two adjacent changes, and other information).

[0113] Step S24: determining the current behavior of the subject according to the eye positioning feature; the current behavior includes viewing sample images and drawing.

[0114] Step S25: Record the time when the subject looks at the sample picture and the time when the subject draws the picture; specifically, after analyzing the entire video, a list of the positions of the gaze focus and the time when each gaze focus is at that position can be obtained, from which it can be determined whether the subject is looking at the sample picture or drawing at each moment, thereby obtaining the time when the subject looks at the sample picture and the time when the subject draws the picture.

[0115] Step S26: determining the gaze feature according to the time of viewing the sample image and the time of drawing the image.

[0116] As a specific implementation method, the time of viewing the sample image is recorded as t s The drawing time is t d When t s and t d When they are adjacent, it is considered as a switch of the focus between the sample area and the drawing area. s t d It means that the focus of attention is shifted from the sample area to the drawing area. d t s It means that the focus of attention is shifted from the sample area to the drawing area. s t d and the nearest t after it d t s The time interval between the two switches is the continuous gaze time of the drawing area. The time between the two switches is the continuous gaze time of the sample image / drawing time. Similarly, t d t s and the nearest t after it s t d The shortest time interval between them is the time of continuous fixation on the sample image area. Based on this, the features related to the switching of gaze focus can be obtained, that is, the gaze features.

[0117] (1) Number and frequency of gaze focus switching

[0118] t s t d , t d t sEach of these represents a switch in gaze focus. By counting all switch actions, we can get the number of gaze switches. Based on the recorded time of gaze focus switches, we can get features such as gaze switch frequency.

[0119] (2) Drawing time and sample drawing review time

[0120] like Figure 7 As shown, the dotted box is t s t d , that is, when the focus of the eyes shifts from the sample area to the drawing area, the framed t d t s , represents the moment when the focus of attention shifts from the sample area to the drawing area. During this period, the drawing time is t D =t5-t2. The time for observing the sample image is t S =t n -t5.

[0121] Sum up all the drawing times to get the total drawing time. Sum up all the observation sample time to get the total observation sample time From drawing time Find the longest one, which is the longest continuous drawing time, and observe the sample drawing time. Find the longest and shortest among them, which are the longest continuous observation time and the shortest continuous observation time.

[0122] Step S3: extracting features from the process information to obtain graphic features and process features of the subject.

[0123] S3 specifically includes:

[0124] Step S31: segmenting the process information according to the set time length, counting the strokes, thinking time and drawing time in each time period, and obtaining the initial process features.

[0125] In practical applications, process features include dividing the entire process into time segments and counting the strokes, thinking time, drawing time, etc. in each time segment. Specifically, the information in the complex graphics drawing process can include more types, as well as new features or factors extracted based on this information.

[0126] Step S32: extracting initial graphic features based on the graphic drawn by the subject; the graphic features include graphic area, graphic position, graphic drawing time, graphic thinking time and stroke length.

[0127] In practical applications, graphic features are extracted from the final drawn graphic, such as its area, position, drawing time, thinking time, and stroke length. Methods for extracting complex graphic features include, but are not limited to, methods for capturing key points and angles such as pentagon edges and intersections in the pentagon replication test; methods for capturing the hour and minute hands and measuring their lengths in the digital clock drawing test; and methods for measuring the area, position, thinking time, and drawing time of the graphic. These methods also include methods for extracting graphic features from commonly used complex graphic drawing tests.

[0128] Step S33: performing dimensionality reduction on the initial process features and the initial graphic features respectively to obtain the graphic features and process features of the subject.

[0129] Step S4: The graphic features, the process features, and the gaze features are combined to obtain the subject's graphic drawing features. Specifically, the graphic features and the process features are closely related. After combining these two parts, a complex graphic drawing feature is obtained. The complex graphic drawing feature is then combined with the gaze features to obtain the subject's graphic drawing feature.

[0130] Step S5: inputting the graphic drawing features into the trained neural network model to predict the cognitive impairment result of the subject; the cognitive impairment result includes whether there is cognitive impairment and the level of cognitive impairment.

[0131] As a specific implementation, the neural network model may adopt BP neural network, decision tree, random forest, SVM, K-nearest neighbor and other models.

[0132] In addition, the method for recording graphics drawing process information provided by the present invention further includes:

[0133] Step S0: training the neural network model; specifically, the training process of the neural network model includes:

[0134] Step S01: Obtain process information of the tester when drawing a graphic and the tester's facial video.

[0135] Step S02: determining the gaze features of the tester during the drawing process based on the tester's facial video.

[0136] Step S03: extracting features from the process information to obtain the tester's graphic features and process features.

[0137] Step S04: combining the graphic features, the process features and the gaze features to obtain the graphic drawing features of the tester.

[0138] Step S05: Determine the cognitive impairment result of the test subject based on the test subject's score on the scale; specifically, a professional doctor makes a judgment based on current clinical diagnostic criteria and gives advice.

[0139] Step S06: using the test subject's graphic drawing features as input and the test subject's cognitive impairment results as output, training the neural network model to obtain a trained neural network model.

[0140] During the execution of each step, step S31 and step S32 can be performed in parallel; step S2 (ie, the video processing module) and the trajectory (handwriting) analysis module composed of step S31, step S32 and step S33 are performed in parallel and synchronously.

[0141] From a functional perspective, Figure 8 and Figure 9 As shown, the system can be divided into two main modules: a video processing module and a trajectory (handwriting) analysis module. The trajectory analysis module is the core of these two main modules and includes the contents of steps S31, S32, and S33. The video analysis module mainly contains the contents of step S2.

[0142] Step S2 captures and stores images, as well as analyzes and processes the captured video. The following parameters can be obtained: total sample viewing time, maximum single sample viewing time, minimum single sample viewing time, average sample viewing time per session, number of switching times, etc. By combining these parameters with the information obtained by the trajectory analysis module, we can better analyze the tester's comprehensive brain abilities, including spatial imagery, visual-spatial ability, and short-term memory. This enriches the model's content and improves its scientific nature and interpretability.

[0143] Step S31 extracts graphic features including the area of ​​the graphic, the area of ​​the minimum circumscribed circle, the area of ​​the minimum circumscribed rectangle, the position of the graphic, etc. based on the drawn graphic and special position points, such as the rightmost (left) point, the top (bottom) point, the center point of the graphic, etc.

[0144] Taking the pentagonal cross graphic copying task as an example, the implementation process of the method for recording graphic drawing process information provided by the present invention is as follows:

[0145] like Figure 10As shown in the figure, the lower right corner is the option to turn on the camera. The subject can choose whether to turn on the camera according to his / her wishes. After that, the subject needs to use the stylus to complete the drawing of complex graphics in the drawing area according to the example picture (the two pentagons in the upper right corner). Before drawing, turn on the start button and camera switch on the right, and the system will start recording the subject's complex graphics drawing (including the drawing handwriting and the facial information collected by the camera during the drawing process). The status of the pentagonal cross graphic copying task during the drawing process is as follows Figure 11 As shown, the status of the clock's copy task during the drawing process is as follows Figure 12 As shown in the figure, the complex graphics and related information drawn are imported into the system and analyzed in the imported model.

[0146] The original size of the image of a pentagon copied by a certain subject is 1024*640. It should be noted that the subject should be asked to draw in the drawing area and not be allowed to draw other graphics except for copying the pentagon to avoid interference. Figure 13 shown.

[0147] The method for recording graphics drawing process information provided by this invention is a multimodal fusion-based process detection method for complex graphics drawing, designed to address the recognition problem of complex graphics. This method uses process data from complex graphics drawing to provide more useful information for complex graphics recognition. It also innovatively employs multimodal fusion to refine the model and proposes a series of strategies to ensure the accuracy and speed of the method, thereby improving the recognition of complex graphics.

[0148] Compared to existing traditional complex graphic drawing tests, the method for recording graphic drawing process information provided by this invention offers advantages such as speed and convenience. It can be widely used for complex graphic drawing recognition in places where the elderly gather, such as communities and nursing homes. In hospital settings, it can serve as a powerful assistant to doctors, providing objective evidence for their diagnoses and reducing their workload. When the evaluator draws complex graphics for self-examination, the system can assess the evaluator's drawings and offer suggestions. This provides a reference for self-examination and serves as a valuable assistant to doctors.

[0149] like Figure 14 As shown, a system for recording graphics drawing process information, the system comprising:

[0150] Acquisition module 1 is used to obtain process information of the subject when drawing a figure and the facial video of the subject; the process information includes the starting position of each stroke, the stopping position of each stroke, the starting time of each stroke, the stopping time of each stroke and the trajectory length of each stroke.

[0151] The gaze feature determination module 2 is used to determine the gaze features of the subject during the drawing process based on the facial video of the subject; the gaze features include the number of gaze focus switches, the frequency of gaze focus switches, the drawing time, and the time of observing the sample picture.

[0152] The feature extraction module 3 is used to extract features from the process information to obtain graphic features and process features of the subject.

[0153] The splicing module 4 is used to splice the graphic features, the process features and the gaze features to obtain the graphic drawing features of the subject.

[0154] The prediction module 5 is used to input the graphic drawing features into the trained neural network model to predict the cognitive impairment result of the subject; the cognitive impairment result includes whether there is cognitive impairment and the level of cognitive impairment.

[0155] In addition, the system for recording graphics drawing process information further includes a training module; specifically, the training module includes:

[0156] The acquisition submodule is used to obtain the process information of the tester when drawing the graphics and the tester's facial video.

[0157] The gaze feature determination submodule is used to determine the gaze features of the tester during the drawing process based on the facial video of the tester.

[0158] The feature extraction submodule is used to extract features from the process information to obtain the tester's graphic features and process features.

[0159] The splicing submodule is used to splice the graphic features, the process features and the gaze features to obtain the graphic drawing features of the tester.

[0160] The result determination submodule is used to determine the cognitive impairment result of the test subject according to the test subject's score on the scale.

[0161] The training submodule is used to train the neural network model with the test subject's graphic drawing features as input and the test subject's cognitive impairment results as output to obtain a trained neural network model.

[0162] Wherein, the gaze feature determination module 2 includes:

[0163] The single-frame image extraction submodule is used to extract a single-frame image from the facial video of the subject.

[0164] The eye image determination submodule is used to apply the YOLO target detection algorithm to the single-frame image to obtain the eye image of the subject; the eye image includes an eye image with open eyes and an eye image with closed eyes.

[0165] The analysis submodule is used to apply a convolutional neural network to analyze the eye image of the open eyes to obtain eye positioning features.

[0166] The behavior determination submodule is used to determine the current behavior of the subject according to the eye positioning characteristics; the current behavior includes looking at sample pictures and drawing.

[0167] The recording submodule is used to record the time when the subject looks at the sample picture and the time when the subject draws the picture.

[0168] The gaze feature determination submodule is used to determine the gaze feature according to the time of viewing the sample image and the time of drawing the image.

[0169] In addition, the feature extraction module 3 includes:

[0170] The segmentation submodule is used to segment the process information according to the set time length, count the strokes, thinking time and drawing time in each time period, and obtain the initial process characteristics.

[0171] The initial graphic feature extraction submodule is used to extract initial graphic features based on the graphic drawn by the test subject; the graphic features include graphic area, graphic position, graphic drawing time, graphic thinking time and stroke length.

[0172] The dimensionality reduction submodule is used to perform dimensionality reduction on the initial process features and the initial graphic features respectively to obtain the graphic features and process features of the subject.

[0173] Unlike traditional complex graphic drawing tests, the present invention uses a tablet computer and a smart stylus as testing tools. When the tester draws a complex graphic, the system not only records the drawn graphic, but also records all the process information of the tester when drawing the graphic, including but not limited to the position of the pen, the start and end time of the strokes, the length of each stroke, and the time spent thinking when drawing the graphic, etc. In addition, with the help of the camera on the tablet, the gaze position of the tester when drawing the graphic can also be recorded. Based on this, the brain functions of the tester, such as visual space and spatial thinking ability, can be better evaluated. The information collected by the effects of this series of functions is so comprehensive and detailed that even professional doctors who concentrate on watching the entire drawing process can observe more than this. When the present invention is applied in a hospital scenario, compared with traditional tests, the present invention can better preserve the information during the graphic drawing process and provide analysis suggestions, which can provide effective assistance to doctors when making diagnoses; in large-scale evaluation scenarios such as community screening, it can provide technical support for the use of complex graphic drawing tests in large populations.

[0174] The comprehensive and detailed information collected by this series of functions far exceeds what can be obtained through traditional methods relying solely on doctor observation. When applied in hospital settings, compared to traditional testing methods, this invention can better preserve information during the graph drawing process and provide analytical suggestions, which can effectively assist doctors in making diagnoses. In large-scale assessment scenarios such as community screening, it can provide technical support for the use of complex graph drawing tests on large populations.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0176] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for recording graphics drawing process information, characterized in that: The method comprises: Obtaining process information of the subject drawing the figure and a facial video of the subject; the process information includes the starting position of each stroke, the stopping position of each stroke, the starting time of each stroke, the stopping time of each stroke, and the length of the trajectory of each stroke; Determining the gaze features of the subject during the drawing process based on the facial video of the subject; specifically comprising: Extracting a single frame image from the facial video of the subject; Applying the YOLO target detection algorithm to the single-frame image to obtain an eye picture of the subject with eyes open; Applying a convolutional neural network to analyze the eye image of the open eyes to obtain eye positioning features; Determining the current behavior of the subject according to the eye positioning feature; the current behavior includes looking at a sample picture and drawing; Recording the time when the subject looks at the sample picture and the time when the subject draws the picture; determining gaze characteristics according to a time of viewing the sample image and a time of drawing the image; The gaze characteristics include the number of gaze focus switches, the frequency of gaze focus switches, the drawing time, and the time of observing the sample drawing; Extracting features from the process information to obtain graphic features and process features of the subject; splicing the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the subject; The graphic drawing features are input into a trained neural network model to predict the cognitive impairment result of the subject; the cognitive impairment result includes whether cognitive impairment exists and the level of cognitive impairment.

2. The method for recording graphics drawing process information according to claim 1, characterized in that: The training process of the neural network model includes: Obtaining process information of the tester when drawing the graphics and the tester's facial video; determining, based on the tester's facial video, gaze features during the tester's drawing process; Performing feature extraction on the process information to obtain graphic features and process features of the tester; splicing the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the tester; Determining the cognitive impairment result of the test subject according to the test subject's score on the scale; The neural network model is trained with the test subject's graphic drawing features as input and the test subject's cognitive impairment results as output to obtain a trained neural network model.

3. The method for recording graphics drawing process information according to claim 1, characterized in that: The feature extraction of the process information to obtain the graphic features and process features of the subject specifically includes: Divide the process information into segments according to set time lengths, count the strokes, thinking time, and drawing time in each time segment, and obtain initial process features; Extracting initial graphic features based on the graphic drawn by the subject; the graphic features include graphic area, graphic position, graphic drawing time, graphic thinking time and stroke length; Dimensionality reduction is performed on the initial process features and the initial graphic features respectively to obtain graphic features and process features of the subject.

4. A system for recording graphics drawing process information, characterized in that: The system comprises: An acquisition module is used to obtain process information of the subject drawing the figure and the subject's facial video; the process information includes the starting position of each stroke, the stopping position of each stroke, the starting time of each stroke, the stopping time of each stroke and the length of each stroke; The gaze feature determination module is used to determine the gaze features of the subject during the drawing process based on the facial video of the subject; specifically, it includes: A single-frame image extraction submodule is used to extract a single-frame image from the facial video of the subject; an eye image determination submodule, configured to apply a YOLO target detection algorithm to the single-frame image to obtain an eye image of the subject with their eyes open; an analysis submodule, configured to analyze the open eye image using a convolutional neural network to obtain eye positioning features; A behavior determination submodule is used to determine the current behavior of the subject according to the eye positioning feature; the current behavior includes looking at sample pictures and drawing; A recording submodule, used to record the time when the subject looks at the sample picture and the time when the subject draws the picture; a gaze feature determination submodule, configured to determine the gaze feature according to the time of viewing the sample image and the time of drawing the image; The gaze characteristics include the number of gaze focus switches, the frequency of gaze focus switches, the drawing time, and the time of observing the sample drawing; A feature extraction module is used to extract features from the process information to obtain graphic features and process features of the subject; a splicing module, configured to splice the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the subject; The prediction module is used to input the graphic drawing features into the trained neural network model to predict the cognitive impairment results of the subject; the cognitive impairment results include whether cognitive impairment exists and the level of cognitive impairment.

5. The system for recording graphics drawing process information according to claim 4, characterized in that: The system further includes a training module; the training module includes: The acquisition submodule is used to obtain the process information of the tester when drawing the figure and the tester's facial video; A gaze feature determination submodule, configured to determine the gaze features of the tester during the drawing process based on the tester's facial video; A feature extraction submodule, configured to extract features from the process information to obtain graphic features and process features of the tester; a splicing submodule, configured to splice the graphic features, the process features, and the gaze features to obtain the graphic drawing features of the tester; A result determination submodule, for determining the cognitive impairment result of the test subject according to the test subject's score on the scale; The training submodule is used to train the neural network model with the test subject's graphic drawing features as input and the test subject's cognitive impairment results as output to obtain a trained neural network model.

6. The system for recording graphics drawing process information according to claim 4, characterized in that: The feature extraction module includes: The segmentation submodule is used to segment the process information according to the set time length, count the strokes, thinking time and drawing time in each time period, and obtain the initial process characteristics; An initial graphic feature extraction submodule is used to extract initial graphic features based on the graphic drawn by the subject; the graphic features include graphic area, graphic position, graphic drawing time, graphic thinking time and stroke length; The dimensionality reduction submodule is used to perform dimensionality reduction on the initial process features and the initial graphic features respectively to obtain the graphic features and process features of the subject.

Images