A mouse control method and system based on visual tracking

Abstract

The application discloses a mouse control method and system based on visual tracking, which comprises the following steps: dividing the display area of a display device, and starting a visual tracking framing analysis service after the camera of the display device determines whether a user is close to the device; processing the collected user image and positioning after recognition according to the divided sub-display area, training a mouse control state detection model based on the YOLOv5 algorithm; determining whether the mouse control service based on visual tracking is needed, analyzing the current user state to generate corresponding mouse control instruction information; and the mouse receiving the mouse control instruction information to perform mouse control operation on the display device. According to the visual tracking, the mouse pointer position is positioned in real time, and the evaluation of the mouse pointer prediction is increased, so that the use experience and control effect are improved, and the accuracy of the control on a large screen or a multi-screen display device is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action recognition, and particularly relates to a mouse control method and system based on visual tracking. BACKGROUND

[0002] At present, multi-screen or large-screen display devices and split-screen display devices are more applied to office occasions. When cooperating with a mouse for operation, it is often necessary to control the mouse to move in a large range or across screens. At this time, in order to reduce the moving distance of the mouse, the current main method is to adjust the DPI value of the mouse to control the moving distance of the mouse pointer under equal distance. This method can effectively solve the problem that the moving distance of the mouse controlled by the wrist is large when the mouse pointer needs to move a long distance, thereby causing discomfort in use. However, due to the high DPI, the mouse pointer often suddenly deviates from the line of sight, and it is necessary to find the mouse pointer on the screen, thereby affecting the use experience.

[0003] At the same time, the current mouse control has a body type control operation based on image processing, such as a mouse control method corresponding to a hand gesture head shape and an eyeball moving track. The traditional visual tracking control mouse control method only stays in the control of a single screen. There is a blank for multi-screen or large-screen display devices. There are problems such as poor experience, poor control effect, and general flexibility. Therefore, we propose a mouse control method and system based on visual tracking.

[0004] In the multi-screen or large-screen display system and mouse switching control method with the application number CN201911356809.7 and the name of a multi-screen or large-screen display system and mouse switching control method, the mouse switching control method is applied to a multi-screen or large-screen display system including a main display screen and at least one extended display screen, and includes: acquiring user images collected by each camera; each camera is installed on the main display screen and the extended display screen; inputting each user image into a neural network model, and predicting a screen currently focused by a user by using the neural network model to obtain a prediction result; and according to the prediction result, controlling the mouse to switch to the screen currently focused by the user. The multi-screen or large-screen display system and mouse switching control method of the present application embodiment are based on visual attention self-learning. By analyzing the visual attention direction of the current user, the screen currently operated by the user is predicted, and the mouse is automatically switched to the corresponding screen position, thereby improving the experience of the user.

[0005] In the application number CN202110399657X, the name is a kind of glasses mouse and its control method, the glasses mouse, including power supply and display device, still include: optical pre-processing module, optical reduction processing module, calculation unit, positioning unit, operation corresponding unit and output control unit. Simultaneously, a kind of method based on using optical combination unit, control display device operation is provided: user's eye action, the method includes receiving eye user's movement information, receives user's eye information, generates control command eye movement information and flicker information corresponding with the movement information of the user, and generates control command control display device module operation.

[0006] Therefore, based on image processing, there are poor experience, poor control effect and general flexibility in the mouse control of multi-screen or large-screen device, therefore we propose a mouse control method and system based on visual tracking. SUMMARY

[0007] In view of the above existing problems in the mouse control of multi-screen or large-screen device, the present application is proposed.

[0008] Therefore, one object of the present application is to provide a mouse control method and system based on visual tracking, which uses visual image processing and neural network prediction to finely recognize and position the divided regions of multi-screen or large-screen, and real-time position the mouse pointer according to visual tracking, while increasing the evaluation of mouse pointer prediction, improving the use experience and control effect, and further improving the accuracy of large-screen or multi-screen display device control.

[0009] To solve the above technical problems, the present application provides the following technical solutions:

[0010] In one aspect, the present application provides a mouse control method based on visual tracking, comprising:

[0011] Divide the display area of the display device, divide the panoramic screen picture of the display device into matrix type regions, and divide at least four sub-display areas of the display device;

[0012] The camera of the display device determines whether the user is close to the device, and if so, the visual tracking framing analysis service is started;

[0013] In response to the visual tracking framing analysis service, the user image collected by the camera is processed and recognized and positioned according to the divided sub-display area, and the mouse control state detection model based on YOLOv5 algorithm is trained and the result is outputted;

[0014] determining whether the mouse control service based on visual tracking is needed, and if so, analyzing the current user state, i.e., responding to the visual tracking analysis service and receiving the output result, and generating the corresponding mouse control instruction information according to the output result;

[0015] The mouse receives the mouse control instruction information to perform the mouse control operation of the display device.

[0016] As a preferred scheme of the present application, the mouse control state detection model based on the YOLOv5 algorithm is trained, specifically in response to the visual tracking analysis service, by delimiting the region of the image collected by the camera and identifying the region, and using the YOLOv5 algorithm to detect the mouse control state, finding the head contour and eye contour in the identified region, and after establishing the mouse control state detection dataset, dividing the mouse control state detection dataset as a training sample, and training the mouse control state detection model based on the YOLOv5 algorithm.

[0017] As a preferred scheme of the present application, the visual tracking analysis service is determined by the display device built-in camera to determine the user's head contour and eye contour, including head state, eye gaze and moving direction state, and blinking state.

[0018] Specifically, the head prediction box and eye prediction box of the user are obtained based on the prediction box of the YOLOv5 algorithm, the head state and eye focus direction of the user are identified, the line of sight direction is determined according to the eye focus direction, the eye focus direction of the user is obtained, the eye image is input into the line of sight prediction model to obtain the current user line of sight vector information, the line of sight prediction model is obtained by training a neural network model according to sample eye images and contour annotation information and annotation vectors corresponding to the sample eye images, and the contour annotation information includes eye contour and pupil contour.

[0019] According to the current user line of sight vector information, the current mouse position information is matched and located, and the next mouse position information is predicted and preloaded.

[0020] As a preferred scheme of the present application, according to the current user line of sight vector information, the current mouse position information is matched and located, and the next mouse position information is predicted and preloaded, specifically as follows:

[0021] The user's blinking is taken as a gaze period, and the current mouse position information is located and the next mouse position information is loaded in the gaze period.

[0022] According to the user's line of sight vector information, the matching and configuration of the mouse position information and the next mouse position information are carried out, and the matching of the next mouse position information is specifically carried out by determining the line of sight vector data of the previous frame, the current frame and the next frame according to the user's line of sight vector information, performing multiple linear regression fitting to predict the line of sight vector data of the next frame according to the line of sight vector data of the previous frame and the current frame, and then performing loop verification after determining the parameters by the least square method to obtain the predicted line of sight vector data. The predicted line of sight vector data is calibrated as the next mouse position information and is preloaded.

[0023] As a preferred scheme of the present application, wherein: further comprising according to the predicted line of sight vector data and the data of the user's current operation mouse and / or keyboard, performing current fixation period evaluation analysis on the predicted line of sight vector data, and performing online update on the neural network model according to the evaluation result, and the evaluation analysis is as follows:

[0024] The predicted score of the current line of sight vector data is calculated, and the comprehensive score is calculated, the current fixation period mouse position information is corrected according to the comprehensive score, and the matching and configuration of the next fixation period are performed again, as follows:

[0025]

[0026] Wherein, Q n is the comprehensive score of the nth line of sight vector data, m is the number of items of the nth line of sight vector data, α n is the optimization adjustment coefficient of the nth line of sight vector data, and is generated according to the accuracy of the evaluation of the mouse position information of the previous fixation period, β n is the weight correction coefficient of the nth line of sight vector data, indicating whether the position of the nth line of sight vector data is fixed, and if the position is fixed, β n is 1, w n is the weight coefficient of the nth line of sight vector data, q n is the score of the nth line of sight vector data.

[0027] S n = DW[η k (w a *x n ) k ,(w b *y n ) k ];

[0028] Wherein, S n is the score of the nth line of sight vector data, D is a preset coefficient value, W is the weight vector value of w a and w b , k is the kth level predicted by the neural network model, η kw is a discount factor of the kth level of the neural network model a , w b are respectively the weight of the target value weight vector value and the true value of the nth line-of-sight vector data in the neural network model, x n , y n are respectively the predicted target value and the true value of the nth line-of-sight vector data under the kth level of the neural network model.

[0029] As a preferred scheme of the present application, wherein: further comprising verifying the identity of the user, using the opencv library to analyze the user image data collected by real-time visual tracking, including image serialization, dynamic background separation, adaptive adjustment of light changes, identifying the face and eye key features of the user, including the opening and closing state of the eyes, the moving direction of the eyeballs and the expression changes of the face, generating a feature data set;

[0030] Based on the feature data set, an isolation forest algorithm is applied to analyze the facial and eye movement feature data of the user, identify abnormal behaviors that do not conform to the normal mouse control state mode, including line-of-sight shifting, abnormal facial expressions, turning the head to think, talking to someone next to them and leaving, identify the behavior characteristics of mouse control abnormalities and mouse control shifting, and generate an abnormal behavior identification result of the mouse control state.

[0031] As a preferred scheme of the present application, wherein: according to the current user line-of-sight vector information, the user line-of-sight range is analyzed, and the divided sub-display areas are identified and positioned according to the user line-of-sight range, specifically:

[0032] The user image collected by the camera is input into the neural network model, and the current attention of the user is predicted by the neural network model, i.e. the attention and scores formed by different sub-display areas divided in the panoramic screen picture, the attention and scores formed by different sub-display areas are sorted, the target controlled sub-display area is determined according to the sorting of different sub-display areas, and the target controlled sub-display area is identified and updated and matched with the subsequent next target controlled sub-display area.

[0033] As a preferred scheme of the present application, wherein: the determination parameter is determined by the least square method, and after the parameter is determined, the predicted line-of-sight vector data is obtained, the predicted line-of-sight vector data is calibrated as the next position information of the mouse and is preloaded

[0034] As a preferred scheme of the present application, wherein: it is judged whether the mouse control service based on visual tracking is needed, the mouse control service is started by visual tracking, voice or mouse control key, the visual tracking presets the head movement state corresponding to the starting of the mouse control service, and the user matches the starting of the mouse control service by controlling the head movement state.

[0035] In one aspect, the present application provides a system for visual tracking-based mouse control method, comprising:

[0036] A division module is configured to divide the display area of a display device, perform matrix area division on the panoramic screen picture of the display device, and divide at least four sub-display areas of the display device;

[0037] A first judgment module is configured to determine whether a user is close to the device through the camera of the display device, and if so, start a visual tracking framing analysis service;

[0038] An acquisition module is configured to respond to the visual tracking framing analysis service by processing the user image collected by the camera;

[0039] A positioning module is configured to position after recognition according to the divided sub-display area;

[0040] A second judgment module is configured to determine whether a mouse control service based on visual tracking is needed, and if so, analyze the current user state, i.e., respond to the visual tracking framing analysis service and receive the output result, and generate corresponding mouse control instruction information according to the output result;

[0041] A mouse control module is configured to receive the mouse control instruction information to perform mouse control operation on the display device.

[0042] The present application has the following advantages: the present application uses visual image processing and neural network prediction to finely recognize and position the divided areas of a multi-screen or large screen, and real-time position the mouse pointer according to visual tracking, while increasing the evaluation of the mouse pointer prediction, thereby improving the use experience and control effect, and further improving the accuracy of the control of the large screen or multi-screen display device. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor. Among them:

[0044] Figure 1 It is a flowchart of the overall method of the present application;

[0045] Figure 2 It is a flowchart of the training of the mouse control state detection model based on YOLOv5 algorithm in the method of the present application;

[0046] Figure 3A flow chart for predicting the line-of-sight vector data for the current gaze period evaluation analysis in the method of the present application;

[0047] Figure 4 A flow chart for generating the abnormal behavior recognition result of the mouse control state in the method of the present application;

[0048] Figure 5 A schematic diagram of the module structure of the system of the present application. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0050] Reference Figures 1-4 For one embodiment of the present application, the embodiment provides a mouse control method based on visual tracking, as shown in the flow chart, the steps are as follows: Figure 1

[0051] First, the display area of the display device is divided, and the panoramic screen picture of the display device is divided into a matrix type area, at least four sub-display areas of the display device are divided;

[0052] Second, the camera of the display device judges whether the user is close to the device, if yes, the visual tracking framing analysis service is started, otherwise the current state is maintained;

[0053] ​In response to the visual tracking framing analysis service, a mouse control state detection model based on the YOLOv5 algorithm is trained and outputs results by processing the user image collected by the camera and positioning after recognition according to the divided sub-display area. Specifically, in response to the visual tracking framing analysis service, the mouse control state detection model based on the YOLOv5 algorithm is trained, and after the image collected by the camera is regionally delimited and the region is recognized, the YOLOv5 algorithm is used to detect the mouse control state, the head contour and eye contour in the recognition region are found, and after the mouse control state detection dataset is established, the mouse control state detection dataset is divided as a training sample, and the mouse control state detection model based on the YOLOv5 algorithm is trained. This embodiment further illustrates that the YOLOv5 algorithm is a single-stage target detection algorithm, which greatly improves the speed and accuracy based on YOLOv4. Specifically, it includes: input Mosaic data enhancement, adaptive anchor box calculation, adaptive image scaling operation; benchmark Focus structure and CSP structure; Neck SPP and FPN+PAN structure; output loss function GIOULoss and prediction box screening DlOU nms. Specifically applied in the visual tracking mouse control technical solution of the embodiment. Specifically as follows:

[0054] Input, input represents the input user image picture. The input image size of the network is 608*608, and this stage usually includes an image preprocessing stage, that is, the input image is scaled to the input size of the network, and normalization and other operations are performed. In the network training stage, YOLOv5 uses Mosaic data enhancement operation to improve the training speed of the model and the accuracy of the network; and proposes an adaptive anchor box calculation and adaptive image scaling method.

[0055] Benchmark network, the benchmark network is usually a network of some excellent classifier species, which is used to extract some general feature representation. In YOLOv5, not only CSPDarknet53 structure is used, but also Focus structure is used as the benchmark network.

[0056] Neck network, the Neck network is usually located in the middle position between the benchmark network and the head network, and it can further improve the diversity and robustness of the features.

[0057] Head output, Head is used to complete the output of the target detection result. For different detection algorithms, the number of branches of the output end is different, and usually includes a classification branch and a regression branch. YOLOv4 uses GIOU Loss to replace Smooth L1 Loss function, thereby further improving the detection accuracy of the algorithm.

[0058] Then, it is determined whether mouse control service based on visual tracking is needed. If so, the current user state is analyzed, that is, the visual tracking framing analysis service is responded to and the output results are received, and corresponding mouse control command information is generated based on the output results; if not, tracking is stopped.

[0059] Finally, the mouse receives the mouse control instruction information to perform mouse control operations on the display device.

[0060] In this preferred embodiment, the visual tracking framing analysis service determines the user's head contour and eye contour through the built-in camera of the display device, including head state, eye gaze and movement direction, and blinking state;

[0061] like Figure 2 The process shown specifically involves obtaining the user's head prediction box and eye prediction box based on the YOLOv5 algorithm, identifying the user's head state and eye focus direction, determining the gaze direction based on the eye focus direction, specifically obtaining the user's eye focus direction, and inputting the eye image into the gaze prediction model to obtain the current user gaze vector information. The gaze prediction model is obtained by training a neural network model based on the contour annotation information and annotation vector corresponding to the sample eye image. The contour annotation information includes the eye contour and pupil contour.

[0062] The current mouse position is located by matching the current user's gaze vector information, and the next mouse position information is predicted and preloaded.

[0063] This embodiment further illustrates that the current mouse position information is matched and located based on the current user's gaze vector information, and the predicted next mouse position information is preloaded, as detailed below:

[0064] The user's blink is taken as a gaze cycle, and the current mouse position information is located and the next mouse position information is loaded during the gaze cycle.

[0065] The mouse position information and the next mouse position information are matched and configured based on the user's gaze vector information. Specifically, the matching of the next mouse position information is achieved by determining the user's gaze vector information as the gaze vector data of the previous frame, the current frame, and the next frame. Based on the gaze vector data of the previous and current frames, multiple linear regression is used to fit and predict the gaze vector data of the next frame. After determining the parameters by least squares method, the prediction gaze vector data is obtained through iterative verification. The predicted gaze vector data is then labeled as the next mouse position information and preloaded. If the matching fails, repositioning and error handling are performed.

[0066] like Figure 4In addition to the illustrated flow, the embodiment also includes verification of the user's identity, using the opencv library to analyze real-time visual tracking of the user image data collected by the viewfinder, including image serialization, dynamic background separation, adaptive adjustment to changes in lighting, identifying the user's facial and eye key features, including eye opening and closing state, eye movement direction, and facial expression changes, generating a feature data set, and failing to re-adjust the camera to start image data collection and preprocessing;

[0067] It is emphasized that based on the feature data set, the isolated forest algorithm is applied to analyze the user's facial and eye movement feature data, identify abnormal behaviors that do not conform to the normal mouse control state mode, including eye line shift, abnormal facial expression, turning head to think, talking to the person next to them, and leaving, identifying mouse control abnormality and mouse control transfer behavior characteristics, and generating mouse control state abnormal behavior recognition results.

[0068] The embodiment preferably analyzes the user's eye line range according to the current user eye line vector information, identifies and positions the divided sub-display areas according to the user eye line range, specifically:

[0069] The user image collected by the camera is input into the neural network model, and the current attention of the user is predicted by the neural network model, i.e. the attention and score formed by the different sub-display areas divided by the panoramic screen picture, the attention and score formed by the different sub-display areas are sorted, the target controlled sub-display area is determined according to the sorting of the different sub-display areas, and the target controlled sub-display area is identified and updated for the subsequent next target controlled sub-display area.

[0070] The embodiment specifically determines whether the mouse control service needs to be based on visual tracking, and the mouse control service is started by visual tracking, voice or mouse control key. The visual tracking presets the head movement state corresponding to the start of the mouse control service, and the user matches the start of the mouse control service by controlling the head movement state.

[0071] As described above, this embodiment uses a camera to recognize the user's head and eye movements, identifying changes in the user's state and determining whether the user's movement requires mouse movement or is not for operation, such as turning the head to think or talking to someone. When the user's movement is identified as requiring mouse movement, the system calculates the user's field of vision based on the state of the user's eyes and pupils, and automatically controls the mouse pointer to move within that target area for precise visual tracking. At this point, after processing the visual image information and predicting it through a neural network, the corresponding position of the mouse within the corresponding area can be generated, completing the recognition, positioning, and tracking. This allows the user to intelligently operate the mouse to perform operations such as switching, moving, or clicking between large or multi-screen display devices.

[0072] like Figure 3 To improve the accuracy of controlling large-screen or multi-screen display devices during visual tracking, this embodiment preferably further includes, based on the predicted gaze vector data and the user's current mouse and / or keyboard input data, performing a current gaze cycle evaluation analysis on the predicted gaze vector data, simultaneously assessing whether offline analysis is needed, planning offline analysis, and updating the neural network model online based on the evaluation results. The evaluation analysis is as follows:

[0073] Calculate the prediction score of the current gaze vector data, and calculate the comprehensive score. Based on the comprehensive score, correct the mouse position information for the current gaze cycle, and then perform matching and configuration for the next gaze cycle, as follows:

[0074]

[0075] Among them, Q n Let m be the overall score of the nth gaze vector data, and m be the number of items in the nth gaze vector data. n β is the optimization adjustment coefficient for the nth gaze vector data, generated based on the accuracy of the mouse position information evaluation in the previous gaze cycle. n β is the weight correction coefficient for the nth gaze vector data, indicating whether the position of the nth gaze vector data is fixed. If the position is fixed, then β... n For 1, w n q represents the weighting coefficient of the nth gaze vector data. n The score for the nth gaze vector data;

[0076] S n =DW[η k (w a *x n ) k ,(w b *y n ) k ];

[0077] wherein S n is the score of the nth line-of-sight vector data, D is a preset coefficient value, W is a weight vector value of w a and w b , k is the kth level predicted by the neural network model, η k is a discount factor of the kth level of the neural network model, w a , w b are the weight vector values of the target value and the true value of the nth line-of-sight vector data in the neural network model, respectively, x n , y n are the predicted target value and the true value of the nth line-of-sight vector data at the kth level of the neural network model, respectively.

[0078] Based on the above, the embodiment determines the line-of-sight vector information of the user as the line-of-sight vector data of the previous frame, the current frame and the next frame, and performs comprehensive score analysis according to the line-of-sight vector data of the previous frame and the current frame when the score meets the threshold value, that is, performs multiple linear regression fitting to predict the line-of-sight vector data of the next frame, and determines the parameters through the least square method and then performs loop verification, obtains the predicted line-of-sight vector data, and calibrates the predicted line-of-sight vector data as the next position information of the mouse and performs preloading. In this way, not only the prediction and preloading of the next frame position of the mouse are realized, but also the predicted line-of-sight vector data is evaluated and analyzed according to the current fixation period, and the neural network model is updated online according to the evaluation result, which greatly improves the accuracy of controlling the large screen or multi-screen display device.

[0079] As shown in the flowchart, Figure 5 the embodiment provides a system of a mouse control method based on visual tracking, comprising:

[0080] a division module for dividing the display area of the display device, performing matrix region division on the panoramic screen picture of the display device, and dividing at least four sub-display areas of the display device;

[0081] a first judgment module for judging whether the user is close to the device through the camera of the display device, and starting the visual tracking framing analysis service if yes;

[0082] an acquisition module for responding to the visual tracking framing analysis service and processing the user image collected by the camera;

[0083] a positioning module for positioning after recognition according to the divided sub-display area;

[0084] The second judging module is configured to judge whether the mouse control service based on the visual tracking is needed, and if so, analyze the current user state, i.e. respond to the visual tracking analysis service and receive the output result, and generate the corresponding mouse control instruction information according to the output result;

[0085] The mouse control module is configured to receive the mouse control instruction information to perform the mouse control operation on the display device.

[0086] In summary, the hardware module of the embodiment can combine the method in the first embodiment, utilize visual image processing and neural network prediction, perform fine recognition and positioning after dividing the multiple screens or large screen into regions, and real-time position the mouse pointer according to the visual tracking, and further improve the accuracy of the control of the large screen or multiple screen display device.

[0087] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another.

[0088] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0089] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one feature. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0090] Any processes or methods described in the flow charts or elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) of the process. Moreover, in some embodiments, the various functions or steps can be performed in the order shown or in a different order. Additionally, one or more steps or functions can be combined or omitted.

[0091] Logic and / or steps represented in the flow charts or elsewhere herein, for example, can be embodied in computer-executable instructions or code in which the steps or functions represented are performed in sequence, in parallel, or in a different order, including substantially concurrently or in reverse order, depending on the functionality involved. The code can be stored in any form of computer- readable medium or memory of a computer, including volatile or non-volatile storage elements, surfaces, or other media.

[0092] It should be understood that some or all of the various embodiments of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, some or all of the steps or methods can be implemented by software or firmware stored in a memory and executed by an appropriate instruction execution system. All or part of the steps of the above-described embodiments can be instructed by a program to the relevant hardware, which can be stored in a computer-readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0093] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software function module. When the above-mentioned integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0094] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, and these should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A visual tracking-based mouse control method, characterized by, Comprise: Divide the display area of the display device, divide the panoramic screen picture of the display device into matrix area, and divide at least four sub-display areas of the display device; The camera of the display device determines whether the user is close to the device, and if so, the visual tracking framing analysis service is started, the current mouse position information is matched and positioned according to the current user gaze vector information, and the next mouse position information is predicted and preloaded. Specifically, the user's blink is taken as a gaze period, and the current mouse position information is positioned and the next mouse position information is loaded in the gaze period. According to the user gaze vector information, the mouse position information and the next mouse position information are matched and configured, that is, by determining the user gaze vector information as the line-of-sight vector data of the previous frame, the current frame and the next frame, according to the line-of-sight vector data of the previous frame and the current frame, the line-of-sight vector data of the next frame is predicted by multiple linear regression fitting, and the parameters are determined. After cyclic verification, the predicted line-of-sight vector data is obtained, which is calibrated as the next mouse position information and preloaded; It also includes evaluating and analyzing the predicted line-of-sight vector data according to the predicted line-of-sight vector data and the data of the user's current operation of the mouse and / or keyboard, and updating the neural network model online according to the evaluation result. The evaluation and analysis are as follows: Calculate the prediction score of the current line-of-sight vector data and the comprehensive score, correct the mouse position information in the current gaze period according to the comprehensive score, and match and configure again in the next gaze period, as follows: ； in, For the first n The combined score of each gaze vector data. m For the first n The number of items in each gaze vector data. For the first n The optimization adjustment coefficients for each gaze vector data are generated based on the accuracy of the mouse position information evaluation from the previous gaze cycle. For the first n The weight correction coefficients for the nth gaze vector data represent the weights of the nth gaze vector data. n Are the positions of the gaze vector data fixed? If the positions are fixed, then... =1, For the first n The weighting coefficients of each gaze vector data. For the first n The score of each gaze vector data; ； in, For the first n The score of each gaze vector data. The preset coefficient value, for and The weight vector values, k The first prediction for the neural network model k Hierarchy For the neural network model k Tier discount factor , The first n The weights of each gaze vector data point in the neural network model are assigned to the target value weight vector and the true value. , The first n The first gaze vector data in the neural network model k The predicted target value and the actual value at each level; In response to the visual tracking framing analysis service, the user image collected by the camera is processed and positioned according to the divided sub-display area. Specifically, the user image collected by the camera is input into the neural network model, and the current attention of the user is predicted by the neural network model, that is, the attention and score formed by different sub-display areas of the panoramic screen picture. According to the attention and score formed by different sub-display areas, the target controlled sub-display area is determined according to the sorting of different sub-display areas, and the target controlled sub-display area is identified and updated for the next target controlled sub-display area. The mouse control state detection model based on YOLOv5 algorithm is trained and the output result is output; Determine whether the mouse control service based on visual tracking is needed, and if so, analyze the current user state, that is, in response to the visual tracking framing analysis service and receiving the output result, generate the corresponding mouse control instruction information according to the output result; The mouse receives the mouse control instruction information to perform mouse control operation on the display device It also includes verifying the user's identity, using the opencv library to analyze the user image data collected by real-time visual tracking framing, including image serialization, dynamic background separation, and adaptive adjustment of light changes, identifying the user's face and eye key features, including eye opening and closing state, eye movement direction and facial expression changes, and generating a feature data set. The Isolation Forest algorithm is applied based on the feature data set to analyze the facial and eye movement feature data of the user, identify abnormal behaviors that do not conform to the normal mouse control state mode, including gaze shift, abnormal facial expression, head turning for thinking, talking to the person next to him, and leaving, identify the behavior characteristics of mouse control abnormalities and mouse control shift, and generate an abnormal behavior identification result of the mouse control state.

2. The visual tracking-based mouse control method of claim 1, wherein, The mouse control state detection model training based on the YOLOv5 algorithm, specifically in response to the visual tracking framing analysis service, after the region is delimited and recognized in the image collected by the camera, the YOLOv5 algorithm is used to realize the detection of the mouse control state, the head outline and eye outline in the recognition region are found, after the mouse control state detection data set is established, the mouse control state detection data set is divided as a training sample, and the mouse control state detection model training based on the YOLOv5 algorithm.

3. The visual tracking-based mouse control method of claim 1, wherein, The visual tracking framing analysis service determines the head outline and eye outline of the user through the built-in camera of the display device, including the head state, eye gaze fixation, and moving direction state, and the blinking state; Specifically, the head prediction box and eye prediction box of the user are obtained based on the prediction box of the YOLOv5 algorithm, the head state and the focusing direction of the eyes of the user are identified, the gaze direction is determined according to the focusing direction of the eyes, the focusing direction of the eyes of the user is obtained, the eye image is input into the gaze prediction model to obtain the current user gaze vector information, the gaze prediction model is obtained by training a neural network model according to sample eye images and the contour annotation information and annotation vector corresponding to the sample eye images, and the contour annotation information includes the eye contour and the pupil contour.

4. The visual tracking-based mouse control method of claim 1, wherein, The determined parameters are verified in a loop after the parameters are determined by the least squares method, the predicted gaze vector data is obtained, the predicted gaze vector data is calibrated as the next mouse position information and is preloaded.

5. The visual tracking-based mouse control method of claim 1, wherein, It is judged whether the mouse control service needs to be based on visual tracking, the mouse control service is started by visual tracking, voice or mouse control key, the visual tracking presets the head movement state corresponding to the starting of the mouse control service, and the user matches the starting of the mouse control service by controlling the head movement state.

6. A system applied to the visual tracking-based mouse control method according to claim 1, characterized in that, It includes: The division module is used for dividing the display area of the display device, performing matrix region division on the panoramic screen picture of the display device, and dividing at least four sub-display areas of the display device; The first judgment module is used for judging whether the user is close to the device through the camera of the display device, and the visual tracking framing analysis service is started if so; The acquisition module is used for responding to the visual tracking framing analysis service and processing the user image collected by the camera; The positioning module is used for positioning after recognition according to the divided sub-display area; The second judgment module is used for judging whether the mouse control service needs to be based on visual tracking, and analyzing the current user state according to the output result if so, that is, responding to the visual tracking framing analysis service and receiving the output result, and generating corresponding mouse control instruction information according to the output result; A mouse control module is configured to receive the mouse control instruction information and perform mouse control operation on the display device.

Images