Target equipment control method, electronic equipment and storage medium

Abstract

The invention discloses a target equipment control method, electronic equipment and a storage medium, and the method comprises the steps: carrying out the motion tracking of a target object, and obtaining the motion data of the target object; determining an operation intention of the target object based on the motion data of the target object; and controlling the target device to perform corresponding operation based on the operation intention of the target object. In this way, the operation convenience of the target equipment can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction, and in particular to a control method for a target device, an electronic device and a storage medium. Background Art

[0002] Object detection is a very active research direction in computer vision, pattern recognition and machine learning. Object detection is widely used in many fields. Object detection is a basic problem in visual research and also a very challenging problem.

[0003] As smart devices become more popular in people's lives, the market demand for the functions and interaction methods of smart devices continues to increase. Traditional manual operation methods may limit the experience and may not be convenient enough in some cases. Summary of the invention

[0004] The present invention provides a control method of a target device, an electronic device and a storage medium to solve the problem that the traditional manual operation method is not convenient enough.

[0005] In order to solve the above technical problems, the present invention provides a control method for a target device, comprising: tracking the motion of a target object to obtain motion data of the target object; determining the operation intention of the target object based on the motion data of the target object; and controlling the target device to perform corresponding operations based on the operation intention of the target object.

[0006] Among them, the step of determining the operation intention of the target object based on the motion data of the target object includes: determining the operation intention of the target object based on the motion data of the target object through a motion analysis model; wherein the motion analysis model is obtained by training based on training data, and the training data includes training motion data and corresponding training operation intentions.

[0007] Among them, the step of tracking the motion of the target object and obtaining the motion data of the target object includes: tracking the motion of the target object in multiple usage states of the target device respectively to obtain multiple training motion data of the target object; determining the training operation intention corresponding to each training motion data in each usage state; training the motion analysis model in training based on the multiple training motion data in each usage state and the corresponding training operation intention until the motion analysis model is obtained.

[0008] Among them, the usage status includes: reading text status, the training operation intentions include: turning pages, pausing, exiting, automatic scrolling, adjusting font size, fast reading and slow reading; and / or the usage status includes: watching video status, the training operation intentions include: pausing, exiting, adjusting volume, fast watching and slow watching.

[0009] Among them, the step of controlling the target device to perform a corresponding operation based on the operation intention of the target object also includes: in response to receiving an operation adjustment instruction, controlling the target device to perform a corresponding operation based on the operation adjustment instruction; establishing a corresponding relationship between the operation adjustment instruction and the motion data of the target object, and using the corresponding operation adjustment instruction and the motion data of the target object to train the motion analysis model again.

[0010] Among them, the step of tracking the motion of the target object and obtaining the motion data of the target object includes: in response to the target device receiving a tracking instruction, obtaining the current usage status of the target device; the step of determining the operation intention of the target object based on the motion data of the target object includes: determining the operation intention of the target object in the current usage status based on the motion data of the target object.

[0011] Among them, the steps of tracking the motion of the target object and obtaining the motion data of the target object include: capturing a detection image including the target object by a camera device of the target device; the detection image includes multiple continuous image frames; performing image recognition on each image frame to determine the position of the target object in each image frame; and determining the motion data of the target object by comprehensively analyzing the position of the target object in each image frame.

[0012] Among them, the target object includes the pupil; the step of tracking the motion of the target object and obtaining the motion data of the target object includes: tracking the motion of the pupil to obtain the motion data of the pupil, and the motion data includes one or more of the gaze point, gaze duration, eye saccade, movement trajectory and pupil zoom.

[0013] To solve the above technical problems, the present invention further provides an electronic device, which includes: a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement any of the above target device control methods.

[0014] In order to solve the above technical problems, the present invention further provides a computer-readable storage medium, which stores program data, and the program data can be executed to implement any of the above-mentioned control methods for the target device.

[0015] The beneficial effects of the present invention are as follows: different from the prior art, the present invention obtains the motion data of the target object by tracking the motion of the target object; determines the operation intention of the target object based on the motion data of the target object; and controls the target device to perform corresponding operations based on the operation intention of the target object, thereby realizing remote control of the target device by the mobile target object, greatly improving the convenience, usability and practicality of controlling the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1is a flow chart of an embodiment of a control method for a target device provided by the present invention;

[0017] Figure 2 is a flow chart of another embodiment of a method for controlling a target device provided by the present invention;

[0018] Figure 3 is a schematic diagram of a training process of an implementation method of a motion analysis model;

[0019] Figure 4 is a structural schematic diagram of an embodiment of an electronic device provided by the present invention;

[0020] Figure 5 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of a control method of a target device provided by the present invention.

[0023] Step S11: Track the motion of the target object to obtain the motion data of the target object.

[0024] The target object of this embodiment may include but is not limited to pupils, fingers, heads and other parts. By tracking the motion data of the target object, the user's operation intention can be determined. The motion data may include but is not limited to parameters such as motion trajectory, amplitude, frequency, deformation, etc. The determination is specifically based on the type of the target object.

[0025] In a specific application scenario, the target object can be tracked by using sensor position detection and / or image detection for identification, etc., which can be set based on the type of the target object. Sensor position detection can be infrared sensor detection, ultrasonic sensor detection, or laser detection, etc.

[0026] Step S12: determining the operation intention of the target object based on the motion data of the target object.

[0027] Different motion data corresponds to different operation intentions, and the operation intention of the target object is determined based on the motion data of the target object.

[0028] In a specific application scenario, a correspondence list between various motion data and various operation intentions may be pre-constructed, and then based on the correspondence list in this step, the operation intention of the target object corresponding to the motion data of the target object is determined.

[0029] In a specific application scenario, the deep learning model can be pre-trained to learn the correspondence between multiple motion data and multiple operation intentions, and make predictions, so as to determine the operation intention of the target object corresponding to the motion data of the target object in this step.

[0030] The specific method for determining the operation intention of the target object based on the motion data of the target object is not limited here.

[0031] Step S13: controlling the target device to perform corresponding operations based on the operation intention of the target object.

[0032] After the operation intention of the target object is determined, the target device is controlled to perform corresponding operations based on the operation intention.

[0033] Among them, the target devices include but are not limited to mobile phones, tablets, TVs, computers, bracelets, watches, smart homes and other smart devices, so as to achieve convenient control of the target devices.

[0034] Through the above steps, the control method of the target device of this embodiment obtains the motion data of the target object by tracking the motion of the target object; determines the operation intention of the target object based on the motion data of the target object; and controls the target device to perform corresponding operations based on the operation intention of the target object, thereby realizing remote control of the target device by the mobile target object, greatly improving the convenience, usability and practicality of controlling the target device.

[0035] See also Figure 2 , Figure 2 It is a flow chart of another embodiment of the control method of the target device provided by the present invention.

[0036] Step S21: photographing a detection image including a target object through a camera device of the target device; the detection image includes a plurality of continuous image frames.

[0037] Before step S21, in response to the target device receiving the tracking instruction, the current use status of the target device is obtained. The tracking instruction is an instruction to start the control method of the target device, that is, the user issues the tracking instruction to start the function of automatically controlling the target device to perform corresponding operations based on the motion data of the target object. If the user does not issue the tracking instruction, it means that he may not need this function, and the target device works normally.

[0038] There are many usage states of the target device, such as reading text or watching video. Even the same motion data may correspond to different operation intentions, so it is necessary to distinguish the usage states.

[0039] In a specific application scenario, a simple and intuitive interface can be designed to turn the above functions on and off, display the current operation intention, and provide adjustable setting options, such as adjusting sensitivity, selecting supported applications, etc., to improve the operation convenience of the target device.

[0040] The camera device may include a camera on the target device that can face the target object, including a front camera or a steering camera, etc. The camera device captures a detection image including the target object; the detection image includes a plurality of continuous image frames.

[0041] The target object of this embodiment can be the pupil. By tracking the pupil's eye movement, the operation intention is determined and the target device is controlled to achieve convenient control. The pupil movement is tracked by tracking the position of the pupil or according to the movement of the pupil relative to the head. The obtained motion data includes but is not limited to one or more of the gaze point, gaze duration, eye saccade, movement trajectory, pupil zoom, etc., and the operation intention is determined based on the pupil's motion data.

[0042] This embodiment determines the user's operation intention based on the movement data of the user's pupil, and can use the user's pupil, which is naturally moving, to complete the control of the target device, thereby reducing the user's additional operation methods and greatly improving the convenience and efficiency of the user's control of the target device.

[0043] The detection image includes multiple continuous image frames to reflect the complete motion data of the target object, and the specific number of image frames can be set based on actual conditions.

[0044] The detection image is preprocessed, which may include but is not limited to operations such as cropping, grayscale adjustment, filtering, and denoising to improve the accuracy of subsequent image recognition.

[0045] Step S22: performing image recognition on each image frame to determine the position of the target object in each image frame; and determining the motion data of the target object by integrating the position of the target object in each image frame.

[0046] After acquiring a plurality of continuous image frames, image recognition is performed on each image frame to determine the position of the target object in each image frame; and the motion data of the target object is determined by combining the position of the target object in each image frame.

[0047] In a specific application scenario, the position of the pupil can be obtained through eye tracking hardware or software libraries, such as Tobii EyeTracker or OpenCV. Integrate eye tracking function in the application to ensure that the position of the pupil can be captured in real time and the movement data of the pupil can be obtained.

[0048] In a specific application scenario, multiple image frames can be input into the trained eye movement tracking model in sequence. The model outputs the pupil positioning result of each image frame. The pupil movement data is obtained by combining the pupil positioning results of each image frame.

[0049] Step S23: determining the operation intention of the target object based on the motion data of the target object through a motion analysis model.

[0050] After the motion data of the target object is obtained, the operation intention of the target object in the current use state is determined based on the motion data of the target object through a motion analysis model.

[0051] The motion analysis model is trained based on training data, and the training data includes training motion data and corresponding training operation intentions.

[0052] See also Figure 3 , Figure 3 1 is a schematic diagram of a training process of a motion analysis model implementation. This implementation is performed before step S21.

[0053] Step S301: Under various usage states of the target device, the target object is tracked respectively to obtain various training motion data of the target object.

[0054] The target device may be in different usage states during use. In different usage states, even the same motion data may correspond to different operation intentions. Therefore, during training, the various usage states of the target device are first distinguished, and the motion of the target object is tracked separately in the various usage states of the target device to obtain various training motion data of the target object.

[0055] The training motion data may be acquired through eye tracking hardware or software libraries, such as Tobii Eye Tracker or OpenCV, or may be determined through continuous multi-frame image recognition.

[0056] The training motion data may include but are not limited to fixed point, left movement, right movement, upward movement, downward movement or tilt movement, etc. When the target object is the pupil, the training motion data is eye movement data, including but not limited to one or more of gaze point, gaze duration, eye saccade, movement trajectory and pupil zoom.

[0057] In a specific application scenario, eye movement data is collected in different situations, such as reading text, watching videos, etc. The collected eye movement data is preprocessed, including filtering, denoising and other operations, to improve the accuracy of the subsequent model's intention learning.

[0058] Step S302: Determine the training operation intention corresponding to each training motion data in each usage state.

[0059] In a specific application scenario, the training operation intention of the training motion data under different usage states can be determined through manual labeling.

[0060] In a specific application scenario, the command operation received by the target device can be obtained in real time, and the operation intention of the training motion data can be determined based on the command operation closest to the time axis of the training motion data.

[0061] In a specific application scenario, the training operation intentions can be divided into several categories. Then the trained machine learning model is used to classify the eye movement data to determine the corresponding training operation intentions.

[0062] If the usage state includes: reading text, the training operation intention includes: turning pages, pausing, exiting, automatically scrolling, adjusting font size, fast reading, and slow reading. If the usage state includes: watching video, the training operation intention includes: pausing, exiting, adjusting volume, fast viewing, and slow viewing. The usage state of the target device and the types of training operation intentions under it are not limited here.

[0063] Step S303: training the motion analysis model in training based on the various training motion data in each use state and the corresponding training operation intentions until a motion analysis model is obtained.

[0064] The motion analysis model can choose to use deep learning models, such as convolutional neural networks, recurrent neural networks, fully connected neural networks, long short-term memory networks, deep belief networks, or generative adversarial networks, etc., to process eye movement data and perform intent classification learning.

[0065] During training, the training operation intentions corresponding to the training motion data under each usage state can be used as labels to perform supervised training on the motion analysis model in training until the motion analysis model learns the training operation intentions corresponding to the training motion data under each usage state and has prediction capabilities, thereby obtaining a trained motion analysis model.

[0066] The above training methods are used to test in different scenarios and user groups, collect feedback, and make improvements and optimizations based on the feedback. The performance of the motion analysis model is continuously monitored to ensure its stability and reliability.

[0067] In step S23, the operation intention of the target object in the current use state is determined based on the motion data of the target object through the trained motion analysis model.

[0068] Step S24: controlling the target device to perform corresponding operations based on the operation intention of the target object.

[0069] In a specific application scenario, when the target device is currently in the state of reading text, if the operation intention is to turn the page, the target device is controlled to turn the text down one page. In a specific application scenario, when the target device is currently in the state of watching a video, if the operation intention is to pause, the target device is controlled to pause the video playback.

[0070] Step S25: In response to receiving the operation adjustment instruction, control the target device to perform a corresponding operation based on the operation adjustment instruction.

[0071] When or after controlling the target device to perform a corresponding operation based on the operation intention of the target object, if an operation adjustment instruction is received, it means that the operation intention may be confirmed incorrectly, and the target device is controlled to perform a corresponding operation based on the operation adjustment instruction.

[0072] In a specific application scenario, if a corresponding relationship is established between the operation adjustment instruction and the motion data of the target object, and the motion analysis model is trained again using the corresponding operation adjustment instruction and the motion data of the target object, the accuracy of the intention analysis of the motion analysis model can be further improved. This re-optimization process can be performed each time an operation adjustment instruction is obtained, or when the operation adjustment instruction and the motion data of the target object reach a certain value.

[0073] The above settings can continuously optimize and improve the motion analysis model to improve accuracy and response speed, and enable the motion analysis model to self-adjust and improve based on feedback from actual use.

[0074] Through the above steps, the control method of the target device of this embodiment obtains the motion data of the target object by tracking the motion of the target object; determines the operation intention of the target object based on the motion data of the target object through a motion analysis model; and controls the target device to perform corresponding operations based on the operation intention of the target object, thereby using deep learning combined with eye tracking to realize remote control of the target device by moving the target object, greatly improving the convenience, usability and practicality of controlling the target device. Reduce the inconvenience caused by manual operation and provide a more intelligent and personalized user experience. In addition, this embodiment determines the operation intention based on the motion data of the user's pupil, and can use the user's pupil, which is originally able to move, to complete the control of the target device, thereby reducing the user's additional operating means and greatly improving the convenience and efficiency of the user's control of the target device.

[0075] Based on the same inventive concept, the present invention also proposes an electronic device, which can be executed to implement the control method of the target device in any of the above embodiments. Figure 4 , Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device includes a processor 41 and a memory 42.

[0076] The processor 41 is used to execute the program instructions stored in the memory 42 to implement the steps of the control method of any target device described above. In a specific implementation scenario, the electronic device may include but is not limited to: a microcomputer, a server, and in addition, the electronic device may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0077] Specifically, the processor 41 is used to control itself and the memory 42 to implement the steps of any of the above embodiments. The processor 41 can also be called a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip with signal processing capabilities. The processor 41 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 41 can be implemented by an integrated circuit chip.

[0078] The above solution can improve the convenience, usability and practicality of controlling the target device.

[0079] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, see Figure 5 , Figure 5 1 is a schematic diagram of the structure of an embodiment of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 50 stores at least one program data 51, and the program data 51 is used to implement any of the above methods. In one embodiment, the computer-readable storage medium 50 includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0080] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0082] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium.

[0084] The above are only implementation modes of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for controlling a target device, characterized in that: include: Tracking the motion of the target object to obtain motion data of the target object; determining the operation intention of the target object based on the motion data of the target object; The target device is controlled to perform a corresponding operation based on the operation intention of the target object.

2. The method for controlling a target device according to claim 1, characterized in that: The step of determining the operation intention of the target object based on the motion data of the target object comprises: Determining the operation intention of the target object based on the motion data of the target object through a motion analysis model; The motion analysis model is obtained through training based on training data, and the training data includes training motion data and corresponding training operation intentions.

3. The method for controlling a target device according to claim 2, characterized in that: The step of tracking the motion of the target object to obtain the motion data of the target object includes: Under various use states of the target device, the target object is tracked respectively to obtain various training motion data of the target object; Determine the training operation intention corresponding to each of the training motion data in each of the use states; The motion analysis model in training is trained based on the multiple training motion data under each of the usage states and the corresponding training operation intentions until the motion analysis model is obtained.

4. The method for controlling a target device according to claim 3, characterized in that: The use status includes: text reading status, and the training operation intention includes: page turning, pausing, exiting, automatic scrolling, adjusting font size, fast reading and slow reading; and / or The use status includes: video watching status, and the training operation intention includes: pause, exit, adjust volume, fast watch and slow watch.

5. The method for controlling a target device according to claim 2, characterized in that: The step of controlling the target device to perform a corresponding operation based on the operation intention of the target object further includes: In response to receiving the operation adjustment instruction, controlling the target device to perform a corresponding operation based on the operation adjustment instruction; A correspondence is established between the operation adjustment instruction and the motion data of the target object, and the motion analysis model is trained again using the corresponding operation adjustment instruction and the motion data of the target object.

6. The method for controlling a target device according to any one of claims 1 to 5, characterized in that: The step of tracking the motion of the target object to obtain the motion data of the target object includes: In response to the target device receiving the tracking instruction, obtaining a current usage status of the target device; The step of determining the operation intention of the target object based on the motion data of the target object comprises: The operation intention of the target object in the current use state is determined based on the motion data of the target object.

7. The method for controlling a target device according to claim 1, characterized in that: The step of tracking the motion of the target object to obtain the motion data of the target object comprises: Capturing a detection image including the target object by a camera device of the target device; the detection image includes a plurality of continuous image frames; Performing image recognition on each of the image frames to determine the position of the target object in each of the image frames; The motion data of the target object is determined by comprehensively analyzing the position of the target object in each of the image frames.

8. The method for controlling a target device according to claim 1, characterized in that: The target object includes a pupil; and the step of tracking the motion of the target object to obtain the motion data of the target object includes: The pupil is tracked to obtain motion data of the pupil, wherein the motion data includes one or more of gaze point, gaze duration, eye saccade, movement trajectory, and pupil zoom.

9. An electronic device, characterized in that: The electronic device comprises: a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the control method of the target device according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and the program data can be executed to implement the control method of the target device according to any one of claims 1 to 8.

Images