Display terminal control method, display terminal and computer readable storage medium

[0042] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0043] The present invention provides a display terminal control method, referring to figure 1 , figure 1 It is a schematic flowchart of the first embodiment of the display terminal control method of the present invention, the method includes steps:

[0044] Step S10, acquiring user image information, and performing gesture recognition on the user image information to acquire gestures to be executed;

[0045] The user image information may be acquired through the camera of the display terminal, and before gesture recognition is performed on the user image information, the user image information may be preprocessed to improve the accuracy of gesture recognition.

[0046] Step S20, matching a preset gesture corresponding to the gesture to be executed in a preset gesture feature library, and acquiring a preset instruction corresponding to the preset gesture;

[0047] The preset gesture feature library contains a plurality of preset gestures. The preset gestures are set when the display terminal leaves the factory. Each preset gesture corresponds to a preset instruction. After setting, the user can change the preset command corresponding to the preset gesture according to his own needs. Furthermore, users can add preset gestures according to their own needs, and assign corresponding preset instructions to them.

[0048] Step S30, executing an operation corresponding to the preset instruction.

[0049] After acquiring the preset instruction corresponding to the preset gesture, perform the operation corresponding to the preset instruction, the preset instruction may include an instruction to open or close the corresponding application, commonly used applications include Netflix, youtube and prime video, etc., can also include basic operations on the display terminal, such as shutdown, volume adjustment, and channel change. The preset instructions in this embodiment can be set as any operations that the display terminal can perform. For specific preset instructions settings, this application does not limit.

[0050] In this embodiment, a corresponding operation is performed by recognizing a user's gesture, so that the display terminal can be controlled without an external device, which is convenient for the user.

[0051] Further, see figure 2 , in the second embodiment of the display terminal control method of the present invention proposed based on the first embodiment of the present invention, the step S10 includes steps:

[0052] Step S11, performing skin color detection on the user image information to obtain a binarized image based on the user's skin color;

[0053] Step S12, performing gesture recognition on the user image according to the binarized image;

[0054] Step S13, generating a gesture to be executed according to the gesture recognition result.

[0055] Skin color is one of the most intuitive features on the surface of the human body. By detecting the skin color, the human body can be well distinguished from the surrounding environment. In this embodiment, the ellipse skin color detection model is used, that is, each pixel in the user image information Mapped to the YCrCb space, where Y represents the brightness, that is, the gray scale value; Cr represents the chroma, and Cb represents the saturation, and then judges whether each pixel is in the skin color area in the YCrCb space, if the pixel is in the YCrCb space skin color area, then mark this pixel as a skin color pixel, and set this pixel to white, if the pixel is not in the skin color area in YCrCb space, then mark this pixel as an environment pixel, and set this pixel Set it to black. After judging all the pixels of the user image, a black and white image based on the user's skin color can be obtained, that is, a binarized image. The white area is the skin color area, and the black area is the environment area. After the binarized image based on the user's skin color is acquired, gesture recognition is performed based on the binarized image to obtain a gesture recognition result, and the gesture recognition result is used as a gesture to be executed.

[0056]In this embodiment, the accuracy of gesture recognition can be improved by acquiring a binarized image based on the user's skin color, and performing gesture recognition on the user image based on the binarized image.

[0057] Further, see image 3 , in the third embodiment of the display terminal control method of the present invention proposed based on the second embodiment of the present invention, the step S21 includes steps:

[0058] Step S211, performing morphological processing on the binarized image, and performing morphological processing on the processed binary image;

[0059] Step S212, performing contour extraction on the binarized image after morphological processing to obtain a gesture contour image;

[0060] Step S213, perform gesture recognition on the user image according to the gesture profile image.

[0061] Morphological processing is used to extract image components that are meaningful to express and describe the shape of the region from the image, so that the subsequent recognition work can capture the most essential shape features of the target object. The steps of the morphological processing are mainly erosion and expansion, and the expansion or erosion operation is to convolve the image with the kernel. Kernel In most cases, the kernel is a small center with a reference point and a solid square or disk. Dilation is an operation to find a local maximum, so that the highlighted area in the image, that is, the skin color area in the binarized image, gradually Growth and erosion are operations for finding local minimum values, so that the highlighted areas in the image are gradually reduced. The noise in the binarized image can be eliminated by the closing operation of first expansion and then erosion, and the holes inside the skin color area of ​​the binary image can be eliminated by the opening operation of first erosion and then expansion.

[0062] After obtaining the binarized image after morphological processing, edge detection and segmentation are performed on the binarized image to extract the contour of the binarized image. The step of contour extraction may also be to subtract the erosion image from the dilated image in the morphological processing to obtain the gesture contour. This embodiment does not limit the specific contour extraction method.

[0063] In this embodiment, the gesture contour image is obtained by performing morphological processing and contour extraction on the binarized image, and then gesture recognition is performed on the user image according to the gesture contour image, so that the accuracy of gesture recognition can be improved.

[0064] Further, in the fourth embodiment of the display terminal control method of the present invention proposed based on the third embodiment of the present invention, the step S213 includes steps:

[0065] Step S2131, performing feature extraction on the gesture contour image to obtain a Fourier descriptor corresponding to the gesture contour image;

[0066] Step S2132, input the Fourier descriptor corresponding to the gesture contour image into the trained gesture recognition model, and run the trained gesture recognition module;

[0067] The step of described step S13 comprises:

[0068] Step S131, using the predicted gesture output by the gesture recognition model as the gesture to be executed.

[0069] A Fourier descriptor is an image feature, specifically, a feature parameter used to describe a contour. The basic idea is to use the Fourier transform of the object boundary information as the shape feature, transform the contour feature from the space domain to the frequency domain, and extract the frequency domain information as the feature vector of the image. That is, a vector is used to represent a contour, and the contour is digitized, so that different contours can be better distinguished, and then the purpose of object recognition can be achieved.

[0070] In this embodiment, a corresponding Fourier descriptor is obtained by performing feature extraction on a gesture contour image, and gesture recognition is performed according to the Fourier descriptor, so that the accuracy of gesture recognition can be improved.

[0071] Further, in the fifth embodiment of the display terminal control method of the present invention proposed based on the fourth embodiment of the present invention, steps are included before the step S10:

[0072] Step S40, collecting Fourier descriptors corresponding to different preset gestures as a sample library;

[0073] In step S50, the gesture recognition model to be trained is trained according to the sample database until the training is completed to obtain the trained gesture recognition model.

[0074] see Figure 4 , Figure 4 includes gesture images corresponding to preset gestures 1-10. It should be noted, Figure four The preset gestures listed in are only used in the present invention and are not intended to limit the present invention. Different numbers or shapes of preset gestures can be set according to actual usage requirements.

[0075] The gesture image is collected and saved to the gesture image library, the gesture image can be rotated or translated to expand the gesture image library, and the Fourier descriptor is calculated and saved for the gesture contour line in each photo in the entire gesture image library, and the obtained Existing sample library.

[0076] Further, the step S50 includes the steps of:

[0077] Step S51, input the Fourier descriptor in the sample library into the gesture recognition model to be trained, and output the predicted gesture after running the gesture recognition model;

[0078] Step S52, modifying the parameters of the gesture recognition model to be trained according to the output predicted gesture and the preset gesture corresponding to the input Fourier descriptor;

[0079] Step S51 is executed until the gesture recognition model meets the preset condition, and the gesture recognition model is set as a trained gesture recognition model.

[0080] In this embodiment, the gesture recognition model adopts an SVM (Support Vector Machine, support vector machine) model. Wherein, the process of model training may include: inputting the Fourier descriptor into the gesture recognition model, so that the gesture recognition model outputs the predicted gesture, and increasing the cumulative training times by 1; comparing the Fourier descriptor corresponding to The preset gesture and the predicted gesture to obtain a loss function; adjust the parameters of the gesture recognition model according to the loss function to update the gesture recognition model; determine whether the accumulated training times reach the preset training threshold; Wherein, when the cumulative number of training times reaches the preset training threshold, the training is stopped, and the gesture recognition model when the preset training threshold is reached is used as the gesture recognition model after training; when the cumulative number of training times does not reach the preset training threshold , obtain a new Fourier descriptor and execute the step of inputting the Fourier descriptor into the gesture recognition model. In other embodiments, the model training can also determine whether the training is completed by judging whether the loss function meets the preset requirements. The loss function is used to measure the degree of inconsistency between the predicted value of the model and the real value, that is, the degree of inconsistency between the predicted gesture in this embodiment and the preset gesture corresponding to the Fourier descriptor, which is a non-negative real value function.

[0081] This embodiment presents a complete process of gesture recognition training, so that the trained gesture recognition model meets the needs of actual use.

[0082] Further, in the seventh embodiment of the display terminal control method of the present invention proposed based on the second embodiment of the present invention, the step S13 includes the steps of:

[0083] Step S131, acquiring gesture recognition results within a first preset time, and judging whether the gesture recognition results within the first preset time are consistent;

[0084] Step S132, if yes, generate a gesture to be executed according to the gesture recognition result.

[0085] In actual use, due to the need to continuously acquire user image information and identify it, when the user does not subjectively make control gestures, there may be errors in recognizing user hand movements as preset gestures and performing corresponding preset gestures. The operation of setting instructions will affect the normal use of users. In order to solve this problem, a first preset time is set, and only when the user's hand gesture is recognized as a preset gesture and the first preset time of the preset gesture remains unchanged, the preset operation corresponding to the preset gesture is executed.

[0086] In this embodiment, the preset instruction corresponding to the preset gesture is executed only when the user maintains a preset gesture within the first preset time, so as to avoid misrecognition from affecting the user's use.

[0087] Further, in the eighth embodiment of the display terminal control method of the present invention proposed based on the second embodiment of the present invention, the step S13 includes the steps of:

[0088] Step S131, obtaining the gesture trajectory within the second preset time according to the plurality of gesture recognition results acquired within the second preset time;

[0089] Step S132, using the gesture trajectory as a gesture to be executed.

[0090] When a preset gesture is recognized, start timing. Within the second preset time, each time a preset gesture is recognized, the gesture position is recorded. When the second preset time ends, the recorded gesture positions are sequentially recorded according to the recording time. Connect to obtain the gesture trajectory, and match the corresponding preset instructions in the preset gesture feature library according to the gesture trajectory.

[0091] In this embodiment, by acquiring gesture trajectories within the second preset time period and executing corresponding preset instructions according to the tidying trajectories, the capacity of preset gestures is expanded, enabling the user to use more gestures for shortcut operations.

[0092] refer to Figure 5 In terms of hardware structure, the display terminal may include components such as a communication module 10 , a memory 20 , and a processor 30 . In the display terminal, the processor 30 is respectively connected to the memory 20 and the communication module 10, the memory 20 stores a computer program, and the computer program is executed by the processor 30 at the same time, and the computer The steps in the above method embodiments are realized when the program is executed.

[0093] The communication module 10 can be connected with external communication equipment through the network. The communication module 10 can receive requests from external communication devices, and can also send requests, instructions and information to the external communication devices. The external communication devices can be other display terminals, servers or Internet of Things devices, such as TVs and the like.

[0094] The memory 20 can be used to store software programs and various data. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program required by a function (such as acquiring user image information, performing gesture recognition on the user image information, and Acquire the gesture to be executed), etc.; the storage data area may include a database, and the storage data area may store data or information created according to the use of the system. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

[0095] The processor 30 is the control center of the display terminal. It uses various interfaces and lines to connect various parts of the entire display terminal. By running or executing software programs and/or modules stored in the memory 20, and calling the Data, execute various functions of the display terminal and process data, so as to monitor the display terminal as a whole. The processor 30 may include one or more processing units; optionally, the processor 30 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem The tuner processor mainly handles wireless communication. It can be understood that the modem processor may not be integrated into the processor 30 .

[0096] although Figure 5 Not shown, but the above display terminal may further include a circuit control module, which is used to connect with a power supply to ensure normal operation of other components. Those skilled in the art can understand that, Figure 5The structure of the display terminal shown in the above does not constitute a limitation to the display terminal, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

[0097] The present invention also proposes a computer-readable storage medium on which a computer program is stored. The computer readable storage medium may be Figure 5 The memory 20 in the display terminal can also be at least one of ROM (Read-Only Memory, read-only memory)/RAM (Random Access Memory, random access memory), magnetic disk, optical disk, and the computer can Reading the storage medium includes several instructions to make a terminal device with a processor (which may be a TV, a car, a mobile phone, a computer, a server, a terminal, or a network device, etc.) execute the methods described in various embodiments of the present invention.

[0098] In the present invention, the terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance. Personnel can understand the specific meanings of the above terms in the present invention according to specific situations.

[0099] In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

[0100] Although the embodiments of the present invention have been shown and described above, the protection scope of the present invention is not limited thereto. It can be understood that the above-mentioned embodiments are exemplary and should not be construed as limitations of the present invention. A skilled person may make changes, modifications and substitutions to the above-mentioned embodiments within the scope of the present invention, and these changes, modifications and substitutions shall all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.