A man-machine interactive communication method and system for a display screen

Abstract

The application provides a man-machine interactive communication method and system of a display screen, which comprises collecting user gesture actions and generating initial gesture action information, pre-processing the obtained gesture action information to obtain advanced gesture action information, processing the obtained advanced gesture action information to obtain final gesture action, searching for a control instruction matched with the final gesture action information in an instruction library, and inputting the control instruction to the display screen. The application divides a video stream into k image frames, divides the k image frames into a plurality of macroblocks, compares the macroblock differences at the same coordinate point according to a threshold, and improves the recognition rate. The application solves the problems of low recognition rate of the existing man-machine interactive method of the display screen, the inability to accurately recognize specific gestures made by the user, and the inability to quickly respond after recognizing the specific gestures made by the user, thereby reducing the man-machine interactive rate of the display screen.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and particularly relates to a human-computer interaction communication method of a display screen and a system thereof. BACKGROUND

[0002] With the rapid development of science and technology, human-computer interaction technology has been applied to many aspects, for example, the human-computer interaction mode of a display screen. At present, in the human-computer interaction of the display screen, a user needs to make a specific gesture, and then a human-computer interaction device identifies the specific gesture, finally inputs a control instruction corresponding to the specific gesture to a control module of the display screen, and finally completes gesture control of the display screen.

[0003] However, the current human-computer interaction technology of the display screen is not high enough in technology, so that the recognition rate of the human-computer interaction device is low when the human-computer interaction is performed, and the specific gesture made by the user cannot be accurately identified. Meanwhile, after the specific gesture made by the user is identified, a response cannot be quickly made, thereby reducing the human-computer interaction rate of the display screen.

[0004] Therefore, it is necessary to provide a human-computer interaction communication method of a display screen and a system thereof to solve the above technical problems. SUMMARY

[0005] To solve the above technical problems, the present application provides a human-computer interaction communication method of a display screen and a system thereof, which solves the problem that the current human-computer interaction method of the display screen has a low recognition rate, cannot accurately identify the specific gesture made by the user, and cannot quickly respond after the specific gesture made by the user is identified, thereby reducing the human-computer interaction rate of the display screen.

[0006] The present application provides a human-computer interaction communication method of a display screen, which comprises the following operation steps:

[0007] Collecting a user gesture action and generating initial gesture action information;

[0008] Preprocessing the obtained gesture action information to obtain advanced gesture action information;

[0009] Performing feature processing on the obtained advanced gesture action information to obtain a final gesture action;

[0010] Searching for a control instruction matched with the final gesture action information in an instruction library, and inputting the control instruction to the display screen.

[0011] Preferably, in the step of collecting a user gesture action and generating initial gesture action information, the following steps are included:

[0012] Obtaining a video stream of the user gesture action through a camera device;

[0013] extracting k image frames from the video stream according to a set time interval Δt, wherein k≥1 and is a positive integer;

[0014] sequentially extracting gesture-related information from the k image frames to generate initial gesture action information.

[0015] Preferably, in the step of sequentially extracting gesture-related information from the k image frames, the following steps are included:

[0016] establishing a background model to generate a mask image with the same size as a single image frame;

[0017] determining gesture images corresponding to the k image frames through the mask image, and then removing the background of each image frame using the background model to obtain k gesture images corresponding to the k image frames.

[0018] Preferably, in the step of preprocessing the obtained gesture action information to obtain advanced gesture action information, the following steps are included:

[0019] performing desaturation processing on all obtained image frames to reduce the data volume of the gesture action information;

[0020] performing image enhancement processing on the desaturated image frames to enhance the image features of the gesture-related information in the image frames;

[0021] performing denoising processing on the image frames after the image enhancement processing to reduce the interference of various noises, and finally obtaining advanced gesture action information containing the k image frames.

[0022] Preferably, the desaturation processing on the obtained gesture action information includes at least one or a combination of multiple of the following processing modes: grayscale processing or binary processing of the image frames.

[0023] Preferably, the image enhancement processing on the desaturated image frames includes at least one or a combination of multiple of the following processing modes: grayscale transformation, histogram correction, image smoothing, or image sharpening of the image frames.

[0024] Preferably, in the step of performing feature processing on the obtained advanced gesture action information, the following steps are included:

[0025] segmenting each image frame to divide it into m rows and n columns of non-overlapping macroblocks;

[0026] extracting features from the k image frames in the advanced gesture information to obtain feature information of each macroblock;

[0027] searching a gesture model library to match a gesture model with the greatest correlation to the feature information, and determining final gesture action information.

[0028] Preferably, the gesture model library includes several gesture models corresponding to all control instructions in the instruction library.

[0029] Preferably, in the step of searching the gesture model library and matching the gesture model with the greatest correlation with the feature information, the step includes:

[0030] Searching all gesture models in the gesture model library that meet the threshold condition by setting the threshold value;

[0031] Comparing all threshold values to obtain the gesture model with the smallest threshold value, and finally determining the final gesture action information.

[0032] Based on the above communication method, the application also provides a display screen human-computer interaction communication system, which includes a collection module, a preprocessing module, a feature processing module and a gesture matching module, wherein,

[0033] The collection module is used to collect user gesture actions and generate initial gesture action information.

[0034] The preprocessing module is used to preprocess the obtained gesture action information to obtain advanced gesture action information.

[0035] The feature processing module is used to perform feature processing on the obtained advanced gesture action information to obtain the final gesture action.

[0036] The gesture matching module is used to search the instruction library for control instructions matching the final gesture action information and input the control instructions to the display screen.

[0037] Compared with the related art, the display screen human-computer interaction communication method and system provided by the application have the following beneficial effects:

[0038] The application sets the collection module, the preprocessing module, the feature processing module and the gesture matching module, collects initial gesture action information through the collection module, optimizes the initial gesture action information through the preprocessing module and the feature processing module, obtains the final gesture action information through the gesture matching module, and finally searches for control instructions matching the final gesture action information to instruct the display screen to move, thereby solving the problem of low recognition rate of the existing display screen human-computer interaction method, which cannot accurately recognize the specific gesture made by the user, and simultaneously solving the problem of slow response after recognizing the specific gesture made by the user, thereby reducing the human-computer interaction rate of the display screen. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The application provides a display screen human-computer interaction communication method and system, and the method steps are shown in the figure.

[0040] Figure 2A system block diagram of a system of a display screen human-computer interaction communication method and system provided by the application is shown in the figure;

[0041] Figure 3 A video stream diagram of a display screen human-computer interaction communication method and system provided by the application is shown in the figure. DETAILED DESCRIPTION

[0042] The application will be further described below in combination with the drawings and embodiments.

[0043] The application provides a display screen human-computer interaction communication method, which, as shown in the figure, specifically comprises the following operation steps: Figure 1

[0044] Step 1: Collecting user gesture actions and generating initial gesture action information;

[0045] In Step 1, specifically comprising:

[0046] Firstly, acquiring a video stream of user gesture actions by a camera device, which is installed on the display screen, and the optimal installation position is the middle of the upper edge of the display screen, and the camera device can capture gestures in a set area,

[0047] In order to reduce the power consumption of the camera device, that is, to reduce the time (when there is no gesture) of capturing by the camera device, the camera device is also equipped with a sensor for detecting whether a human body is located in the set area, that is, a pyroelectric infrared sensor, and in the time when there is no gesture, the pyroelectric infrared sensor cannot detect the presence of a human body, and in this time, the camera device is not started; if it is detected that a human body is located in the set area, the camera device is started, which represents that the user needs to input a signal to the display screen through a gesture at this time,

[0048] In addition, in order to avoid the false action of the pyroelectric infrared sensor, a time parameter needs to be set, that is, whether the time parameter of detecting that a human body is located in the set area is greater than the set time parameter, if yes, an action signal is output, otherwise, no; in order to meet the diversified needs, the camera device can be closed for a long time when capturing gestures;

[0049] Secondly, k image frames are extracted from the video stream according to a set time interval Δt, wherein k≥1 and is a positive integer, for example: the set time interval Δt is 0.1s, at this time, the time axis of the acquired video stream is independently separated according to 0.1s as an interval, thereby forming a plurality of image frames arranged according to the time axis, if the time axis is 1s, at this time, k=10, that is, 10 image frames distributed according to the time axis are extracted from the acquired video stream, and the image frames are processed subsequently;

[0050] ​Finally, the information related to the gesture in the k image frames is extracted in sequence, the information unrelated to the gesture is removed, the calculation amount is reduced, the system power consumption is reduced, the initial gesture action information is generated from the information related to the gesture, and in this step, the following steps are included:

[0051] First, a background model is established, and a mask image with the same size as a single image frame is generated.

[0052] Secondly, the gesture image corresponding to the k image frames is determined through the mask image, and then the background of each image frame is removed by using the background model to obtain k gesture images corresponding to the k image frames.

[0053] Second step: pre-processing the obtained gesture action information to obtain advanced gesture action information.

[0054] In the second step, the following steps are included:

[0055] First, the obtained all image frames are subjected to desaturation processing, and the desaturation manner includes at least one or a combination of multiple combinations of the gray scale or binary processing manner of the image frame. In the pre-processing process, through the desaturation step, all image frames only have black and white, the number of colors contained in the image frame is reduced, thereby reducing the data amount in the image frame, which is helpful for subsequent rapid processing.

[0056] Secondly, the image frame after desaturation is subjected to image enhancement processing to enhance the image features of the gesture-related information in the image frame, effectively improve the quality of the image frame, and improve the image frame definition, wherein the enhancement manner is one or a combination of multiple combinations of the gray scale transformation, histogram correction, image smoothing or image sharpening processing manner.

[0057] Finally, the image frame after image enhancement processing is subjected to denoising processing to reduce the interference of various noises, and finally the advanced gesture action information containing k image frames is obtained, wherein the denoising manner can be one of spatial domain filtering or frequency domain filtering, and the picture definition of each image frame is improved.

[0058] Third step: performing feature processing on the obtained advanced gesture action information to obtain the final gesture action.

[0059] In the third step, the following steps are included:

[0060] First, as Figure 3As shown, k image frames in the video stream are arranged in time axis, and the k image frames are segmented to divide k non-overlapping m rows n columns macroblocks, it should be noted that each image frame is segmented on the region without background, that is, the area of each macroblock in the image frame is smaller than the area of the image frame, so as to reduce the number of macroblocks, but the area of the macroblock is larger than the actual gesture area, enough to cover the gesture area, and ensure more accurate recognition of gesture information.

[0061] Secondly, feature extraction is performed on the k image frames in the advanced gesture information to obtain feature information of each macroblock, in this step, in order to improve the recognition accuracy, the feature information on each macroblock is extracted;

[0062] Then, the image frames are matched one by one in time axis order, searched in the gesture model library, and the gesture model with the greatest correlation with the feature information is matched to determine the final gesture action information.

[0063] The gesture model library contains many gesture models, each gesture model corresponds to a plurality of standard image frames arranged in time axis, and the standard image frames are pre-recorded, and each standard image frame in the gesture model library also has non-overlapping m rows n columns macroblocks.

[0064] For example, first, the feature information of all macroblocks of the first image frame is extracted, and the standard image frames of all gestures in the gesture model library are searched to match a plurality of standard image frames with the feature information of all macroblocks of the first image frame, then the feature information of all macroblocks in the second image frame is extracted, and on the basis of the first image frame search, the feature matching is continued in all separated standard image frames, until the last gesture is matched in the gesture model library, and the gesture is the final gesture action.

[0065] Meanwhile, in the comparison of macroblocks, the difference between the macroblocks at the same coordinate point in the standard image frame in the gesture model library and the input image frame can be calculated by setting a threshold value, if the difference is less than the threshold value, it is considered that the two macroblocks at the same coordinate point match each other, all macroblocks of each image frame are compared one by one, and the threshold value is determined to be the smallest, and the final gesture action is determined.

[0066] In addition, the gesture model library includes a plurality of gesture models corresponding to all control commands in the command library.

[0067] Fourth step: searching for a control command matched with the final gesture action information in the command library, and inputting the control command to the display screen to finally control the action of the display screen.

[0068] Based on the above communication method, the application also provides a man-machine interactive communication system of a display screen, such as Figure 2As shown, including acquisition module, preprocessing module, feature processing module and gesture matching module, wherein,

[0069] The acquisition module is used for collecting user gesture actions and generating initial gesture action information.

[0070] The preprocessing module is used for preprocessing the acquired gesture action information to obtain advanced gesture action information.

[0071] The feature processing module is used for performing feature processing on the obtained advanced gesture action information to obtain final gesture action.

[0072] The gesture matching module is used for searching the control instruction matched with the final gesture action information in the instruction library and inputting the control instruction to the display screen.

[0073] The working principle of the human-computer interaction communication method and system of the display screen provided by the application is as follows: through the acquisition module, the preprocessing module, the feature processing module and the gesture matching module, the initial gesture action information is collected by the acquisition module, and is optimized by the preprocessing module and the feature processing module, the final gesture action information is obtained by using the gesture matching module, and finally the matched control instruction is searched to indicate the display screen action, which solves the problems of low recognition rate of the existing human-computer interaction method of the display screen, the specific gesture made by the user cannot be accurately recognized, and after the specific gesture made by the user is recognized, the response cannot be quickly made, thereby reducing the human-computer interaction rate of the display screen.

[0074] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a machine that implements the function specified in the flowchart and / or block diagram. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The function specified in one flow or multiple flows and / or blocks

[0075] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0076] The foregoing is merely illustrative of the embodiments of this application and is not intended to limit the scope of the application. Variations and modifications thereof can be made by those skilled in the art without departing from the spirit or scope of the application. Any equivalent embodiments within the spirit and scope of the application are to be included as well. Therefore, the above description should not be construed as limiting, but merely as exemplification of the presently preferred embodiments. Other arrangements, which are not specifically disclosed herein, can be devised and will be encompassed within the spirit and scope of the present application as defined by the claims.

[0077] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the innovation falling outside the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of human-machine interactive communication with a display screen, characterized in that, The communication method comprises the following operation steps: Collecting user gesture actions and generating initial gesture action information; Pretreating the obtained gesture action information to obtain advanced gesture action information; Feature processing the obtained advanced gesture action information to obtain final gesture action; Searching the instruction library for control instructions matching the final gesture action information and inputting the control instructions to the display screen; In the step of sequentially extracting gesture-related information in k image frames, comprising: Establishing a background model to generate a mask image of the same size as a single image frame; Determining the corresponding gesture image in the k image frames through the mask image, and then removing the background of each image frame using the background model to obtain k gesture images corresponding to the k image frames; In the step of pretreating the obtained gesture action information to obtain advanced gesture action information, comprising: Desaturation processing all obtained image frames to reduce the data volume of gesture action information; Image enhancement processing the desaturated image frames to enhance the image features of gesture-related information in the image frames; Denoising the image frames after image enhancement processing to reduce the interference of various noises, and finally obtaining advanced gesture action information containing k image frames; In the step of feature processing the obtained advanced gesture action information, comprising: Segmenting each image frame to divide it into a plurality of non-overlapping m rows and n columns of macroblocks; Feature extraction of the k image frames in the advanced gesture information to obtain feature information of each macroblock; Searching the gesture model library to match the gesture model with the greatest correlation to the feature information to determine the final gesture action information.

2. The method of claim 1, wherein, In the step of collecting user gesture actions and generating initial gesture action information, comprising: Obtaining a video stream of user gesture actions through a camera device; Extracting k image frames from the video stream at a set time interval Δt, where k≥1 and is a positive integer; Sequentially extracting gesture-related information in the k image frames to finally generate initial gesture action information.

3. The method of claim 2, wherein, The desaturation processing of the obtained gesture action information at least includes one or more combinations of gray scale or binary processing of the image frames.

4. The method of claim 3, wherein, The image enhancement processing of the desaturated image frames at least includes one or more combinations of gray scale transformation, histogram correction, image smoothing, or image sharpening processing of the image frames.

5. The method of claim 4, wherein, The gesture model library includes a plurality of gesture models corresponding one-to-one to all control instructions in the instruction library.

6. The method of claim 5, wherein, In the step of searching the gesture model library to match the gesture model with the greatest correlation to the feature information, comprising: Searching all gesture models in the gesture model library that meet the threshold value condition by setting the threshold value; Comparing all threshold values to obtain the gesture model with the smallest threshold value, and finally determining the final gesture action information.

7. The communication system of a man-machine interactive communication method for a display screen according to any one of claims 1-6, characterized in that, It comprises a collection module, a pretreatment module, a feature processing module, and a gesture matching module, wherein: The collection module is used to collect user gesture actions and generate initial gesture action information; The pretreatment module is used to pretreat the obtained gesture action information to obtain advanced gesture action information; The feature processing module is configured to perform feature processing on the obtained advanced gesture action information to obtain a final gesture action. The gesture matching module is configured to search for a control instruction matched with the final gesture action information in an instruction library, and input the control instruction to the display screen.

Images