A text display method, device, set-top box and medium

Abstract

The present invention discloses a text display method, device, set-top box and medium. The method is applied to a set-top box, comprising: obtaining at least one frame of video image in a video data stream; processing each of the video images to obtain a target gesture set; identifying the target gesture set according to a generative adversarial network model to obtain text corresponding to the target gesture set; and displaying the text corresponding to the target gesture set. Using this method, the target gesture set is obtained by processing each acquired video image, and then the target gesture set is identified based on the generative adversarial network model, so that the text corresponding to the target gesture set can be accurately obtained, thereby improving the accuracy of text display and thus enhancing the interactive experience.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a text display method, device, set-top box and medium. Background Art

[0002] In recent years, a remote live interactive consultation method has emerged through consultation apps. The consultation app can include intelligent gesture assistance, that is, the text obtained by gesture recognition is displayed on the screen to assist interactive communication during the consultation process.

[0003] For intelligent gesture assistance, the existing technical solutions mainly include two methods. The first method requires wearing specific gloves on the hands to recognize the text corresponding to the gesture. The second method uses the information entropy algorithm to segment the hand from the background image, extract the target image and apply it to the video data stream to recognize the text corresponding to the gesture. However, in the existing technical solutions, only a small number of gestures can be recognized, and the recognition accuracy of Chinese character gesture strokes is low, which affects the user interaction experience. Summary of the invention

[0004] The present invention provides a text display method, device, set-top box and medium to improve the accuracy of text display and thus enhance the interactive experience.

[0005] According to one aspect of the present invention, there is provided a text display method comprising:

[0006] Acquire at least one frame of video image in the video data stream;

[0007] Processing each of the video images to obtain a target gesture set;

[0008] The target gesture set is identified according to the generative adversarial network model to obtain the text corresponding to the target gesture set;

[0009] Display the text corresponding to the target gesture set.

[0010] According to another aspect of the present invention, there is provided a text display device, comprising:

[0011] An acquisition module, used for acquiring at least one frame of video image in a video data stream;

[0012] A processing module, used for processing each of the video images to obtain a target gesture set;

[0013] A recognition module, used to recognize the target gesture set according to the generative adversarial network model and obtain the text corresponding to the target gesture set;

[0014] The display module is used to display the text corresponding to the target gesture set.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text display method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text display method described in any embodiment of the present invention when executed.

[0020] An embodiment of the present invention provides a text display method, device, set-top box and medium. The method is applied to a set-top box to obtain a target gesture set by processing each acquired video image, and then the target gesture set is recognized based on a generative adversarial network model. The text corresponding to the target gesture set can be accurately obtained, the accuracy of text display is improved, and thus the interactive experience is enhanced.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a text display method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a schematic diagram of a generative adversarial network model structure provided according to Embodiment 1 of the present invention;

[0025] Figure 3 is a flow chart of a text display method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 is a structural schematic diagram of a text display method provided according to Embodiment 2 of the present invention;

[0027] Figure 5 is a structural schematic diagram of a text display device provided according to Embodiment 3 of the present invention;

[0028] Figure 6 It is a structural schematic diagram of a set-top box for implementing the text display method of the first embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 1 is a flowchart of a text display method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of displaying text. The method can be executed by a text display device. The text display device can be implemented in the form of hardware and / or software. The text display device can be configured in a set-top box. Figure 1 As shown, the method includes:

[0033] S110: Acquire at least one frame of video image in a video data stream.

[0034] The video data stream may be considered as video data acquired by a camera in a set-top box, and the video data may be composed of one or more video image frames.

[0035] In this embodiment, after obtaining the video data stream, at least one frame of video image in the video data stream can be obtained. This embodiment does not limit the method of obtaining the video image. For example, the video data stream can be deframed to convert the video data into one or more frames of video images.

[0036] S120: Process each of the video images to obtain a target gesture set.

[0037] The target gesture set may be understood as a state set of gestures represented by each video image. The target gesture set may include a state set of gestures from the beginning to the end, such as a start state set.

[0038] After obtaining one or more video frames, the video images need to be processed so as to identify the gestures in the video images to obtain a target gesture set. In this step, the means of obtaining the target gesture set is not limited. For example, the gesture image corresponding to each frame of the video image can be extracted, and then each gesture image can be segmented and processed to obtain the target gesture set. For example, the first frame of the video image and the last frame of the gesture can be obtained according to the timestamp of each video image, and then the two frames and the frames between the two frames are respectively identified to obtain the target gesture set. This embodiment does not limit this.

[0039] S130: Identify the target gesture set according to the generative adversarial network model to obtain text corresponding to the target gesture set.

[0040] In this embodiment, the generative adversarial network model can be considered as a deep learning model used to identify a target gesture set, wherein the specific number of layers of the generative adversarial network model and the model training process are not limited. For example, in the model training stage, tens of thousands of video images can be used as the initial data for model training, and the Adam optimization algorithm can be used for training.

[0041] Specifically, the target gesture set can be recognized according to the generative adversarial network model to obtain the text corresponding to the target gesture set, wherein the specific steps of recognition are not limited, as long as the text corresponding to the target gesture set can be obtained, and will not be further expanded here.

[0042] S140: Display the text corresponding to the target gesture set.

[0043] After obtaining the text corresponding to the target gesture set, the text may be displayed to assist the user's interactive communication. This embodiment does not limit the position of the displayed text, etc., which may be set by the system or relevant personnel.

[0044] A text display method provided in a first embodiment of the present invention obtains at least one frame of video image in a video data stream; processes each of the video images to obtain a target gesture set; identifies the target gesture set according to a generative adversarial network model to obtain text corresponding to the target gesture set; and displays the text corresponding to the target gesture set. Using this method, a target gesture set is obtained by processing each of the obtained video images, and then the target gesture set is identified based on a generative adversarial network model, so that the text corresponding to the target gesture set can be accurately obtained, thereby improving the accuracy of text display and thus enhancing the interactive experience.

[0045] In one embodiment, the identifying the target gesture set according to the generative adversarial network model to obtain the text corresponding to the target gesture set includes:

[0046] Use the generative adversarial network model to recognize the target gesture set and output dynamic gesture images;

[0047] The dynamic gesture image is recognized based on a preset neural network model to obtain text corresponding to the target gesture set.

[0048] Among them, the dynamic gesture image can be considered as a dynamic image corresponding to the gesture, which can be used to determine the text represented by the gesture. The preset neural network model can be understood as a model for recognizing dynamic gesture images to obtain corresponding text. Before recognition, the preset neural network model can be trained in advance. For example, the preset neural network model can be trained based on a reference data set, and the reference data set can include a number of digital characters, symbols, and Chinese characters.

[0049] Specifically, the generative adversarial network model can first be used to identify each set in the target gesture set to output a dynamic gesture image corresponding to the target gesture set; then the dynamic gesture image output is recognized based on a preset neural network model to obtain the corresponding text.

[0050] In one embodiment, the generative adversarial network model comprises multiple layers, including a convolutional layer, a pooling layer, a convolutional layer, a pooling layer, a fully connected layer, and a fully connected layer in sequence.

[0051] In this embodiment, in order to ensure that the integrated software development kit (SDK) is not too large, the design can be performed on the original AlexNet model to obtain the generative adversarial network model required in this embodiment.

[0052] The generative adversarial network model may include multiple layers, which may include convolutional layer, pooling layer, convolutional layer, pooling layer, fully connected layer and fully connected layer in sequence. The specific number of layers is not limited here, such as 6 layers. This step does not limit the parameters and activation functions of each layer.

[0053] For example, Figure 2 is a schematic diagram of a generative adversarial network model structure provided according to the first embodiment of the present invention, such as Figure 2 As shown, the first layer is a convolutional layer. In this layer, the target gesture set of the original image (i.e., each video image) can be input. The size is 120*120*1, the size of the convolution filter can be 50*50, the depth is 32, and the activation function can use the ReLU function.

[0054] The second layer is a pooling layer, the filter size is 30*30, the step size can be 5, and the complement is all 0. The third layer can be a convolution layer, the convolution filter size is 60*60, the depth is 64, and the activation function uses the Sigmoid function. The fourth layer can be a pooling layer, the filter size is 25*25, the step size can be 5, and the complement is all 0.

[0055] The last two layers are the fully connected layer and the fully connected layer respectively. In addition, the middle Flatten layer is used to average the input data, turning the multi-dimensional input into a one-dimensional input for conversion to the fully connected layer.

[0056] Exemplarily, in the process of training the generative adversarial network model, Adam can also be used as an optimizer, and 104,500 gesture pictures (i.e., gesture images) can be used as model training data, and then the default parameters can be set to lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0.

[0057] Embodiment 2

[0058] Figure 3 It is a flowchart of a text display method provided according to the second embodiment of the present invention, and the second embodiment is optimized on the basis of the above embodiments. In this embodiment, each of the video images is processed to obtain a target gesture set, which is further specified as follows: for each frame of video image, a gesture image corresponding to the video image is extracted, and the gesture image includes a bounding box; each of the gesture images is processed according to a gesture segmentation algorithm and a dynamic time planning algorithm to obtain a target gesture set, and the target gesture set includes a start state set and an end state set.

[0059] like Figure 3 As shown, the method includes:

[0060] S210: Obtain at least one frame of video image in a video data stream.

[0061] S220 . For each frame of video image, extract a gesture image corresponding to the video image, where the gesture image includes a boundary box.

[0062] In this step, the gesture image in each frame of the video image can be first extracted, and then each gesture image is processed to obtain the target gesture set. There is no limit to the means of extracting the gesture image. For example, the edge of the hand in the video image can be first identified, and then the edge can be connected to obtain the corresponding gesture image; the gesture image can also be extracted based on the difference between the skin color and the background in the skin color model.

[0063] S230. Process each of the gesture images according to a gesture segmentation algorithm and a dynamic time planning algorithm to obtain a target gesture set, where the target gesture set includes a start state set and an end state set.

[0064] In this embodiment, the target gesture set may include a start state set and an end state set, the start state set may be a state set when the gesture starts, and the end state set may be a state set when the gesture ends. The gesture segmentation algorithm may be considered as an algorithm for segmenting a gesture image into multiple gesture sets, and the dynamic time planning algorithm may be used to ensure the consistency of the gesture set capture time and achieve time "alignment".

[0065] Specifically, after extracting each gesture image, each gesture image can be processed according to the gesture segmentation algorithm and the dynamic time planning algorithm to obtain a start state set and an end state set. For example, while performing gesture segmentation on each gesture image based on the gesture segmentation algorithm, the segmented images can be sorted in time to obtain a start state set and an end state set; or each gesture image can be firstly segmented based on the gesture segmentation algorithm, and then the segmented gesture set can be processed based on the dynamic time planning algorithm to obtain a start state set and an end state set.

[0066] S240: Identify the target gesture set according to the generative adversarial network model to obtain text corresponding to the target gesture set.

[0067] S250: Display the text corresponding to the target gesture set.

[0068] A second embodiment of the present invention provides a method for displaying text, which obtains at least one frame of video image in a video data stream; for each frame of video image, extracts a gesture image corresponding to the video image, wherein the gesture image includes a boundary box; processes each of the gesture images according to a gesture segmentation algorithm and a dynamic time planning algorithm to obtain a target gesture set, wherein the target gesture set includes a start state set and an end state set; identifies the target gesture set according to a generative adversarial network model to obtain text corresponding to the target gesture set; and displays the text corresponding to the target gesture set. Using this method, by extracting the gesture image corresponding to each frame of video image, a guarantee is provided for the precise segmentation of subsequent gestures. At the same time, each gesture image is processed according to the gesture segmentation algorithm and the dynamic time planning algorithm to improve the accuracy of gesture recognition, thereby making the displayed text more accurate.

[0069] In one embodiment, extracting the gesture image corresponding to each frame of video image includes:

[0070] For each frame of video image, the video image is preprocessed using the HSV spatial edge algorithm;

[0071] Inferring the specific range of the gesture in the preprocessed video image using the gesture recognition probability corresponding to the preprocessed video image;

[0072] Extracting a gesture image from the preprocessed video image according to the specific range using an HSV spatial edge algorithm;

[0073] A bounding box is drawn for the gesture image based on a minimum bounding box algorithm.

[0074] Among them, the HSV spatial edge algorithm may refer to an algorithm that utilizes the discontinuous nature of grayscale values ​​to segment the target area based on grayscale mutations. In this embodiment, the target area may be an area containing the edge of the gesture, that is, a preprocessed video image.

[0075] The gesture recognition probability can be considered as the probability that the gesture indicated in the preprocessed video image is the target gesture. The target gesture can be a standard gesture in a reference gesture library. The reference gesture library can be a set of standard gestures pre-set by relevant personnel to determine the gesture recognition probability. The minimum bounding box algorithm can be used to draw the bounding box.

[0076] In this embodiment, for each frame of video image, the HSV spatial edge algorithm can be used to preprocess the video image to obtain a preprocessed video image; the specific range of the gesture in the preprocessed video image is inferred based on the gesture recognition probability corresponding to the preprocessed video image; then the HSV spatial edge algorithm is used to extract the gesture image in the preprocessed video image according to the specific range of the gesture, and finally the minimum bounding box algorithm is used to draw a bounding box for the gesture image to complete the extraction of the gesture image.

[0077] The gesture recognition probability corresponding to the preprocessed video image can be determined based on the reference gesture library. The specific determination method is not limited, as long as the gesture recognition probability can be obtained.

[0078] In one embodiment, the gesture recognition probability is obtained by evaluating a coordinate pair based on a reference gesture library, wherein the coordinate pair is formed by coordinates of two joint points on the same arm in the video image.

[0079] First, it is necessary to obtain the coordinate pair. The coordinate pair can be considered as a coordinate pair formed by the coordinates of two joint points on the same arm in a frame of video image, which can be used to obtain the gesture recognition probability. The specific position of the joint point is not limited, for example, it can be a finger, elbow, wrist, etc.

[0080] In this embodiment, the coordinate pairs in the preprocessed video image can be obtained for subsequent steps. This embodiment does not limit the specific acquisition means. For example, the joint points in the preprocessed video image can be identified, and then the coordinates of two joint points on the same arm are taken separately to form a coordinate pair.

[0081] Furthermore, the gesture recognition probability can be obtained by evaluating the coordinate pair according to the standard gesture in the reference gesture library. For example, the minimum angle between the vector formed by the coordinate pair corresponding to the current moment and each standard vector corresponding to each standard gesture can be calculated first, and then the historical minimum angles corresponding to the first set number of coordinate pairs before the current moment are calculated. Finally, the gesture recognition probability can be calculated based on the minimum angle and each historical minimum angle. The first set number can be set by relevant personnel, and this embodiment does not limit this.

[0082] For example, first, the vector angles between the vector formed by the coordinate pair at the current moment and the standard vectors corresponding to the standard gestures can be calculated, and the average value ave(θ r );Then take 9 historical minimum angles θ t,min ,t∈[1,9], based on the minimum angle θ min The minimum variance can be obtained by combining 9 (i.e. the first set number) historical minimum angles Then, according to the calculated average value ave(θ r) and minimum variance To determine the gesture probability

[0083] Finally, we can calculate the gesture probability P ar , the minimum angle is θ min , set the threshold T θ , using the formula To calculate the gesture recognition probability, the gesture probability can be considered as the calculated initial gesture recognition probability. The final gesture recognition probability, that is, the gesture recognition probability, needs to be further determined based on the minimum angle and the set threshold.

[0084] It is understandable that the gesture recognition probability may also be modified in combination with the speech recognition probability, and the speech recognition probability may be considered as a probability determined by analyzing the audio data.

[0085] In one embodiment, the processing of each of the gesture images according to the gesture segmentation algorithm and the dynamic time planning algorithm to obtain a target gesture set includes:

[0086] Segmenting each of the gesture images based on a gesture segmentation algorithm to obtain an initial gesture set;

[0087] The initial gesture set is processed based on a dynamic time planning algorithm to obtain a target gesture set.

[0088] The initial gesture set can be understood as a gesture set initially obtained by segmenting each gesture image, and the consistency of gesture time needs to be ensured according to subsequent time planning.

[0089] In this embodiment, each gesture image can be segmented based on a gesture segmentation algorithm to obtain an initial gesture set consisting of multiple segmented images; then the initial gesture set is processed using a dynamic time planning algorithm to obtain a target gesture set.

[0090] The following is an exemplary description of the text display method provided by the embodiment of the present invention.

[0091] Figure 4 is a structural diagram of a text display method provided according to Embodiment 2 of the present invention. Figure 4 As shown, step 1: first, the HSV spatial edge algorithm in OpenCV can be used to process each frame of video image (i.e., preprocessing), and then the gesture recognition probability is used to infer the specific range of the image gesture, and then the HSV spatial edge algorithm can be used to extract the correct gesture image from the background (i.e., the gesture image in the preprocessed video image), and finally the minimum bounding rectangle (Minimum Bounding Rectangle, MBR) in OpenCV, that is, the minimum bounding box algorithm, is used to draw a bounding box for the gesture image.

[0092] Step 2: Use the gesture segmentation algorithm to segment each gesture image, and the segmented images form a gesture set (i.e., the initial gesture set). Secondly, in order to ensure the consistency of the captured gesture time (time gap), the initial gesture set can be processed based on the dynamic time planning (Dynamic Time Warping, DTW) algorithm, and these gesture sets are divided into a start state set and an end state set (i.e., the target gesture set).

[0093] The trained LCGAN model (i.e., generative adversarial network model) is used to recognize the two sets (i.e., target gesture sets) respectively, and finally a dynamic gesture image in gif format is output.

[0094] Step 3: Input the gif dynamic gesture image output in step 2 into MobileNet (i.e., the preset neural network model) for recognition, and output the recognized text on the screen (i.e., display the text corresponding to the target gesture set). The preset neural network model training process can be: use the Chinese handwriting data set (consisting of 171 alphanumeric characters and symbols and 3,866 Chinese characters) to train in MobileNet.

[0095] Through the above description, it can be found that the embodiment of the present invention can integrate the function of intelligent gesture assistance in the consultation App, which can assist people with pronunciation disabilities to communicate. In addition, with the continuous improvement of hardware conditions in recent years, deep learning has performed outstandingly in solving many problems such as visual recognition, speech recognition, and natural language processing. Therefore, the embodiment of the present invention proposes a gesture recognition method based on a generative adversarial network (LCGAN) in the intelligent gesture assistance function. The deep video image (i.e., the video image in the video data stream) collected by the 4K smart set-top box PTZ high-definition camera is used as the input source, and the gestures in the video image are detected, segmented, and recognized using the LCGAN model. After processing, the recognized text is finally displayed on the screen.

[0096] The following are the main steps of integrating the text display method into the medical consultation app in this embodiment:

[0097] First, configure the dataset. Download the Chinese handwriting dataset, which consists of 171 alphanumeric characters, symbols, and 3,866 Chinese characters. Use the SIFT algorithm to build a feature vector template library for Chinese characters and digital sign language gestures.

[0098] Then, start training the LCGAN model. Specifically, configure the Python environment, install the Tensoflow framework, install and import the torchaudio, omegaconf and soundfile modules, use the load module in the hub module of Tensoflow to specify the feature vector template library, and finally use the glob module to start training the model.

[0099] Finally, turn the model into a dynamic library. Configure the C++ environment, expose the interface that the model needs to input data, package the model into a dynamic library (so library), and integrate the packaged dynamic library into the Android system on the Android side; then use NDK-build to package and compile again, and the Android side directly transmits the video data stream to the algorithm through the interface exposed on the C++ side.

[0100] Embodiment 3

[0101] Figure 5 is a structural schematic diagram of a text display device provided according to Embodiment 3 of the present invention. Figure 5 As shown, the device comprises:

[0102] The acquisition module 310 is used to acquire at least one frame of video image in the video data stream;

[0103] A processing module 320, configured to process each of the video images to obtain a target gesture set;

[0104] A recognition module 330 is used to recognize the target gesture set according to the generative adversarial network model to obtain text corresponding to the target gesture set;

[0105] The display module 340 is used to display the text corresponding to the target gesture set.

[0106] A text display device provided in the third embodiment of the present invention acquires at least one frame of video image in a video data stream through an acquisition module 310; processes each of the video images through a processing module 320 to obtain a target gesture set; recognizes the target gesture set according to a generative adversarial network model through a recognition module 330 to obtain text corresponding to the target gesture set; and displays the text corresponding to the target gesture set through a display module 340. With this device, the target gesture set is obtained by processing each acquired video image, and then the target gesture set is recognized based on a generative adversarial network model, so that the text corresponding to the target gesture set can be accurately obtained, thereby improving the accuracy of text display and thus enhancing the interactive experience.

[0107] Optionally, the processing module 320 includes:

[0108] An extraction unit, configured to extract a gesture image corresponding to each frame of the video image, wherein the gesture image includes a boundary box;

[0109] The processing unit is used to process each of the gesture images according to the gesture segmentation algorithm and the dynamic time planning algorithm to obtain a target gesture set, wherein the target gesture set includes a start state set and an end state set.

[0110] Optionally, the extraction unit comprises:

[0111] For each frame of video image, the video image is preprocessed using the HSV spatial edge algorithm;

[0112] Inferring the specific range of the gesture in the preprocessed video image using the gesture recognition probability corresponding to the preprocessed video image;

[0113] Extracting a gesture image from the preprocessed video image according to the specific range using an HSV spatial edge algorithm;

[0114] A bounding box is drawn for the gesture image based on a minimum bounding box algorithm.

[0115] Optionally, the gesture recognition probability is obtained by evaluating a coordinate pair according to a reference gesture library, where the coordinate pair is formed by coordinates of two joint points on the same arm in the video image.

[0116] Optionally, the identification module 330 includes:

[0117] Use the generative adversarial network model to recognize the target gesture set and output dynamic gesture images;

[0118] The dynamic gesture image is recognized based on a preset neural network model to obtain text corresponding to the target gesture set.

[0119] Optionally, the processing unit includes:

[0120] Segmenting each of the gesture images based on a gesture segmentation algorithm to obtain an initial gesture set;

[0121] The initial gesture set is processed based on a dynamic time planning algorithm to obtain a target gesture set.

[0122] Optionally, the generative adversarial network model comprises multiple layers, including, in sequence, a convolutional layer, a pooling layer, a convolutional layer, a pooling layer, a fully connected layer, and a fully connected layer.

[0123] The text display device provided in the embodiment of the present invention can execute the text display method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0124] Embodiment 4

[0125] Figure 6 is a schematic diagram of the structure of a set-top box that implements the text display method of the first embodiment of the present invention. Figure 6 As shown, the set-top box provided by the fourth embodiment of the present invention includes: a camera 1; and a controller 2, which is in communication connection with the camera 1.

[0126] The controller 2 includes: at least one processor 21; and a storage device 22 that is communicatively connected to the at least one processor 21; the processor 21 in the controller 2 may be one or more processors. Figure 6 A processor 21 is taken as an example; the storage device 22 is used to store one or more programs; the one or more programs are executed by the one or more processors 21, so that the one or more processors 21 implement the text display method as described in any one of the embodiments of the present invention.

[0127] The processor 21 and the storage device 22 in the set-top box may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0128] The storage device 22 in the set-top box is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the text display method provided in Embodiment 1 or Embodiment 2 of the present invention (for example, the attached Figure 5 The modules in the text display device shown include: an acquisition module 310, a processing module 320, a recognition module 330, and a display module 340). The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the storage device 22, that is, the text display method in the above method embodiment is implemented.

[0129] The storage device 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 22 may further include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] Furthermore, when one or more programs included in the controller 2 are executed by the one or more processors 21, the program performs the following operations:

[0131] Acquire at least one frame of video image in the video data stream;

[0132] Processing each of the video images to obtain a target gesture set;

[0133] The target gesture set is identified according to the generative adversarial network model to obtain the text corresponding to the target gesture set;

[0134] Display the text corresponding to the target gesture set.

[0135] Embodiment 5

[0136] Embodiment 5 of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it is used to execute a text display method, and the method includes:

[0137] Acquire at least one frame of video image in the video data stream;

[0138] Processing each of the video images to obtain a target gesture set;

[0139] The target gesture set is identified according to the generative adversarial network model to obtain the text corresponding to the target gesture set;

[0140] Display the text corresponding to the target gesture set.

[0141] Optionally, when the program is executed by a processor, it can also be used to execute the text display method provided by any embodiment of the present invention.

[0142] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device.

[0143] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the above.

[0145] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A text display method, characterized in that: Applied to a set-top box, the method comprises: Acquire at least one frame of video image in the video data stream; Processing each of the video images to obtain a target gesture set; Recognize the target gesture set according to a generative adversarial network model to obtain text corresponding to the target gesture set; Displaying the text corresponding to the target gesture set; The step of processing each of the video images to obtain a target gesture set includes: For each frame of video image, extracting a gesture image corresponding to the video image; Processing each of the gesture images according to a gesture segmentation algorithm and a dynamic time planning algorithm to obtain the target gesture set, wherein the target gesture set includes a start state set and an end state set; The step of identifying the target gesture set according to the generative adversarial network model to obtain text corresponding to the target gesture set includes: Using a generative adversarial network model to recognize the target gesture set and output a dynamic gesture picture; The dynamic gesture image is recognized based on a preset neural network model to obtain text corresponding to the target gesture set.

2. The method according to claim 1, characterized in that The gesture image includes a bounding box.

3. The method according to claim 1, characterized in that The step of extracting the gesture image corresponding to each frame of the video image includes: For each frame of video image, the video image is preprocessed using the HSV spatial edge algorithm; Inferring the specific range of the gesture in the preprocessed video image using the gesture recognition probability corresponding to the preprocessed video image; Extracting a gesture image from the preprocessed video image according to the specific range using an HSV spatial edge algorithm; A bounding box is drawn for the gesture image based on a minimum bounding box algorithm.

4. The method according to claim 3, characterized in that The gesture recognition probability is obtained by evaluating a coordinate pair based on a reference gesture library, wherein the coordinate pair is formed by the coordinates of two joint points on the same arm in the video image.

5. The method according to claim 1, characterized in that The step of processing each of the gesture images according to the gesture segmentation algorithm and the dynamic time planning algorithm to obtain a target gesture set includes: Segmenting each of the gesture images based on a gesture segmentation algorithm to obtain an initial gesture set; The initial gesture set is processed based on a dynamic time planning algorithm to obtain a target gesture set.

6. The method according to claim 1, characterized in that The generative adversarial network model comprises multiple layers, including a convolutional layer, a pooling layer, a convolutional layer, a pooling layer, a fully connected layer and a fully connected layer in sequence.

7. A text display device, characterized in that: include: An acquisition module, used for acquiring at least one frame of video image in a video data stream; A processing module, used for processing each of the video images to obtain a target gesture set; A recognition module, used to recognize the target gesture set according to a generative adversarial network model to obtain text corresponding to the target gesture set; A display module, used to display the text corresponding to the target gesture set; Wherein, the processing module includes: An extraction unit, configured to extract a gesture image corresponding to each frame of the video image; A processing unit, configured to process each of the gesture images according to a gesture segmentation algorithm and a dynamic time planning algorithm to obtain the target gesture set, wherein the target gesture set includes a start state set and an end state set; Among them, the recognition module is specifically used to use a generative adversarial network model to recognize the target gesture set and output a dynamic gesture picture; based on a preset neural network model, the dynamic gesture picture is recognized to obtain the text corresponding to the target gesture set.

8. A set-top box, characterized in that: The set-top box comprises: Camera; A controller is connected to the camera for communication, and the controller includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text display method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text display method according to any one of claims 1 to 6 when executed.

Images