Television control method, device and storage medium based on multimodal instructions

Abstract

The present application discloses a TV control method, device and storage medium based on multimodal instructions, which belongs to the technical field of TV set-top boxes. The method includes: obtaining at least one modal data collected by the TV and / or associated devices; determining a fusion instruction based on the context information of the modal data in time sequence; and controlling the TV to execute the control action corresponding to the fusion instruction. The present application collects at least one modal data from the TV and / or associated devices, and then uses the context information of the modal data in time sequence to determine a fusion instruction, so that the system can understand the user's complex intentions. Finally, according to the determined fusion instruction, the system controls the TV to execute the corresponding multiple control actions without the user having to perform voice control and gesture control separately, so that it can respond to multiple types of incoming instructions at the same time sequence, and realize seamless multimodal interaction.

Description

Technical Field

[0001] The present application relates to the technical field of television set-top boxes, and in particular to a television control method, device and storage medium based on multimodal instructions. Background Art

[0002] At present, smart TVs have begun to try to combine multiple input methods to achieve interaction with the TV. In the related technology, by integrating microphone arrays and cameras, basic voice and gesture recognition functions can be realized, and users can perform corresponding operations according to voice commands or gestures. However, the related technology does not classify the command types, but regards all commands as commands that need to be executed and then executes them step by step. Therefore, it is impossible to respond to multiple types of commands that are input at the same time in parallel, causing users to frequently switch input methods.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a TV control method, device and storage medium based on multimodal instructions, aiming to solve the technical problem in the related technology that, by integrating a microphone array and a camera, basic voice and gesture recognition functions can be realized, but multiple types of instructions input simultaneously cannot be responded to in parallel, resulting in the user needing to frequently switch input methods.

[0005] To achieve the above-mentioned objectives, the present application provides a TV control method based on multimodal instructions, the method comprising the following steps: by acquiring at least one modal data collected by a TV and / or associated devices, combining the contextual information of the modal data in historical processing data, which may be historical instruction operation data or user preferences, making adjustments based on this information, determining a fusion instruction that can be output simultaneously, and thereby determining the action performed by the TV based on this fusion instruction.

[0006] In one embodiment, the step of determining the fusion instruction based on the context information of the modality data in time sequence includes: Identify the modal data by a modal recognition engine, and determine the modal instructions corresponding to each modal data; Based on the context information of the modal instruction in time sequence, the modal instruction is processed to obtain an instruction result; The instruction results corresponding to each of the modal instructions are fused to determine the fused instruction.

[0007] In one embodiment, the step of identifying the modal data by a modal recognition engine and determining a modal instruction corresponding to each modal data includes: Receiving a modal signal transmitted by a sensor, preprocessing the modal signal, and determining the modal data; extracting modal features of the modal data; The modal features are identified based on the machine learning algorithm in the modal recognition engine to obtain the corresponding modal instructions.

[0008] In one embodiment, before the step of processing the modal instruction based on the context information of the modal instruction in time sequence to obtain the instruction result, the step includes: Determine user preferences and historical operation instructions based on historical operation data; The semantic features of the modal instructions at the same time sequence are extracted to determine the modal instructions of the same type.

[0009] In one embodiment, the step of processing the modal instruction based on the context information of the modal instruction in time sequence to obtain the instruction result includes: Merging the modal instructions of the same type to determine the total number of target modal instructions; If the total number of target modal instructions includes at least one of the modal instructions, receiving environmental information transmitted by an environmental sensor, and determining the reliability of the modal instruction based on the environmental information; The initial weight of each of the modal instructions is adjusted based on the reliability and historical operation instructions to determine the instruction result.

[0010] In one embodiment, the step of fusing the instruction results corresponding to each of the modal instructions to determine the fused instruction includes: Determining an initial weighted score based on the command result corresponding to the modal identification coefficient; Adjusting the initial weighted score based on historical operation instructions to determine the weighted score; Output the fusion instruction whose weighted score is higher than a preset threshold.

[0011] In one embodiment, the step of obtaining at least one modality data collected by the television and / or associated equipment includes: At least one of the modal data is acquired based on preset sensors in the television and / or associated devices, wherein the preset sensors include a microphone array, a camera, and an environmental sensor.

[0012] In one embodiment, the television control method based on multimodal instructions further includes: Based on the environmental sensor detecting the environmental status, adjust the screen display parameters; Based on the adjusted picture display parameters, the display picture is optimized.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a TV control device based on multimodal instructions, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the TV control method based on multimodal instructions as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the TV control method based on multimodal instructions as described above are implemented.

[0015] One or more technical solutions proposed in the present application have at least the following technical effects: the present application first collects at least one modal data, such as voice, gesture, facial recognition, etc., from a TV and / or associated devices to provide input information for subsequent processing, and then uses the contextual information of the modal data in time sequence for the system to determine a fusion instruction. This instruction integrates the meanings of various modal data so that the system can understand the user's complex intentions. Finally, based on the determined fusion instruction, the system controls the TV to perform corresponding control actions, such as adjusting the volume and switching channels at the same time, without the need for the user to perform voice control and gesture control separately, thereby being able to respond to multiple types of incoming instructions at the same time and achieve seamless multimodal interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 This is a flow chart of a first embodiment of a TV control method based on multi-modal instructions of the present application; Figure 2 This is a flow chart of a second embodiment of a TV control method based on multi-modal instructions of the present application; Figure 3 This is a flow chart of a third embodiment of a TV control method based on multi-modal instructions of the present application; Figure 4 This is a flow chart of a fourth embodiment of a television control method based on multimodal instructions of the present application; Figure 5This is a flowchart of a fifth embodiment of a television control method based on multimodal instructions of the present application; Figure 6 This is a flow chart of a sixth embodiment of a television control method based on multimodal instructions of the present application; Figure 7 This is a flow chart of a seventh embodiment of a television control method based on multimodal instructions of the present application; Figure 8 Demonstrate the overall architecture of the TV control system based on multi-modal instructions of this application; Fig. 9 It is a structural diagram of a television control device based on multimodal instructions in a hardware operating environment involved in an embodiment of the present application.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] It should be noted that Figure 8The overall architecture of a TV control system based on multimodal instructions is displayed, including the components of the perception layer, processing layer, and application layer and their connections. First, data is collected by the perception layer, which is equipped with a microphone array, camera, and sensor. The audio is transmitted to the speech recognition engine through the audio interface of the microphone array, and then recognized by the speech recognition engine and transmitted to the central processing unit through the data interface. After the human movements and images are captured by the camera, they are transmitted to the gesture recognition engine through the video interface to recognize gestures, and then to the facial recognition engine to recognize facial features. After recognition, the computer-understandable instructions are transmitted to the central processing unit through the data interface. Other data, such as ambient temperature, brightness, and volume, are collected by sensors, or data transmitted through other associated devices, such as mobile phones, computers, and smart phones that are connected to the TV control system based on multimodal instructions via Bluetooth. The central processor obtains the recognition results of various modes, processes them at the same time, and sends the processing results to the application layer. The interactive instructions generated by the central processor can be received through the user interface, and then the corresponding interactive results, such as program lists and content recommendations, can be displayed. At the same time, further interactive operations can be performed, the central processor sends the user data to the privacy protection module, and the privacy protection module encrypts and anonymizes the user data, or content management system, such as managing TV content, supports content search, recommendation and other functions, smart home control system, can control smart devices in the home, such as lights, air conditioners, etc., the output form of the interactive instructions is not limited here.

[0023] The multi-interaction mode includes at least one mode of interacting with the TV operation, wherein the interaction modes include but are not limited to infrared remote control, simple voice commands, touch screen control, gesture control, and facial control. In the TV remote control system, display screen control scenarios based on multi-interaction modes are often involved. For example, users can switch channels, adjust volume, etc. through voice commands, and can also turn pages, zoom, etc. through gestures.

[0024] However, when processing data from multiple interactive modes, the multi-interactive display screen control system cannot respond in parallel to multiple types of commands that are simultaneously input due to the lack of effective integration of multiple input modes, resulting in users having to frequently switch input modes during use.

[0025] In order to solve the above problems, the present application proposes a TV control method based on multimodal instructions, which specifically includes: obtaining at least one modal data collected by the TV and / or associated devices; determining a fusion instruction based on the context information of the modal data in time sequence; and controlling the TV to execute a control action corresponding to the fusion instruction. The method collects at least one modal data through the TV and its associated devices. The modal data can be audio data, image data, and sensor data. The instructions for multimodal data fusion are determined by the context information of the modal data in time sequence, and the fused instructions control the TV to execute the corresponding instruction actions. It can respond to multiple types of incoming instructions at the same time sequence to achieve seamless multimodal interaction.

[0026] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0027] Based on this, the first embodiment of the present application provides a television control method based on multi-modal instructions, referring to Figure 1 , Figure 1 This is a flow chart of Embodiment 1 of the television control method based on multimodal instructions of the present application.

[0028] In this embodiment, the television control method based on multimodal instructions includes steps S100 to S300: Step S100, obtaining at least one modality data collected by the television and / or associated equipment.

[0029] It should be noted that the execution subject of this embodiment can be a TV control system based on multimodal instructions, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, etc., and this embodiment does not make specific limitations on this. The following takes the TV control system based on multimodal instructions as an example to illustrate this embodiment and the following embodiments. Modal data can be data collected by various methods or sensors, and can be images, sounds, temperature, humidity or power consumption, etc.; the TV itself is usually equipped with a camera (for video calls or gesture control), a microphone (for voice control or sound collection), an infrared sensor (for collecting surrounding environment information, etc.), and the associated equipment can be a smart speaker, a smart home control system, a remote control, etc., and the modal data and associated equipment are not limited here.

[0030] When the user speaks a command for at least one modal data, the TV and / or associated devices collect at least one modal data and send the collected modal data to the corresponding modal data recognition engine through the data interface, including a speech recognition engine, a gesture recognition engine, and a facial recognition engine.

[0031] Specifically, step S100 includes step S110: Step S110, acquiring at least one of the modal data based on preset sensors in the television and / or associated devices, wherein the preset sensors include a microphone array, a camera, and an environmental sensor.

[0032] It should be noted that the microphone array is used to capture the user's voice commands, which can improve the accuracy and robustness of voice recognition in noisy environments; the camera is used to capture the user's gestures and facial features; environmental sensors include ambient light sensors, infrared sensors, etc., which are used to sense environmental changes, such as light intensity, temperature, etc., and help the system automatically adjust the display effect or interaction strategy according to environmental conditions.

[0033] By pre-setting various sensors in the TV and related devices, the user's input signals are captured and converted into modal data that the system can understand. The modal data may include voice data, gesture data, facial data, etc. These data are used for subsequent processing processes such as voice recognition, gesture recognition and facial recognition.

[0034] As an optional solution for a sensor to collect modal data, when the sensor detects a vibrating gesture, it converts it into an analog signal, and then the analog signal usually needs to be amplified, filtered, and processed by a signal conditioning circuit to improve the signal quality.

[0035] Step S200, determining a fusion instruction based on the temporal context information of the modal data.

[0036] It should be noted that, first of all, for different modal data, it is necessary to perform time alignment on the different modal data to ensure the consistency of the data on the time axis, and then dynamically adjust the modal weight according to the reliability and contextual information of different input methods to ensure that the generated fusion instructions are more accurate and natural.

[0037] As an optional way to dynamically adjust the weights, an initial weight is assigned to each input method (modality) in the initial stage of the multimodal data fusion algorithm. These initial weights can be set based on the following factors: Different input methods have different reliability in different environments. For example, speech recognition is more reliable in a quiet environment, but may not be as accurate in a noisy environment; or the user's usage habits and preferences may also affect the allocation of initial weights. For example, some users may prefer to use gesture control, while others may prefer voice commands; and the system can assign an initial weight to each input method according to the default settings, and these weights can be adjusted later based on actual usage.

[0038] Exemplarily, the initial weight distribution is as follows: voice recognition: 0.4, gesture recognition: 0.3, facial recognition: 0.3. The weight of each input method is adjusted in real time according to the user's current operation and context information to ensure that the generated instructions are more accurate and natural.

[0039] Step S300, controlling the television to execute a control action corresponding to the fusion instruction.

[0040] After the fusion instruction is determined, the fusion instruction is executed to control the television to perform the action corresponding to the fusion instruction.

[0041] It should be noted that actions of different modes can correspond to different command actions. Specifically, users can quickly switch channels, search for content, adjust the volume, etc. through voice commands. For example, users can say "switch to the news channel" or "turn up the volume"; users can control volume, page turning, zooming, and other operations through gestures. For example, users can make an upward waving gesture to turn up the volume, and waving downward to turn down the volume; the system can identify the user's identity and emotional state through facial recognition technology, thereby providing personalized content recommendations and services. For example, the system can judge the user's emotional state based on his or her facial expression. If the user looks tired, the system can recommend relaxing music or videos.

[0042] For example, the user says "switch to the news channel" and makes an upward waving gesture, while his facial expression shows a happy emotion. The modal recognition engine recognizes the command "switch to the news channel", recognizes the command "turn up the volume" based on the gesture, and recognizes the user's emotional state as "happy" based on the facial recognition. Then, the central processing unit integrates the above recognition results according to the user's current operation and context information, and generates the final interactive instruction "switch to the news channel, turn up the volume, and recommend more similar programs", thereby causing the TV to execute the operations of switching to the news channel, turning up the volume, and the system recommending more similar programs.

[0043] This embodiment proposes a television control method based on multimodal instructions, which generates multiple comprehensive control instructions by understanding and responding to the user's multimodal input (such as voice, gestures, facial expressions, etc.) and analyzing the time correlation and context information of these inputs. It can respond to multiple types of incoming instructions at the same time, realize seamless multimodal interaction, and thus provide a more natural, intuitive and rich user experience.

[0044] Based on the first embodiment, the second embodiment of the present application provides a television control method based on multi-modal instructions, referring to Figure 2 , Figure 2 This is a flow chart of Embodiment 2 of the TV control method based on multi-modal instructions of the present application, step S200 includes steps S210 to S230: Step S210: identifying the modal data through a modal recognition engine, and determining a modal instruction corresponding to each modal data.

[0045] It should be noted that the modal recognition engine includes a speech recognition engine, a gesture recognition engine, and a facial recognition engine. The speech recognition engine converts the captured speech signal into text, supports multiple languages ​​and dialects, and then connects to the central processing unit through a data interface to transmit the recognition results to the central processing unit; the gesture recognition engine converts the captured gesture image into a computer-understandable instruction, supports multiple gestures, and then connects to the central processing unit through a data interface to transmit the recognition results to the central processing unit. The facial recognition engine performs identity authentication and sentiment analysis on the captured facial image, supports multiple facial feature recognition, and then connects to the central processing unit through a data interface to transmit the recognition results to the central processing unit. It should be noted that the central processing unit is used to coordinate the work of each engine, process multimodal input information, generate the final interactive instructions, connect to the speech recognition engine, gesture recognition engine, facial recognition engine, user interface, content management system, smart home control system and privacy protection module through a data interface, and coordinate the work of each component.

[0046] After collecting input modal data through sensors, the modal recognition engine extracts features from the modal data, and then identifies the modal features to determine the modal instructions for the action.

[0047] As an optional solution to determine modal features through a modal recognition engine, features are extracted from voice signals, which are then recognized and converted into text or commands. Compared with gesture recognition and facial recognition, features such as edges, corners, color histograms, etc. are extracted from image data, and then the gesture recognition engine analyzes gesture image features and the facial recognition engine analyzes facial image features to identify specific gestures, as well as the user's identity and emotions.

[0048] Exemplarily, the identified modal recognition result is mapped to a predefined instruction set, such as "turn on the TV", "turn up the volume", etc., and the modal instruction is determined by the mapping result.

[0049] Step S220: Process the modal instruction based on the context information of the modal instruction in timing to obtain an instruction result.

[0050] After the corresponding modal instruction is identified, the user's interaction context in each modal instruction may be a previous interaction record, or a modal instruction of the same type executed last time and a similar instruction to be executed thereafter.

[0051] The reliability of the recognized modal instructions is evaluated according to the context. First, the reliability of the modal instructions is determined based on the modal recognition reliability in the environment. Then, the weight adjustment of the modal instructions is determined based on the reliability. For example, in a noisy environment, the modal instructions recognized by speech will have a correspondingly reduced reliability.

[0052] Step S230, fusing the instruction results corresponding to each of the modal instructions to determine the fused instruction.

[0053] After calculating the modal instructions, a fusion instruction with reduced reliability is generated. According to the adjusted weights, the recognition results of multiple input methods are fused to generate the final interaction instruction.

[0054] As an optional solution for evaluating modal instructions based on contextual information, environmental information is first received, including background noise, light, and facial conditions, and the background in the environmental information is preliminarily identified. Then, based on the command input by the user and the environmental information, the reliability is evaluated, and the weights corresponding to each modal instruction are adjusted. After the instruction result is determined, the adjusted weight is determined, and the adjusted weight is multiplied by the modal recognition coefficient to calculate the weighted score corresponding to each input method, thereby determining the fusion instruction.

[0055] Exemplarily, the recognition result of each input is used as a "vote", and a weighted vote is performed according to the weight, and the instruction with the most votes is selected as the final instruction; or, the recognition processing of each input mode is converted into a numerical value (for example, 1 means on, 0 means off), and then the weighted average is calculated according to the weight, and the closest instruction is selected. In addition to obtaining a single modal instruction, when the fusion instruction is finally output, a weighted score threshold is set, and all weighted scores higher than the preset threshold are regarded as fusion instruction outputs; or all instructions with weighted scores are sorted, and the first N instructions are selected as fusion instruction outputs.

[0056] Exemplarily, based on the reliability, user preference and default settings during modal recognition, an initial weight of 0.4 is assigned to voice recognition, an initial weight of 0.3 is assigned to gesture recognition, and an initial weight of 0.3 is assigned to face recognition. Since the user's action is in a noisy background and the reliability is low, R1 is set to 0.6; the light is moderate in gesture recognition and the reliability is high, R2 is set to 0.9; the user's face is not blocked and the reliability is strong, R3 is set to 0.9.

[0057] The specific adjustment method can be adopted by the following formula;

[0058] in, is the adjusted weight, is the initial weight, No. The reliability score of the input method (between 0 and 1).

[0059] Furthermore, after calculating the weighted scores, the threshold method is used, the threshold is set to 0.3, and the modal instructions corresponding to the weighted scores exceeding 0.3 are output. Specifically, the voice recognition score is 0.3077, the gesture recognition score is 0.3462, and the facial recognition score is 0.3462. Therefore, "reduce the brightness of the TV screen, reduce the volume, and recommend more comfortable programs" are output as fusion instructions.

[0060] This embodiment proposes a television control method based on multimodal instructions, which realizes comprehensive understanding and response to the user's multimodal input by identifying modal data and determining corresponding modal instructions through a modal recognition engine, and then processes these instructions based on the context information in time sequence, providing a more natural, intuitive and efficient interaction method, so that the system can intelligently parse and execute the user's intention according to various input methods such as user's voice, gestures, facial expressions, etc., and fuse the results of various modal instructions to determine the final fused instruction, thereby providing a richer and more personalized user experience.

[0061] Based on any of the above embodiments, the third embodiment of the present application provides a television control method based on multimodal instructions, referring to Figure 3 , Figure 3 This is a flow chart of the third embodiment of the TV control method based on multi-modal instructions of the present application, step S210 includes steps S211 to S213: Step S211, receiving a modal signal transmitted by a sensor, preprocessing the modal signal, and determining the modal data.

[0062] As an optional implementation, the system receives user input signals through various sensors (such as microphones, cameras, tactile sensors, etc.), which may include voice, images, tactile feedback, etc., and performs noise reduction, enhancement, normalization and other processing on these signals to improve the quality of the signals and make them more suitable for subsequent feature extraction and recognition. Then, useful information is extracted from the preprocessed signals and converted into a data format that can be processed by machine learning algorithms to obtain modal data.

[0063] For example, when a user uses a smart TV, he says "volume up" by voice and makes a gesture to turn up the volume towards the camera. The system receives the voice signal through the microphone and captures the user's gesture image and facial image through the camera. Then, the system applies a noise reduction algorithm to these voice signals to reduce environmental background noise, such as the volume of the TV itself or other people's conversations. For gesture images, the system enhances the image contrast to more clearly identify the user's gestures. After preprocessing, the system converts the extracted signals into a data format suitable for processing by machine learning algorithms. For voice signals, they can be converted into feature vectors such as Mel Frequency Cepstral Coefficients (MFCC); for gesture images and facial images, edges, corners, or deep features extracted using deep learning may be extracted; these converted data (modal data) will be used for subsequent modal feature extraction to provide accurate input for recognizing user instructions.

[0064] Step S212: extracting modal features of the modal data.

[0065] Extract key features from modal data, which can represent important information of the original signal, such as the spectral characteristics of speech, the edge features of images, etc.

[0066] For example, the speech signal is converted into an MFCC feature vector, and the edge and corner features of the gesture image and the facial image are extracted at the same time. Then, key features are further extracted from these data. For speech, it is possible to focus on specific spectral changes in MFCC and use this change as a speech feature; for gesture images, focus on specific gesture shapes and motion trajectories, which correspond to gestures for adjusting the volume; for facial images, focus on changes in facial expressions, and use micro-expressions or micro-expression changes as facial features, so that this feature represents the information in the original signal that is most relevant to the user's intention.

[0067] Step S213, identifying the modal features based on the machine learning algorithm in the modal recognition engine to obtain the corresponding modal instructions.

[0068] After extracting the corresponding modal features, the specific information of the user input is obtained by identifying these features.

[0069] As an optional solution for identifying modal features, a well-trained machine learning model, such as a neural network, support vector machine, etc., is used to analyze and classify the extracted features, identify the user's intention, and then, based on the recognition results of the machine learning algorithm, map the identified modal features to a predefined instruction set to generate corresponding control instructions.

[0070] Exemplarily, these features are processed by a pre-trained neural network model. For voice features, the model is based on specific spectral patterns associated with the "volume up" command; for gesture features, the model identifies specific gesture trajectories or shapes that match the action of adjusting the volume; for facial features, the model identifies micro-expressions or micro-expression changes in facial expressions that match specific meanings. The neural network infers through these features and recognizes that the user's intention is to adjust the volume. Once the machine learning model recognizes the user's intention, these recognition results are mapped to a predefined instruction set. For example, if both the voice and gesture recognition results point to "volume up", the system will generate a fusion instruction. The fusion instruction execution action remains unchanged, but the reliability of the corresponding command is improved. Finally, key information is extracted from the user's multimodal input, the user's intention is recognized, and the corresponding control action is executed, thereby achieving an efficient and intuitive interactive experience.

[0071] This embodiment proposes a television control method based on multimodal instructions, which converts the original modal signals transmitted by the user through the sensor into executable control instructions, first receives and preprocesses these signals, then extracts key features that can represent important information of the signals, and finally uses a machine learning algorithm to analyze and identify these features, thereby determining the user's intention and generating corresponding control instructions, so that the television or associated devices can intelligently respond to the user's multimodal interaction.

[0072] Based on any of the above embodiments, the fourth embodiment of the present application provides a television control method based on multimodal instructions, referring to Figure 4 , Figure 4 This is a flowchart of Embodiment 4 of the TV control method based on multi-modal instructions of the present application. Before step S220, steps S218 to S219 are included: Step S218, determining user preferences and historical operation instructions based on the historical operation data.

[0073] It should be noted that before processing the modal data, historical operations are recorded and user preferences are identified, so that channel recommendations can be provided after facial recognition.

[0074] As an optional solution to determine user preferences based on historical operation data, statistical analysis is used to collect user operation data over a certain period of time and analyze user behavior patterns using statistical methods to determine user preferences and historical operation instructions.

[0075] For example, user A often uses a smart TV to watch news channels at 8 p.m. The statistical analysis method will identify this pattern and determine that the user prefers to watch the news at 8 p.m. Therefore, when the user's facial features are identified, the instruction to recommend news channels to user A at 8 p.m. is given higher reliability.

[0076] As another alternative to determine user preferences based on historical operation data, sequential pattern mining methods focus on analyzing the order and frequency of user operations to discover sequential patterns of user behaviors, thereby determining user preferences and historical operation instructions.

[0077] For example, after watching a TV series, user B usually uses gestures to view the actor information and related comments of the show. The sequential pattern mining method will identify this behavior sequence and give higher reliability to the gestures that provide actor information and comment links after user B finishes watching, so as to meet the user's potential needs.

[0078] Step S219, extracting semantic features of the modal instructions at the same time sequence, and determining the modal instructions of the same type.

[0079] When a modal instruction is identified, the semantic features of the modal instruction may be determined, thereby determining whether the modal instructions are of the same type.

[0080] From the various modal features collected from the modal data, it can be seen that modal instructions with the same input method are classified into the same type.

[0081] This embodiment proposes a TV control method based on multi-modal commands. By analyzing the user's historical operations, the system can learn and predict the user's preferences, thereby providing personalized services and recommendations. At the same time, by extracting and comparing the semantic features of modal commands, the system can identify commands of the same type, achieve more accurate command execution and interactive experience, and make the communication between the user and the device more natural and efficient.

[0082] Based on any of the above embodiments, the fifth embodiment of the present application provides a television control method based on multimodal instructions, referring to Figure 5 , Figure 5 This is a flowchart of Embodiment 5 of the TV control method based on multi-modal instructions of the present application, step S220 includes steps S221 to S223: Step S221, merging the modal instructions of the same type to determine the total number of target modal instructions.

[0083] The determined multiple modal instructions are used as an output set, thereby determining the total number of target modal instructions in the output set.

[0084] It should be noted that modal instructions of the same type refer to modal instructions with the same semantics, including modal instructions A and modal instructions B having the same semantics, for example, voice modal instructions and gesture recognition instructions both collect the command of "lowering the volume"; at the same time, for modal data of different timelines of the same modality, historical operation instructions are associated with modal data C, for example, for default setting parameters, historical modal data identifies modal data C. For example, if multiple modal instructions to lower the volume are received within a period of time, it is determined whether to execute the second modal instruction based on the user's preferences and system settings. After merging, the modal instruction to be finally executed is determined, which may be to lower the volume multiple times, or to determine whether to execute the second time based on the user's historical volume setting range after lowering the volume once. For example, after the volume is obviously raised to the normal level, it is still rising. At this time, it may be that the user fell asleep and accidentally touched the remote control, causing the volume to continue to rise. At this time, the system will not immediately raise the volume, and a prompt will be given to the user interface, and the volume will be raised after the confirmation message.

[0085] Step S222: if the total number of target modal instructions includes at least one of the modal instructions, receive environmental information transmitted by an environmental sensor, and determine the reliability of the modal instruction based on the environmental information.

[0086] Collect current environmental information through environmental sensors, such as ambient light sensors, to determine the impact of light conditions on the currently collected data, including the microphone array collecting the current background noise and the ambient light sensor collecting the light intensity. In some adverse environmental conditions, such as strong light or high noise environments, reduce the weight of modal instructions or require more confirmation to reduce misoperation.

[0087] For example, when watching TV, users are accustomed to turning off the surrounding lights. Under this condition, the customer gives commands to the TV through voice, and the TV receives modal data through the microphone array and sensors for analysis. The environmental information received by the environmental sensor confirms that the current environment is dark. Based on this information, the system will improve the reliability of voice commands, which is more inclined to the confirmation of voice commands and reduce the error caused by other modes.

[0088] Step S223, adjusting the initial weight of each of the modal instructions based on the reliability and historical operation instructions, and determining the instruction result.

[0089] According to the reliability assessment results, the weight of each input method is dynamically adjusted to generate the final instruction. Then, based on the adjusted weight, the recognition results of multiple input methods are integrated to generate the final interaction instruction.

[0090] This embodiment proposes a visual control method based on multi-modal instructions, which dynamically adjusts the initial weights of various modal instructions by comprehensively considering the actual environmental information provided by the environmental sensor and the user's historical operation instructions to improve the accuracy of instruction recognition and the responsiveness of the system. According to these environmental factors and the user's previous behavior patterns, the system can adjust the weights of different modal instructions so that the more reliable modal instructions in the current environment have a higher influence, thereby determining the final instruction result.

[0091] Based on any of the above embodiments, the sixth embodiment of the present application provides a television control method based on multimodal instructions, referring to Figure 6 , Figure 6 This is a flowchart of Embodiment 6 of the television control method based on multi-modal instructions of the present application, step S230 includes steps S231 to S233: Step S231, determining an initial weighted score based on the correspondence between the instruction result and the modal identification coefficient.

[0092] It should be noted that the modal recognition coefficient is the determination of the recognition result. Each modal command obtains an initial weighted score according to its recognition coefficient, where the modal recognition coefficient ranges from 0 to 1, where 1 indicates completely reliable recognition and "0" indicates completely unreliable recognition. When only the facial command recognition is unsuccessful, the modal recognition coefficient is assigned a ranking value of "110", thereby determining that the voice and gesture recognition are successful, while the facial recognition command is unsuccessful, and it can be determined that the user's facial features are difficult to recognize.

[0093] Step S232: adjusting the initial weighted score based on the historical operation instructions to determine the weighted score.

[0094] After determining the weighted score of the operation instruction, the result of the corresponding modal instruction calculation is determined by identifying the successful or unsuccessful modal recognition coefficient. For example, when the user wears a mask, the user's facial features cannot be recognized. At this time, the TV can only be controlled by voice commands and gesture commands, and the collected facial data are all error data. Therefore, after calculating the weighted score, the error of the facial data is removed by the modal recognition coefficient, so the fusion instruction at this time is confirmed by the voice command and gesture command.

[0095] Step S233, outputting the fusion instruction whose weighted score is higher than a preset threshold.

[0096] After the initial weighted score is processed with the modal identification coefficient, the remaining fusion instructions can be used as output instructions. A preset threshold is set to determine the requirements for the output of the fusion instructions. For example, the weighted score must be higher than the preset threshold to filter out the optimization results. According to the preset threshold, instructions with a final weighted score higher than this threshold will be output and executed.

[0097] Each modal command is combined with its recognition accuracy (modal recognition coefficient) to calculate an initial weighted score, and then these scores are adjusted according to the user's historical operation commands to determine the final weighted score. Finally, the system outputs those fusion commands whose weighted scores are higher than the preset threshold, ensuring that the most reliable and user-friendly commands are executed.

[0098] Exemplarily, based on the "play music" command issued by the user through voice and the initial weighted score of gesture recognition, the modal recognition coefficient is set to "110" for the voice command, and the modal recognition coefficient is set to 0 for the facial features that are not recognized. After multiple modal commands are recognized, the final weighted score of the modal data is calculated.

[0099] This embodiment proposes a television control method based on multimodal instructions, which determines an initial weighted score by combining each modal instruction with its corresponding modal recognition coefficient, and this coefficient reflects the recognition confidence of the instruction; then, the system adjusts these initial scores according to the user's historical operation instructions to more accurately reflect the user's preferences and behavior patterns; finally, when those fusion instructions with weighted scores higher than a preset threshold are output and executed, such a design can ensure that the most reliable and user-friendly instructions are adopted, thereby improving the system's response quality and user satisfaction.

[0100] Based on any of the above embodiments, the seventh embodiment of the present application provides a television control method based on multimodal instructions, referring to Figure 7 , Figure 7 This is a flowchart of Embodiment 7 of the TV control method based on multi-modal instructions of the present application. After step S300, steps S400 to S500 are also included: Step S400: adjusting the screen display parameters based on the environmental state detected by the environmental sensor.

[0101] The environmental sensor detects the surrounding environmental conditions and automatically adjusts the display parameters of the TV screen to adapt to different viewing environments and reduce user visual fatigue.

[0102] For example, when a user watches TV at sunset, the ambient light gradually dims. The ambient light sensor detects the drop in light intensity, and the system automatically reduces the brightness of the TV screen and adjusts the contrast to adapt to the darker environment; at the same time, the system may also adjust the color temperature of the screen according to the color temperature changes of indoor and outdoor light to maintain the natural color of the picture; users can enjoy the best visual experience at different times without manually adjusting the settings.

[0103] Step S500, optimizing the display image based on the adjusted image display parameters.

[0104] After the screen display parameters are modified and adjusted, the modified display parameters are applied to the screen display to optimize the display screen output.

[0105] Optionally, when optimizing the display screen, the environmental sensor receives changes in environmental information and changes the initial weights of modal data processing, thereby increasing the weight of voice recognition and decreasing the weights of gesture recognition and facial recognition.

[0106] This embodiment proposes a television control method based on multimodal instructions, which detects the surrounding environment status through environmental sensors, adjusts the picture display parameters, and then optimizes the display picture through these adjusted parameters, so that the picture of the television or other display device automatically adapts to different ambient light conditions, providing a more comfortable and natural visual experience.

[0107] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the TV control method based on multimodal instructions of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0108] The present application provides a television control device based on multimodal instructions, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the television control method based on multimodal instructions in the above-mentioned embodiment 1.

[0109] Reference below Fig. 9, which shows a schematic diagram of the structure of a TV control device based on multimodal instructions suitable for implementing the embodiment of the present application. The TV control device based on multimodal instructions in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The illustrated television control device based on multimodal instructions is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0110] like Fig. 9 As shown, the TV control device based on multimodal instructions may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the TV control device based on multimodal instructions are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the television control device based on multimodal instructions to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a television control device based on multimodal instructions with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0111] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0112] The TV control device based on multimodal instructions provided by the present application adopts the TV control method based on multimodal instructions in the above-mentioned embodiment, which can solve the technical problem in the related art that, by integrating a microphone array and a camera, basic voice and gesture recognition functions can be realized, but multiple types of instructions transmitted simultaneously cannot be responded to in parallel, resulting in the user needing to frequently switch input methods. Compared with the prior art, the beneficial effects of the TV control device based on multimodal instructions provided by the present application are the same as the beneficial effects of the TV control method based on multimodal instructions provided by the above-mentioned embodiment, and the other technical features of the TV control device based on multimodal instructions are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0113] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0114] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0115] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the television control method based on multimodal instructions in the above-mentioned embodiment.

[0116] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0117] The computer-readable storage medium may be included in the television control device based on multimodal instructions; or may exist independently without being assembled into the television control device based on multimodal instructions.

[0118] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the TV control device based on multimodal instructions, the TV control device based on multimodal instructions can control the TV based on multimodal instructions.

[0119] Computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0122] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned TV control method based on multimodal instructions, and can solve the technical problem in the related art that, by integrating a microphone array and a camera, basic voice and gesture recognition functions can be realized, but multiple types of instructions transmitted simultaneously cannot be responded to in parallel, resulting in the user needing to frequently switch input methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the TV control method based on multimodal instructions provided by the above-mentioned embodiment, and will not be repeated here.

[0123] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A television control method based on multimodal instructions, characterized in that: Applied to a television, the television control method based on multimodal instructions comprises the following steps: Acquire at least one modality data collected by the television and / or associated equipment; Determining a fusion instruction based on context information of the modality data in time sequence; Control the television to execute a control action corresponding to the fusion instruction.

2. The TV control method based on multimodal instructions according to claim 1, characterized in that: The step of determining the fusion instruction based on the context information of the modal data in time sequence includes: Identify the modal data by a modal recognition engine, and determine the modal instructions corresponding to each modal data; Based on the context information of the modal instruction in time sequence, the modal instruction is processed to obtain an instruction result; The instruction results corresponding to each of the modal instructions are fused to determine the fused instruction.

3. The TV control method based on multimodal instructions as claimed in claim 2, characterized in that: The step of identifying the modal data by a modal recognition engine and determining a modal instruction corresponding to each modal data comprises: Receiving a modal signal transmitted by a sensor, preprocessing the modal signal, and determining the modal data; extracting modal features of the modal data; The modal features are identified based on the machine learning algorithm in the modal recognition engine to obtain the corresponding modal instructions.

4. The TV control method based on multimodal instructions according to claim 2, characterized in that: Before the step of processing the modal instruction based on the context information of the modal instruction in time sequence to obtain the instruction result, the method further comprises: Determine user preferences and historical operation instructions based on historical operation data; The semantic features of the modal instructions at the same time sequence are extracted to determine the modal instructions of the same type.

5. The TV control method based on multimodal instructions according to claim 2, characterized in that: The step of processing the modal instruction based on the context information of the modal instruction in time sequence to obtain the instruction result includes: Merging the modal instructions of the same type to determine the total number of target modal instructions; If the total number of target modal instructions includes at least one of the modal instructions, receiving environmental information transmitted by an environmental sensor, and determining the reliability of the modal instruction based on the environmental information; The initial weight of each of the modal instructions is adjusted based on the reliability and historical operation instructions to determine the instruction result.

6. The TV control method based on multimodal instructions as claimed in claim 2, characterized in that: The step of fusing the instruction results corresponding to each of the modal instructions to determine the fused instruction includes: Determining an initial weighted score based on the command result corresponding to the modal identification coefficient; Adjusting the initial weighted score based on historical operation instructions to determine the weighted score; Output the fusion instruction whose weighted score is higher than a preset threshold.

7. The TV control method based on multimodal instructions according to claim 1, characterized in that: The step of acquiring at least one modality data collected by the television and / or associated equipment comprises: At least one of the modal data is acquired based on preset sensors in the television and / or associated devices, wherein the preset sensors include a microphone array, a camera, and an environmental sensor.

8. The TV control method based on multimodal instructions according to claim 1, characterized in that: The television control method based on multimodal instructions further includes: Based on the environmental sensor detecting the environmental status, adjust the screen display parameters; Based on the adjusted picture display parameters, the display picture is optimized.

9. A control device for multi-modal instruction interaction, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the television control method based on multimodal instructions as claimed in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the television control method based on multimodal instructions as described in any one of claims 1 to 8 are implemented.

Images