Distributed infrared limb movement recognition system and method

Abstract

The invention discloses a distributed infrared limb movement recognition system and method. The system is applied to the smart television and comprises a signal acquisition module and a signal control module, wherein the signal acquisition module comprises a sensor receiving array and an infrared emission matrix, the sensor receiving array comprises a main sensor and auxiliary sensors, the main sensor is arranged at the top of the smart television, the auxiliary sensors are arranged on a first side frame and a second side frame of the smart television, and the first side frame and the second side frame are oppositely arranged; the infrared emission matrix comprises a plurality of infrared emission units which are arranged on a backlight light bar of the smart television. According to the embodiment of the invention, the sensor receiving array formed by the triangular layout of the main sensor and the auxiliary sensor is matched with the plurality of infrared emission units arranged on the backlight light bar, so that the effective detection range of the limb movement is expanded, the interaction requirement of a user is met, and the problem of narrow detection range is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent television interaction, and in particular to a distributed infrared body motion recognition system and method. BACKGROUND

[0002] The current mainstream intelligent television interaction mode mainly relies on traditional remote control or touch interaction and other physical contact methods, visual recognition methods and millimeter wave radar methods, which have defects such as insufficient interaction flexibility, poor environmental adaptability, low motion recognition accuracy, and high cost. With the development of intelligent television interaction technology, the infrared recognition method is also used in the intelligent television interaction mode. However, the traditional infrared recognition method mainly uses a single-point infrared sensor to judge simple motions through infrared signal reflection, which has problems such as narrow detection range and cannot meet complex interaction requirements. SUMMARY

[0003] Embodiments of the present application provide a distributed infrared body motion recognition system and method to at least partially solve the above technical problems.

[0004] In one aspect, the present application provides a distributed infrared body motion recognition system applied to an intelligent television, the system comprising: a signal acquisition module for acquiring a body motion signal of a user in a preset detection area; a signal control module for controlling the intelligent television according to the body motion signal; wherein the signal acquisition module comprises a sensor receiving array and an infrared emission matrix; the sensor receiving array comprises a main sensor and an auxiliary sensor, the main sensor is arranged on the top of the intelligent television, and the auxiliary sensor is arranged on a first side frame and a second side frame of the intelligent television, the first side frame and the second side frame are arranged oppositely; the infrared emission matrix comprises a plurality of infrared emission units, and the infrared emission units are arranged on a backlight bar of the intelligent television.

[0005] In some embodiments, the arrangement position of the infrared emission unit on the backlight bar is determined based on the screen size of the intelligent television and the distribution density of backlight beads on the backlight bar.

[0006] In some embodiments, the infrared emission matrix emits infrared signals through an active pulse modulation method, and the sensor receiving array receives the body motion signal through a phase-sensitive detection technology.

[0007] In some embodiments, the preset detection area is a fan-shaped area; the orthographic projection area of the preset detection area on a first plane is greater than the orthographic projection area of the screen of the intelligent television on the first plane; wherein the first plane is parallel to the plane on which the screen is located.

[0008] In some embodiments, the signal control module comprises: a signal preprocessing unit configured to preprocess the limb movement signal, the preprocessing comprising: noise reduction processing, signal enhancement, and normalization processing; a feature extraction unit configured to extract multi-modal features based on the limb movement signal, and fuse the multi-modal features into a feature vector sequence; a movement recognition unit configured to recognize the input feature vector sequence based on a deep learning model to determine the limb movement of the user; an instruction generation unit configured to generate a control instruction based on the recognized limb movement; and a control execution unit configured to control the smart television based on the control instruction.

[0009] In some embodiments, the multi-modal features comprise: spatio-temporal features, amplitude features, frequency features, and phase features; and the feature extraction unit is further configured to extract the spatio-temporal features reflecting the movement trajectory of the limb movement, extract the amplitude features reflecting the movement strength of the limb movement, extract the frequency features reflecting the movement frequency of the limb movement, and extract the phase features reflecting the movement direction of the limb movement.

[0010] In some embodiments, the deep learning model comprises: a light-weight CNN-LSTM hybrid model; and the light-weight CNN-LSTM hybrid model is subjected to model pruning and INT8 quantization processing.

[0011] In some embodiments, the movement recognition unit is further configured to run the deep learning model on a neural network processing unit of the smart television.

[0012] A distributed infrared limb movement recognition method applied to a smart television, the method comprising: a signal acquisition module acquiring a limb movement signal of a user in a preset detection area; and a signal control module controlling the smart television based on the limb movement signal; wherein the signal acquisition module comprises a sensor receiving array and an infrared emission matrix; the sensor receiving array comprises a main sensor and an auxiliary sensor, the main sensor is arranged on the top of the smart television, and the auxiliary sensor is arranged on a first side bezel and a second side bezel of the smart television, the first side bezel and the second side bezel being oppositely arranged; and the infrared emission matrix comprises a plurality of infrared emission units, the infrared emission units being arranged on a backlight bar of the smart television.

[0013] In another aspect, the embodiments of the present application further provide an electronic device comprising a memory and a processor, the memory storing a computer program or instructions, and the computer program or instructions being executed by the processor to cause the processor to perform the steps of any one of the distributed infrared limb movement recognition methods provided by the embodiments of the present application.

[0014] In another aspect, the embodiment of the present application further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implement the steps of any of the distributed infrared limb action recognition methods provided by the embodiment of the present application.

[0015] In another aspect, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program or instructions, comprising a computer program or instructions, which, when executed by a processor, implement the steps of any of the distributed infrared limb action recognition methods provided by the embodiment of the present application.

[0016] The present application discloses a distributed infrared limb action recognition system and method. The system is applied to a smart television and comprises a signal acquisition module and a signal control module. The signal acquisition module comprises a sensor receiving array and an infrared emission matrix. The sensor receiving array comprises a main sensor and an auxiliary sensor. The main sensor is arranged on the top of the smart television, and the auxiliary sensor is arranged on the first side frame and the second side frame of the smart television. The first side frame and the second side frame are arranged oppositely. The infrared emission matrix comprises a plurality of infrared emission units arranged on the backlight strip of the smart television. The sensor receiving array formed by the triangular layout of the main sensor and the auxiliary sensor cooperates with the plurality of infrared emission units arranged on the backlight strip to expand the effective detection range of the limb action, meet the user interaction demand, and solve the problem of narrow detection range. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The figure is an architecture schematic diagram of the distributed infrared limb action recognition system provided in the embodiment of the present application. Figure 2 The figure is a sensor layout schematic diagram of the signal acquisition module provided in the embodiment of the present application. Figure 3 The figure is a top view of the field of view range of the signal acquisition module provided in the embodiment of the present application. Figure 4 The figure is a structure schematic diagram of the signal control module provided in the embodiment of the present application. Figure 5 The figure is a flow schematic diagram of the distributed infrared limb action recognition method provided in the embodiment of the present application. Figure 6A flowchart of another distributed infrared limb action recognition method provided in the embodiments of the present application; Figure 7 A structural diagram of an electronic device provided in the embodiments of the present application.

[0019] Labeling of the drawings: 100, a distributed infrared limb action recognition system; 110, a signal acquisition module; 120, a signal control module; 111, a sensor receiving array; 1111, a main sensor; 1112, an auxiliary sensor; 112, an infrared emission matrix; 1121, an infrared emission unit; 121, a signal preprocessing unit; 122, a feature extraction unit; 123, an action recognition unit; 124, an instruction generation unit; 125, a control execution unit. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0021] In the following description, specific embodiments of the present application will be described with reference to steps and symbols executed by one or more computers, unless otherwise specified. Therefore, these steps and operations will be mentioned several times by computers, and the computer execution referred to herein includes the operation of a computer processing unit represented by an electronic signal in a structured form. This operation transforms the data or maintains it at a location in the memory system of the computer, which can reconfigure or otherwise change the operation of the computer in a manner known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above description, which does not represent a limitation, and those skilled in the art will understand that the various steps and operations described below can also be implemented in hardware.

[0022] The term "module" or "unit" used herein can be regarded as a software object executed on the operating system. Different components, modules, engines and services described herein can be regarded as implementation objects on the operating system. The apparatus and method described herein are preferably implemented in software, and of course can also be implemented in hardware, both of which are within the protection scope of the present application.

[0023] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" (or comprise), "comprises" (or comprises) and "comprising" (or comprises) when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof. It is further understood that when an element is said to be "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. Furthermore, "connected" or "coupled" as used herein can include wirelessly connected or wirelessly coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0024] In the prior art, the intelligent television interaction mode mainly exists in the following several modes: The traditional physical interaction mode mainly using infrared remote controller and Bluetooth remote controller needs the user to physically contact the device, and has problems of easy loss, operation depending on key memory, etc. The representative product using this interaction mode is most of the middle and low-end televisions, which are equipped with a traditional remote controller and only support basic channel switching, volume adjustment, etc.

[0025] In the mode of using voice and touch interaction, voice interaction relies on a microphone array to receive instructions, which limits its use in noisy environments or quiet scenes; while touch interaction mode requires the user to contact the screen at close range, which is not suitable for long-distance viewing scenes, such as Huawei MateTV Lingxi floating touch interaction, which realizes millimeter-level trajectory capture through 328 built-in finger detection sensors.

[0026] The visual recognition mode captures gesture images through an RGB camera and relies on image algorithm to recognize actions, which has the advantage of rich recognition dimension, but its accuracy decreases in low light environment, and there is a risk of user privacy leakage.

[0027] The millimeter wave radar mode uses millimeter wave to detect body movements, has strong anti-environmental light interference ability, but the cost is high, and the recognition accuracy of subtle movements is insufficient.

[0028] The traditional infrared recognition mode uses a single-point infrared sensor, which can judge simple actions through infrared signal reflection, although it has the advantages of low cost and good privacy, but it has problems of narrow detection range, high response delay, and easy to be disturbed by environmental light, etc., and this mode can only realize basic control and cannot meet the demand of complex interaction.

[0029] In summary, existing smart TV interaction methods have many shortcomings, failing to achieve complex TV control functions and failing to meet users' needs for natural interaction over long distances and in multiple scenarios. Therefore, this application provides a distributed infrared body movement recognition system 100. Figure 1 This is a schematic diagram of the architecture of a distributed infrared limb movement recognition system provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 The distributed infrared limb movement recognition system 100 may include a signal acquisition module 110 and a signal control module 120.

[0030] The signal acquisition module 110 is used to acquire limb movement signals of the user within a preset detection area. The signal acquisition module 110 includes a sensor receiving array 111 and an infrared emission matrix 112. The sensor receiving array 111 includes a main sensor 1111 and an auxiliary sensor 1112. The main sensor 1111 is disposed on the top of the smart TV, and the auxiliary sensor 1112 is disposed on the first side bezel and the second side bezel of the smart TV, which are arranged opposite to each other. The infrared emission matrix 112 includes multiple infrared emission units 1121, which are disposed on the backlight strip of the smart TV.

[0031] The signal control module 120 is used to control the smart TV based on body movement signals.

[0032] Figure 2 This is a schematic diagram of the sensor layout of a signal acquisition module provided in an embodiment of this application. Please refer to [link / reference]. Figure 2The sensor receiving array 111 adopts a triangular layout of a top main sensor 1111 and two side auxiliary sensors 1112, breaks through the narrow field of view angle of a single point sensor, can capture the operation of a user on the side of the screen, and realizes wide-angle coverage of the user's body action interaction. Specifically, the main sensor 1111 installed on the top of the smart television is responsible for core observation. For example, the main sensor 1111 can be arranged at the central position on the top of the smart television, or can be arranged at other positions on the top. The number of the main sensor 1111 can be one or multiple, which is not limited in the application. The horizontal coverage width of the main sensor 1111 is equal to the width of the television screen, which is responsible for collecting the body action signal of the core area in front of the smart television, and provides the most direct signal with the highest signal-to-noise ratio in front of the smart television, which is the main source of information for judging the details of the body action. The auxiliary sensors 1112 installed on the first side frame and the second side frame are responsible for expanding the field of view and spatial alignment, which can collect the body action signal in the oblique direction of the television, and observe the same action from the oblique angle, and provide key parallax information for the system. For example, the first side frame and the second side frame of the smart television correspond to the left and right side frames respectively. The auxiliary sensors 1112 can be symmetrically arranged on the left and right side frames of the television screen, or multiple auxiliary sensors 1112 can be arranged on the left and right side frames respectively. The number of auxiliary sensors 1112 arranged on the two side frames can be the same or different, which is not limited in the application and can be set according to actual use requirements.

[0033] Figure 3 A top view of the field of view range of the signal acquisition module provided in the embodiment of the application. The triangular layout of the sensor receiving array 111 provided in the embodiment of the application breaks through the narrow field of view angle limitation of the single point sensor, realizes wide-angle coverage, and for example, the distributed infrared body action recognition system 100 provided in the embodiment of the application can realize a field of view angle range of about 150°, which greatly improves the user experience.

[0034] Continuing to refer to Figure 2 The signal acquisition module 110 provided in the embodiment of the application further includes an infrared emission matrix 112 integrated with the backlight strip. Multiple infrared emission units 1121 in the infrared emission matrix 112 are arranged at specific positions of the backlight strip. For example, the multiple infrared emission units 1121 can be arranged near each lamp bead on the backlight strip.

[0035] In some embodiments, the setting position of the infrared emission unit 1121 on the backlight bar is determined based on the screen size of the smart TV and the distribution density of the backlight beads on the backlight bar. The position of the infrared emission unit 1121 is calculated in relation to the screen size and the density of the beads, and by optimizing the number and spacing of the infrared emission units 1121, a point-plane layout is formed with the sensor receiving array 111, which expands the horizontal detection angle, enhances the capture ability of the user's body movement in the front area of the TV, and ensures that the sensor can receive a sufficient intensity of the reflected signal at any position within the preset detection area, thereby avoiding the uneven lighting problem caused by the traditional middle single-point emission scheme, that is, the central area is too bright and the two sides are too dark. Illustratively, the preset detection range provided by the embodiments of the present application can cover about 150° horizontal angle and about 2-4 meters effective distance, and the system can capture the user's body movement within the preset detection range. At the same time, since the infrared emission unit 1121 is integrated on the backlight bar, the additional structural parts and assembly costs are saved.

[0036] The signal control module 120 serves as a bridge between the signal acquisition module 110 and the interaction between the smart TV, and is used to control the smart TV according to the body movement signal collected by the signal acquisition module 110. The distributed infrared body movement recognition system 100 provided by the embodiments of the present application can detect the user's body movement within the preset detection range, and implement corresponding control on the smart TV based on the user's body movement.

[0037] The distributed infrared body movement recognition system 100 provided by the embodiments of the present application expands the effective detection range of the body movement through the sensor receiving array 111 formed by the triangular layout of the main sensor 1111 and the auxiliary sensor 1112, cooperates with the plurality of infrared emission units 1121 arranged on the backlight bar, meets the user's interaction demand, and solves the problem of narrow detection range.

[0038] In some embodiments, the infrared emission matrix 112 emits infrared signals through an active pulse modulation method, and the sensor receiving array 111 receives the body movement signal through a phase-sensitive detection technology.

[0039] The infrared emission matrix 112 adopts a matrix infrared emission tube, and in the signal emission stage, the infrared emission matrix 112 synchronously emits infrared pulse signals of a specific wavelength (for example, 840 nm or 940 nm) under the unified timing of the control unit, wherein the pulse frequency can be set to other special values (for example, 100 Hz) different from the common 50 Hz or 60 Hz power frequency interference in the environment light, thereby avoiding the stroboscopic interference of the environment light source. At the same time, the pulse duty cycle can be dynamically configured according to the requirements of the system on the detection distance, accuracy and overall power consumption, so as to balance the performance and energy efficiency.

[0040] In the signal propagation and reflection stage, the infrared pulse signal propagates in space and is reflected when encountering the user's limbs. The reflected signal can be captured by the sensor units in the sensor receiving array 111.

[0041] In the signal receiving and processing stage, since the sensor units in the sensor receiving array 111 internally integrate a phase-sensitive detection circuit, which takes the frequency and phase of the transmitted pulse as a reference, only the transmitted signal that is completely synchronized and matched with the frequency and phase of the reference signal is amplified and passed, while non-coherent infrared noise such as ambient light is suppressed, so that the effective limb movement signal can be extracted.

[0042] In the signal conversion and transmission stage, the sensor converts the captured optical signal through phase-sensitive detection into a high-precision digital electrical signal. Exemplarily, the conversion precision is 16 bits. Then, the digital electrical signal is transmitted in real time to the signal processing module through a high-speed serial bus. To ensure that fast limb movements can be captured, the sampling frequency of the system can be set to 60 fps to ensure the integrity and continuity of the limb movement timing information.

[0043] The distributed infrared limb movement recognition system 100 provided by the embodiments of the present application improves the anti-environmental light interference capability of the system through the cooperative work of pulse modulation and phase-sensitive detection, effectively filters environmental light interference, ensures that the system can work stably in different home environments such as strong light and weak light, greatly improves the robustness of the system in complex environmental light, and solves the problem that the traditional infrared method is easily affected by light.

[0044] In some embodiments, the preset detection area is a fan-shaped area; a normal projection area of the preset detection area on a first plane is greater than a normal projection area of a screen of the smart television on the first plane; and the first plane is parallel to a plane on which the screen is located.

[0045] Continuing to refer to Figure 3 As shown, the preset detection area corresponds to the field of view range of the signal acquisition module 110, and the screen of the smart television can be a flat screen or a curved screen, which is not limited by the present application. When the screen is a flat screen, the first plane is parallel to the plane on which the screen is located; when the screen is a curved screen, the first plane can be defined as parallel to a tangent plane of the curved screen at a center point or parallel to a reference plane containing the outer frame of the screen. The preset detection area is a fan-shaped area, and exemplarily, the fan-shaped area has a horizontal viewing angle of about 150° and an effective distance of about 2-4 meters. The normal projection area of the preset detection area on the first plane is greater than the normal projection area of the screen of the smart television on the first plane, and completely covers and exceeds the physical boundary of the screen itself.

[0046] The distributed infrared limb motion recognition system 100 provided by the embodiments of the present application can collect limb motions of a user in a large range in front of a screen and on both sides of the screen. For example, even if the user is not directly facing the center of the screen but is located in a preset detection area on the side of the screen, the limb motions made by the user can be effectively captured and recognized by the system, breaking the spatial limitation that the traditional interaction mode must be directly facing the screen, realizing uniform coverage of the preset detection area without dead angles, and significantly improving the interaction freedom and use experience of the user in a complex environment.

[0047] Figure 4 FIG. 1 is a structural schematic diagram of a signal control module provided in the embodiments of the present application, as shown in the figure, in some embodiments, the signal control module 120 includes: Figure 4 The signal preprocessing unit 121 is configured to preprocess the limb motion signal, and the preprocessing includes noise reduction processing, signal enhancement, and normalization processing. The feature extraction unit 122 is configured to extract multi-modal features based on the limb motion signal, and fuse the multi-modal features into a feature vector sequence. The motion recognition unit 123 is configured to recognize the input feature vector sequence based on a deep learning model, to determine the limb motion of the user. The instruction generation unit 124 is configured to generate a control instruction based on the recognized limb motion. The control execution unit 125 is configured to control the smart television based on the control instruction.

[0048] The noise reduction processing includes median filtering and Gaussian filtering processing. The median filtering can effectively remove noise from power fluctuations or transient interference in the environment. The Gaussian filtering is a linear smoothing filter, which can suppress high-frequency noise and make the signal waveform smoother, facilitating subsequent feature extraction. Using an adaptive Gaussian model can obtain a foreground signal containing only limb motion and highlight the motion features. The signal preprocessing unit 121 provided in the embodiments of the present application first removes noise through median filtering, and then smooths high-frequency noise through Gaussian filtering, ensuring the improvement of the signal-to-noise ratio.

[0049] The signal enhancement dynamically adjusts the gain by monitoring the amplitude of the signal in real time, to ensure that the small-amplitude motion signal of the user can be effectively captured. For example, when the user performs a small limb motion (such as opening and closing fingers), the signal amplitude is too low, and an adaptive gain control algorithm can be used to automatically increase the gain; on the contrary, if the signal amplitude is too strong (such as the user waving the arm at a close distance), the gain can be reduced to prevent saturation. The signal preprocessing unit 121 provided in the embodiments of the present application expands the dynamic range of the system through adaptive gain control, ensuring that large-amplitude hand waving and subtle finger movements can be adjusted to an appropriate amplitude level.

[0050] ​The normalization processing can map the amplitude of the signal to the interval [0, 1], thereby eliminating the influence of signal amplitude fluctuations caused by differences in user distance, body size, etc.

[0051] The feature extraction unit 122 adopts a multi-modal feature fusion strategy to extract features of multiple different physical properties from the preprocessed multi-channel time series signal, and performs multi-modal fusion to construct a feature vector sequence as the input of the subsequent deep learning model. The feature extraction unit 122 can perform extraction operations in parallel or sequentially to extract information of multiple physical dimensions, and the model can understand a limb movement from multiple angles, reducing overfitting of the model to a single feature mode and better adapting to limb movement changes in different users and different environments.

[0052] In some embodiments, the multi-modal features include: spatio-temporal features, amplitude features, frequency features, and phase features; the feature extraction unit 122 is further configured to extract spatio-temporal features reflecting the motion trajectory of the limb movement, extract amplitude features reflecting the motion intensity of the limb movement, extract frequency features reflecting the motion frequency of the limb movement, and extract phase features reflecting the motion direction of the limb movement.

[0053] The feature extraction unit 122 can combine the preprocessed signals of a plurality of continuous time frames to construct a “time-space” two-dimensional matrix, and by analyzing the matrix, the motion trajectory and dynamic pattern of the limb movement can be extracted. For example, the waveform of a “slide left” gesture will first appear on the right auxiliary sensor 1112, then on the main sensor 1111, and finally on the left auxiliary sensor 1112, so that the system can identify the spatio-temporal trajectory of the movement.

[0054] The feature extraction unit 122 can calculate the peak amplitude and rate of change of each frame of signal to directly reflect the intensity and amplitude of the limb movement. For example, a “forceful wave” action will produce a signal amplitude and rate of change much higher than a “slight swing”, so that the system can distinguish similar actions with different intentions.

[0055] The feature extraction unit 122 can perform a fast Fourier transform on the continuous frame signals within a certain time window to extract the frequency features of the movement to describe the rhythm and speed of the limb movement. For example, “quick shaking” and “slow swinging” may have similar shapes in the time domain waveform, but their main frequencies will be significantly different, and through the frequency features, the two similar limb movements can be distinguished.

[0056] The feature extraction unit 122 can calculate the phase difference between different sensor channel signals of the same limb movement as a phase feature to determine the movement direction of the limb movement. For example, the positive and negative of the phase difference can directly indicate whether the movement is moving left or right, thereby improving the recognition accuracy of navigation-type sliding movements.

[0057] After the above-mentioned multi-modal feature extraction is completed, the feature extraction unit 122 will continue to perform a fusion operation. For example, the feature extraction unit 122 will perform a fusion operation such as splicing or weighted combination on the spatio-temporal feature vector, the amplitude feature vector, the frequency feature vector, and the phase feature vector extracted in the same time segment, and then form a high-dimensional multi-modal feature vector, and then combine the fusion feature vectors corresponding to the continuous time segments in time sequence to form a feature vector sequence.

[0058] The action recognition unit 123 receives the multi-modal fusion feature vector sequence from the feature extraction unit 122, and processes it based on a deep learning model, and finally outputs the recognition result of the user's limb movement.

[0059] In some embodiments, the deep learning model includes a lightweight CNN-LSTM hybrid model, wherein the lightweight CNN-LSTM hybrid model is processed by model pruning and INT8 quantization.

[0060] The convolutional neural network (CNN) is a local feature extractor, which is responsible for processing the input feature vector sequence to output a high-level feature sequence.

[0061] The long short-term memory network (LSTM) is a special recurrent neural network (RNN), which is a time sequence dependency modeler that receives the high-level feature sequence output by the CNN front end, learns and remembers the long-term dependencies in the sequence.

[0062] In order to adapt to the limited computing power environment of the smart TV, the deep learning model can use a lightweight CNN-LSTM hybrid model processed by model compression. For example, the trained model can be pruned, and then INT8 quantization is performed to convert the original 32-bit floating point numbers in the model to 8-bit integers (INT8). After the above-mentioned model compression processing, the model size can be reduced, the storage requirement is reduced, and the integer operation speed is improved.

[0063] In some embodiments, the action recognition unit 123 is further configured to run the deep learning model on a neural network processing unit of the smart TV.

[0064] The optimized deep learning model is run on a neural processing unit (NPU) integrated in a main control chip of the smart television. The hardware architecture of the NPU is designed for matrix and convolution operations of deep learning calculation, and the execution efficiency is much higher than that of a general-purpose CPU. By running the deep learning model on the NPU, the running speed is greatly improved, and under the premise of not affecting high-precision motion recognition, the system can ensure real-time processing of input data, providing users with a smooth and delay-free interactive experience.

[0065] The instruction generation unit 124 maps the recognized abstract limb motion into a standardized control instruction, and inputs the standardized control instruction into a system interface of the smart television through the control execution unit 125, thereby completing a complete closed loop from perceiving the user's limb motion to controlling the smart television to respond. For example, the mapping table of the limb motion and the control instruction can include: waving hands-waking up / sleeping television, sliding left-channel minus, sliding right-channel plus, sliding up-volume plus, circling-starting a specific application, and liking-collecting current content. It should be understood that the above examples are only a specific embodiment, and will not limit the protection scope of the present application, and in actual use, the user's personalized needs can be preset to meet the user's personalized needs.

[0066] The distributed infrared limb motion recognition system 100 provided by the embodiments of the present application forms a distributed array layout through cooperation of the main sensor 1111, the auxiliary sensor 1112 and the infrared emission matrix 112, expands the viewing angle range, supports the recognition of subtle limb motions, improves the recognition accuracy, can realize complex control such as menu navigation, and greatly improves the detection performance; and only detects the motion through the infrared signal, avoids the risk of privacy leakage, and through user adaptive calibration and personalized model, adapts to the motion habits of different users, so that the interaction is more natural. The system scheme provided by the embodiments of the present application cooperates the main sensor 1111, the auxiliary sensor 1112 and the infrared emission matrix 112, and the cost of this set of infrared modules is relatively low, much lower than that of the millimeter wave scheme and the pure vision scheme, and can be adapted to all sizes of television models, solving the problem that the traditional infrared scheme can only be compatible with specific models and cannot be expanded.

[0067] Figure 5 is a flowchart of a distributed infrared limb motion recognition method provided in the embodiments of the present application. Please refer to Figure 5 The distributed infrared limb motion recognition method can be applied to Figure 1 the distributed infrared limb motion recognition system 100 shown in Figure 1The signal acquisition module 110 and the signal control module 120 in the distributed infrared body action recognition system 100 shown interactively. The meaning of the nouns is the same as that in the distributed infrared body action recognition system 100 described above, and the specific implementation details can be referred to the description of the system embodiment.

[0068] As shown in the figure, the distributed infrared body action recognition method includes the following steps (including steps S400 and S410): Figure 5 As shown in the figure, the distributed infrared body action recognition method includes the following steps (including steps S400 and S410): Step S400: The signal acquisition module acquires the body action signal of the user in the preset detection area; Step S410: The signal control module controls the smart TV according to the body action signal; The signal acquisition module 110 includes a sensor receiving array 111 and an infrared emission matrix 112; the sensor receiving array 111 includes a main sensor 1111 and an auxiliary sensor 1112, the main sensor 1111 is arranged on the top of the smart TV, and the auxiliary sensor 1112 is arranged on the first side frame and the second side frame of the smart TV, the first side frame and the second side frame are arranged opposite to each other; the infrared emission matrix 112 includes a plurality of infrared emission units 1121, and the infrared emission units 1121 are arranged on the backlight lamp strip of the smart TV.

[0069] Through the above steps, the user performs the body action, captures the user's body action and recognizes it, and then controls the smart TV according to the user's body action; the sensor receiving array 111 formed by the triangular layout of the main sensor 1111 and the auxiliary sensor 1112, in cooperation with the plurality of infrared emission units 1121 arranged on the backlight lamp strip, realizes the expansion of the effective detection range of the body action, meets the user's interactive demand, and solves the problem of narrow detection range.

[0070] As a specific example, Figure 6 As another kind of distributed infrared body action recognition method provided in the embodiment of the application, a flowchart is shown. Referring to Figure 6 As shown in the figure, the method includes the following steps: Step S500: The user performs a body action; Step S501: The sensor receiving array acquires the body action signal; Step S502: The signal is preprocessed by noise reduction, enhancement and normalization; Step S503: Extracting time and space, amplitude, frequency and phase features; Step S504: The deep learning model recognizes the body action; Step S505: Generating a standardized control instruction; Step S506: The smart TV control interface executes the control instruction to complete the interaction.

[0071] Through the above steps, a complete closed loop from the user performing a limb action to realizing control of the smart television can be realized.

[0072] The distributed infrared limb recognition method provided by the embodiments of the present application has all the technical effects of the distributed infrared limb recognition system described above, and the present application will not be described again.

[0073] The distributed infrared limb action recognition system 100 and method provided by the embodiments of the present application form a distributed array layout through cooperation of the main sensor 1111 and the auxiliary sensor 1112 with the infrared emission matrix 112, expand the view angle range, support recognition of subtle limb actions at the same time, improve the recognition accuracy, can realize complex control such as menu navigation, and the detection performance is greatly improved; and only through infrared signal detection of actions, the risk of privacy leakage is avoided, through user adaptive calibration and personalized model, the action habits of different users are adapted, and the interaction is more natural. The system scheme provided by the embodiments of the present application, the main sensor 1111 and the auxiliary sensor 1112 cooperate with the infrared emission matrix 112, the cost of this set of infrared modules is lower, far lower than the millimeter wave scheme and the pure vision scheme, and can be adapted to all sizes of television models, solving the problem that the traditional infrared scheme can only be compatible with specific models and cannot be expanded.

[0074] Figure 7 A structural schematic diagram of an electronic device provided in some embodiments of the present application. Figure 7 The dashed line in the figure indicates that the unit or module is optional. Figure 7 The electronic device 600 in the figure can be used to implement the methods described in the method embodiments. The electronic device 600 can be a chip, a terminal device or a first server.

[0075] The electronic device 600 can include one or more processors 610. The processor 610 can support the electronic device 600 to implement the methods described in the foregoing method embodiments. The processor 610 can be a general-purpose processor or a special-purpose processor. For example, the processor can be a central processing unit (CPU). Alternatively, the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0076] The electronic device 600 can further include one or more memories 620. The memories 620 store computer programs. The memories 620 can be independent of the processor 610, or integrated in the processor 610.

[0077] The electronic device 600 can further include a transceiver 630. The processor 610 can communicate with other devices or chips through the transceiver 630. For example, the processor 610 can perform data transceiving with other devices or chips through the transceiver 630.

[0078] The computer programs in the memories 620 can be executed by the processor 610, so that the processor 610 performs steps in any of the distributed infrared body action recognition methods provided in the embodiments of the present application.

[0079] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by an instruction, or by relevant hardware controlled by the instruction, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0080] To this end, the embodiments of the present application provide a computer readable storage medium, which stores computer programs, and the computer programs are loaded by a processor to perform steps described in the above method embodiments of the present application. For example, the computer programs loaded by the processor can perform steps in any of the distributed infrared body action recognition methods provided in the embodiments of the present application.

[0081] The specific implementation of each operation / step can refer to the above embodiments, which will not be repeated here.

[0082] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0083] Since the computer programs stored in the computer readable storage medium can perform steps in any of the above method embodiments provided by the embodiments of the present application, the beneficial effects of the methods described in any of the above method embodiments can be achieved, which will be described in detail in the above embodiments and will not be repeated here.

[0084] The embodiment of the present application further provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method provided in various optional implementation manners in the above embodiment.

[0085] The above describes in detail the distributed infrared limb action recognition system 100 and method provided by the embodiment of the present application. The principle and implementation manner of the present application are described by applying specific examples. The above embodiment is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A distributed infrared limb movement recognition system, characterized in that, The system, applied to smart TVs, includes: The signal acquisition module is used to acquire the user's limb movement signals within a preset detection area; The signal control module is used to control the smart TV based on the limb movement signals; The signal acquisition module includes a sensor receiving array and an infrared transmitting matrix; The sensor receiving array includes a main sensor and an auxiliary sensor. The main sensor is disposed on the top of the smart TV, and the auxiliary sensor is disposed on the first side frame and the second side frame of the smart TV, with the first side frame and the second side frame being disposed opposite to each other. The infrared emission matrix includes multiple infrared emission units, which are disposed on the backlight strip of the smart TV.

2. The system according to claim 1, characterized in that, The position of the infrared emitting unit on the backlight strip is determined based on the screen size of the smart TV and the distribution density of the backlight beads on the backlight strip.

3. The system according to claim 1, characterized in that, The infrared emission matrix emits infrared signals via active pulse modulation, and the sensor receiving array receives the limb movement signals via phase-sensitive detection technology.

4. The system according to any one of claims 1 to 3, characterized in that, The preset detection area is a fan-shaped area; The projected area of ​​the preset detection area on the first plane is larger than the projected area of ​​the smart TV screen on the first plane; wherein, the first plane is parallel to the plane on which the screen is located.

5. The system according to claim 1, characterized in that, The signal control module includes: A signal preprocessing unit is used to preprocess the limb movement signal, the preprocessing including: noise reduction, signal enhancement, and normalization. The feature extraction unit is used to extract multimodal features based on the limb movement signal and fuse the multimodal features into a feature vector sequence; An action recognition unit is used to recognize the input feature vector sequence based on a deep learning model in order to determine the user's limb movements; The instruction generation unit is used to generate control instructions based on the recognized limb movements; A control execution unit is used to control the smart TV based on the control instructions.

6. The system according to claim 5, characterized in that, The multimodal features include: spatiotemporal features, amplitude features, frequency features, and phase features; The feature extraction unit is further configured to extract the spatiotemporal features reflecting the motion trajectory of the limb movement, extract the amplitude features reflecting the motion force of the limb movement, extract the frequency features reflecting the motion frequency of the limb movement, and extract the phase features reflecting the motion direction of the limb movement.

7. The system according to claim 5, characterized in that, The deep learning model includes: a lightweight CNN-LSTM hybrid model; The lightweight CNN-LSTM hybrid model is a model that has undergone pruning and INT8 quantization.

8. The system according to any one of claims 5 to 7, characterized in that, The action recognition unit is also used for: The deep learning model is run on the neural network processing unit of the smart TV.

9. A distributed infrared limb movement recognition method, characterized in that, Applied to smart TVs, the method includes: The signal acquisition module collects the user's limb movement signals within a preset detection area; The signal control module controls the smart TV based on the limb movement signals; The signal acquisition module includes a sensor receiving array and an infrared transmitting matrix; The sensor receiving array includes a main sensor and an auxiliary sensor. The main sensor is disposed on the top of the smart TV, and the auxiliary sensor is disposed on the first side frame and the second side frame of the smart TV, with the first side frame and the second side frame being disposed opposite to each other. The infrared emission matrix includes multiple infrared emission units, which are disposed on the backlight strip of the smart TV.

10. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program or instructions, which, when executed by the processor, cause the processor to perform the steps in the distributed infrared limb movement recognition method as described in claim 9.

11. A computer-readable storage medium, characterized in that, It stores computer programs or instructions, which, when executed by a processor, implement the steps in the distributed infrared limb movement recognition method as described in claim 9.

Images