End-side intelligent interaction system and method applied to ultra-high-definition television

Abstract

The application discloses an end-side intelligent interaction system and method applied to ultra-high-definition televisions, relates to the technical field of intelligent television interaction control, and comprises a collected image containing a color image and a depth image, recognizes hand joint points and generates a gesture segment, generates a gesture pointing point according to a palm center point, an index finger palm joint connecting point and an index finger tip point, and carries out smoothing processing on the gesture pointing point to obtain a smoothed pointing point; generates element feature records according to the smoothed pointing point, a focal point jump relationship and screen element information, constructs a screen element relationship structure, calculates target judgment values of each screen element, determines a target screen element and generates a candidate television control instruction; and then selects an inference mode according to the selection times and the success times of the inference mode to generate a current television control instruction, so that the accuracy and stability of end-side somatosensory interaction of the ultra-high-definition television can be improved, and misrecognition caused by gesture shaking and dense interface elements can be reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent TV interactive control technology, specifically to an end-side intelligent interactive system and method applied to ultra-high-definition televisions. Background Technology

[0002] As the display capabilities and application functions of ultra-high-definition televisions continue to improve, the number of content cards, menu items, playback controls, pop-up options, and settings entry points on the television interface is gradually increasing. Users still primarily rely on remote controls, voice commands, or other external interaction methods when browsing content, controlling playback, adjusting volume, and switching interfaces. In human-computer interaction for devices with screens, remote controls and other auxiliary devices suffer from numerous buttons, long operation steps, and the need for users to switch their attention between the remote control and the display interface. Voice interaction also has limitations in applications in noisy environments or when the device's playback volume is high.

[0003] Chinese invention patent application CN109753154A, published on May 14, 2019, discloses a gesture control method and apparatus for a device with a screen. This method detects gestures used to control the device, and when a preset gesture corresponding to browsing control is detected, generates a corresponding browsing control command, thereby enabling browsing control of the device. The solution also involves mapping the hand position corresponding to the gesture to the display interface, and displaying a click prompt when the mapped position matches the clickable object position.

[0004] However, the aforementioned patent applications primarily focus on converting preset gesture operations into corresponding control commands, or mapping hand positions to the display interface for browsing control judgment. For ultra-high-definition television terminal interaction scenarios, user gestures are easily affected by hand tremors, pointing point offsets, changes in capture depth, and dense arrangement of interface elements. Furthermore, different screen elements have focus jump relationships, identical operation relationships, and identical area relationships. If control commands are generated solely based on preset gestures or a single mapped position, problems such as inaccurate judgment of target screen elements, accidental triggering of control commands, and unstable terminal response can easily occur.

[0005] Therefore, the present invention provides an end-side intelligent interaction system and method for use in ultra-high-definition television. Summary of the Invention

[0006] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an edge-side intelligent interaction system and method for ultra-high-definition televisions. By acquiring images including color and depth images, it identifies hand joints and generates gesture segments. Gesture pointing points are generated based on the palm point, the index finger palm-finger connection point, and the index fingertip point. These pointing points are then smoothed to reduce the impact of gesture jitter on screen pointing results. Simultaneously, element feature records are generated based on the pointing dwell ratio of screen elements, gesture direction consistency value, pointing distance, number of jump steps, operable value, and display area. These features are combined with pointing proximity element groups, focus path element groups, same operation element groups, and same area element groups to calculate target judgment values, thereby more accurately determining the target screen element corresponding to the user's intent.

[0007] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: an end-side intelligent interactive system for ultra-high-definition televisions, comprising an image information acquisition module, a gesture processing module, an element feature generation module, an element relationship judgment module, a reasoning and selection module, and a control execution module. The image information acquisition module is used to acquire captured images, screen interface information, and TV terminal operation information, and output them to the gesture processing module; The gesture processing module is used to generate gesture fragments, gesture pointing points, and smooth pointing points based on the captured images; The element feature generation module is used to generate element feature records of screen elements based on the smooth pointing point and screen interface information. The element relationship judgment module is used to construct the screen element relationship structure based on the element feature record, calculate the target judgment value, determine the target screen element and generate candidate TV control instructions. The reasoning and selection module is used to generate the current TV control command based on candidate TV control commands, gesture fragments, and TV terminal operation information; The control execution module is used to execute the current television control commands.

[0008] End-side intelligent interaction methods applied to ultra-high-definition televisions include: S1. Acquire captured images, screen interface information, and TV terminal operation information; S2. Generate smooth pointing points based on gesture fragments and form screen element feature records; S3. Calculate the target judgment value and generate candidate TV control instructions based on the screen element relationship structure; S4. Select the reasoning method based on the number of selections and the number of successes, and generate the current TV control command.

[0009] Preferably, the captured images of the user in front of the television are acquired in chronological order; each captured image includes a color image and a depth image acquired at the same time. The color images from the acquired images are input into the MediaPipe model to obtain 21 hand joints for each hand, as well as the x and y coordinates of these 21 hand joints. The 21 hand joints include one palmar base point, and four joints each for the thumb, index finger, middle finger, ring finger, and little finger. The palmar base point represents the center position of the palm near the wrist. The four joints of the thumb include the thumb palmar base connection point, the thumb palmar finger connection point, the thumb bending point, and the thumb fingertip point. The four joints of the index finger, middle finger, ring finger, and little finger each include the palmar finger connection point, the first bending point, the second bending point, and the fingertip point. A bounding box for the hand is generated based on the coordinates of 21 hand joints in a color image. Specifically, the minimum x-coordinate and minimum y-coordinate of the 21 hand joints are taken as the coordinates of the upper left corner of the bounding box, and the maximum x-coordinate and maximum y-coordinate of the 21 hand joints are taken as the coordinates of the lower right corner of the bounding box. The width, height, and center point coordinates of the bounding box are then calculated. The palm base point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger are selected from 21 hand joint points. The palm center point is obtained based on the average coordinates of the palm base point, the palmar-finger connection point of the index finger, the palmar-finger connection point of the middle finger, the palmar-finger connection point of the ring finger, and the palmar-finger connection point of the little finger. The corresponding depth values ​​are read from the depth image of the same acquired image based on the coordinates of the palm base point, the palm center point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger in the color image. Then, based on the camera intrinsic parameters of the image acquisition device, the palm base point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger are converted into three-dimensional coordinates in the camera coordinate system. The process involves matching the bounding boxes of hands in two adjacent captured images to obtain gesture segments of the same hand. Specifically, the intersection-union ratio (IUR) and center distance between the bounding boxes of hands in the previous and current captured images are calculated. The IUR and distance ranking values ​​are then added together to obtain a matching score, and a Hungarian algorithm is used to complete one-to-one matching. Successfully matched bounding boxes of hands are assigned to the same gesture segment. Newly appearing bounding boxes of hands in the current captured image that have not yet been matched are used as the starting point of new gesture segments. Bounding boxes of hands that existed in the previous captured image but have not yet been matched in the current captured image are used as the ending point of the corresponding gesture segment. The system acquires screen interface information and TV operating information. The screen interface information includes the current list of screen elements, the current focused element number, the display area of ​​each screen element, TV control operations, executable operations, default operations, focus jump relationships, whether operations are allowed, and the area to which they belong. Screen elements refer to display objects on the TV interface that can be selected, clicked, switched, or adjusted, including buttons, content cards, menu items, playback progress bars, volume bars, and pop-up options. TV control operations refer to the types of operations that the TV can perform after a screen element is selected or triggered by the user, including enter, return, move left, move right, move up, move down, play or pause, increase volume, decrease volume, advance progress, and rewind progress. Executable operations refer to one or more TV control operations that the screen element is allowed to trigger. Default operations refer to a TV control operation triggered by the screen element when no matching TV control operation is found based on the gesture direction. The center point of a screen element is determined by the horizontal and vertical midpoints of its display area. The TV operating information includes processor utilization, NPU utilization, memory utilization, and the number of inference tasks queued.

[0010] Preferably, the hand pointing direction is generated based on the palm point, the palm-finger connection point of the index finger, and the fingertip point of the index finger in the same acquired image; specifically: a first direction vector is generated from the palm point to the fingertip point of the index finger, and a second direction vector is generated from the palm-finger connection point of the index finger to the fingertip point of the index finger; the two direction vectors are normalized and then added together, and the result is normalized again to obtain the hand pointing direction; if the length of one of the direction vectors is zero, only the other direction vector is used; if the lengths of both direction vectors are zero, the acquired image is not included in the subsequent pointing calculation; Starting from the three-dimensional coordinates of the palm point and taking the direction of the hand's pointing as the direction, a hand pointing ray is generated; the intersection of the hand pointing ray and the TV screen plane is calculated as the hand pointing point; if there is no intersection point, or the intersection point is outside the display range of the TV screen, the captured image will not participate in the subsequent pointing calculation; if the intersection point is within the display range of the TV screen, the intersection point is converted into TV screen coordinates. The gesture pointing point is smoothed to obtain a smoothed pointing point. For the first gesture pointing point in the current gesture segment, this gesture pointing point is used as the first smoothed pointing point, and both the horizontal and vertical velocities are set to 0. For the second and subsequent gesture pointing points in the current gesture segment, the differences in the screen horizontal and vertical coordinates between the previous gesture pointing point and its predecessor are calculated first. The difference in the screen horizontal coordinate is used as the previous horizontal velocity, and the difference in the screen vertical coordinate is used as the previous vertical velocity. Then, the screen horizontal coordinate of the previous smoothed pointing point is added to the previous horizontal velocity to obtain the current predicted horizontal coordinate, and the screen vertical coordinate of the previous smoothed pointing point is added to the previous vertical velocity to obtain the current predicted vertical coordinate. Then, the current predicted horizontal coordinate and the screen horizontal coordinate of the current gesture pointing point are averaged to obtain the screen horizontal coordinate of the current smoothed pointing point, and the current predicted vertical coordinate and the screen vertical coordinate of the current gesture pointing point are averaged to obtain the screen vertical coordinate of the current smoothed pointing point. If there is no gesture pointing point in the current gesture segment, no target screen element and TV control command are generated. Calculate the pointing dwell ratio for each screen element; for any screen element, count the number of times the smooth pointing point in the current gesture segment falls into the display area of ​​the screen element, divide the number of times by the total number of smooth pointing points in the current gesture segment to obtain the pointing dwell ratio of the screen element. The gesture direction is calculated based on the first and last smooth pointing points in the current gesture segment. If the difference between their horizontal and vertical coordinates is zero, the gesture direction is stationary. If the absolute value of the difference between the horizontal coordinates is greater than or equal to the absolute value of the difference between the vertical coordinates, the horizontal coordinate difference is positive and the gesture direction is rightward; if the horizontal coordinate difference is negative, the gesture direction is leftward. If the absolute value of the difference between the vertical coordinates is greater than the absolute value of the difference between the horizontal coordinates, the vertical coordinate difference is positive and the gesture direction is upward; if the vertical coordinate difference is negative, the gesture direction is downward. A focus transition graph is generated based on the screen interface information. The focus transition graph includes screen elements and focus transition edges. The focus transition edges indicate that the focus can be switched between two screen elements by moving left, right, up, or down. A breadth-first search algorithm is used to calculate the focus transition path from the current focus element to other screen elements. If a screen element does not have a focus transition path and is not the current focus element, then the screen element will not participate in the subsequent target judgment. For each screen element involved in target judgment, an element feature record is generated. The element feature record includes the pointing dwell ratio, gesture direction consistency value, pointing distance, number of jump steps, operable value, display area, default operation number, and interface area number. The gesture direction consistency value is calculated as follows: if the screen element is the current focus element, the gesture direction consistency value is 1; if the screen element is not the current focus element, and the gesture direction is left, right, up, or down, the gesture direction consistency value is obtained by dividing the number of focus jump edges in the focus jump path with the same direction as the gesture direction by the total number of focus jump edges in that path; if the gesture direction... If the focus is on a specific element, the gesture direction of the currently focused element is 1, and the gesture direction of other screen elements is 0. The pointing distance is the distance from the last smooth pointing point to the center point of the screen element. The jump step count is the number of focus jump edges in the focus jump path. The operable value indicates whether the screen element is allowed to be operated; allowed operation is recorded as 1, and disallowed operation is recorded as 0. The display area is the product of the horizontal length and vertical length of the display area. The default operation number is assigned by the TV to the default operation. The interface area number is assigned by the TV to the menu bar area, content recommendation area, playback control area, pop-up area, and settings area.

[0011] Preferably, a screen element relationship structure is constructed based on the screen elements and element feature records participating in the target judgment; the screen element relationship structure includes element nodes and element groups; each element node corresponds to a screen element participating in the target judgment, and the element group is used to connect one or more screen elements with common relationships; if the number of screen elements participating in the target judgment is zero, no target screen element and TV control instructions are generated. Element groups include elements pointing to nearby elements, elements along the focus path, elements with the same operation, and elements in the same region. The proximity element group is generated as follows: If the number of screen elements participating in the target judgment is 1, then that screen element forms the proximity element group; if the number of screen elements participating in the target judgment is greater than 1, then the number of screen elements participating in the target judgment is denoted as N, where N is a positive integer greater than 1, and the N screen elements are sorted in ascending order of pointing distance to obtain the 1st to Nth screen elements; the position index between two adjacent screen elements is denoted as i, where i is a positive integer not greater than N-1, and the pointing distance difference between the i-th screen element and the (i+1)-th screen element is calculated respectively; the i with the largest pointing distance difference is used as the separator index, and the 1st to i-th screen elements are selected to form the proximity element group; if there are multiple i with the largest pointing distance difference, then the i with the smallest value is selected as the separator index; if all pointing distance differences are zero, then all N screen elements form the proximity element group. A focus path element group consists of all screen elements in the focus jump path from the current focus element to a certain screen element; for each type of TV control operation, screen elements that contain the TV control operation in the executable operation are grouped into a group of elements with the same operation; for each type of interface area number, screen elements with that interface area number are grouped into a group of elements with the same area number; if an element group does not contain any screen elements, then the element group is not constructed.

[0012] Preferably, based on the screen element relationship structure and element feature records, the target judgment value of each screen element is calculated; let M be the number of screen elements participating in the target judgment, where M is a positive integer; for any screen element, calculate the pointing dwell ranking value, direction consistency ranking value, pointing distance ranking value, jump step ranking value, operable ranking value, and element group affiliation ranking value respectively; among them, the pointing dwell ranking value is assigned from M to 1 according to the pointing dwell ratio from large to small, the gesture direction consistency ranking value is assigned from M to 1 according to the gesture direction consistency value from large to small, the pointing distance ranking value is assigned from M to 1 according to the pointing distance from small to large, the jump step ranking value is assigned from M to 1 according to the jump step from small to large, the operable ranking value is assigned from M to 1 according to the operable value from large to small, and the element group affiliation ranking value is assigned from M to 1 according to the number of element group affiliations from large to small. For any screen element, if a larger value is more conducive to identifying the screen element as the target screen element, then the ranking value corresponding to that value is 1 plus the number of other screen elements whose value is less than that screen element; if a smaller value is more conducive to identifying the screen element as the target screen element, then the ranking value corresponding to that value is 1 plus the number of other screen elements whose value is greater than that screen element; the larger the ranking value, the higher the priority of the screen element in the corresponding item; the value includes the pointing dwell ratio, gesture direction consistency value, pointing distance, jump steps, operable value, and number of element group affiliations; The number of element groups is equal to the number of element group types to which the screen element belongs; if the screen element belongs to the nearest element group, the number of element groups is incremented by 1; if the screen element belongs to at least one focus path element group, the number of element groups is incremented by 1; if the screen element belongs to at least one same operation element group, the number of element groups is incremented by 1; if the screen element belongs to at least one same area element group, the number of element groups is incremented by 1. Add the following sort values ​​for the same screen element: the sort value for pointing and staying, the sort value for consistent direction, the sort value for pointing distance, the sort value for jump steps, the sort value for operability, and the sort value for element group affiliation, to obtain the target judgment value for that screen element. The screen element with the highest target judgment value is determined as the target screen element; if two or more screen elements have the same target judgment value, the screen element with the highest pointing dwell ratio, the smallest pointing distance, and the fewest jump steps is selected as the target screen element in sequence. Candidate TV control instructions are generated based on the executable operation, default operation, and gesture direction of the target screen element. If there is a TV control operation among the executable operations of the target screen element that is the same as the gesture direction, then the TV control operation is converted into a candidate TV control instruction. If there is no TV control operation that is the same as the gesture direction, then the default operation of the target screen element is converted into a candidate TV control instruction. If there is no default operation of the target screen element, then no candidate TV control instructions are generated.

[0013] Preferably, two inference methods are established: direct pointing inference and action recognition inference. Direct pointing inference directly generates TV control commands based on target screen elements and candidate TV control commands. Action recognition inference inputs the current gesture segment into a temporal convolutional network model, outputs dynamic gesture categories, and then combines them with the executable operations of the target screen elements to generate TV control commands. The dynamic gesture categories include left waving, right waving, up waving, down waving, hovering pointing, and returning waving. For each inference method, the TV records the number of times that inference method is selected and the number of times it is successfully selected during this power-on period; the number of selections is the number of times that inference method is selected as the current inference method; the number of successes is the number of times that the user does not correct the TV control command generated by that inference method. If there is a reasoning method with 0 selection counts, the reasoning method with 0 selection counts is selected first; if both reasoning methods have 0 selection counts, the direct pointing reasoning method is selected. When the number of selections for both reasoning methods is greater than 0, the path selection value for each reasoning method is calculated. The path selection value is obtained by adding the average success value and the exploration compensation value. The average success value is the number of successes for that reasoning method divided by the number of selections for that reasoning method. The exploration compensation value is calculated by first calculating the natural logarithm of the total number of selections for both reasoning methods during this startup period, then multiplying the natural logarithm by 2, then dividing by the number of selections for that reasoning method, and finally taking the square root of the result. The reasoning method with the largest path selection value is selected as the chosen reasoning method. If the two path selection values ​​are the same, the reasoning method with fewer selections is selected as the chosen reasoning method. If the number of selections is also the same, the direct-pointing reasoning method is selected as the chosen reasoning method.

[0014] Preferably, if the selected reasoning method is the direct-pointing reasoning method and a candidate television control instruction has been generated, then the candidate television control instruction is used as the current television control instruction; if no candidate television control instruction has been generated, then the current television control instruction is not generated. If the selected reasoning method is action recognition reasoning, the current gesture segment is input into the temporal convolutional network model, and the dynamic gesture category is output. The training data of the temporal convolutional network model includes the sequence of the center point of the hand bounding box, the sequence of smooth pointing points, the sequence of hand joints, and the label of the dynamic gesture category. If the TV control operation corresponding to the dynamic gesture category is an executable operation of the target screen element, then the TV control operation is used as the current TV control instruction; otherwise, no current TV control instruction is generated. After the TV executes the current TV control command, it updates the selection count and success count of the selected inference method. Specifically: each time the selected inference method is executed, the selection count of that inference method is incremented by 1; if the user does not perform the opposite control operation on the same target screen element in the next gesture segment, the success count of that inference method is incremented by 1; if the user performs the opposite control operation on the same target screen element in the next gesture segment, or if no current TV control command is generated this time, the success count of that inference method does not increase; opposite control operations include left and right movement, up and down movement, and entering and returning. After generating the current TV control command, the inference selection module sends the current TV control command to the control execution module; the control execution module controls the interface display, playback status, channel status or volume status of the UHD TV according to the TV control operation corresponding to the current TV control command, so that the UHD TV performs the corresponding interactive operation.

[0015] (III) Beneficial Effects This invention provides an edge-side intelligent interaction system and method for ultra-high-definition television, which has the following beneficial effects: 1. By acquiring color and depth images from the collected images, hand joints are obtained, and gesture pointing points are generated based on the palm point, the palm-finger connection point of the index finger, and the fingertip point of the index finger. This allows the gesture pointing result to combine the position and depth information of the two-dimensional image, which can reduce the pointing deviation caused by changes in distance and hand angle when relying solely on color image recognition.

[0016] 2. By smoothing the gesture pointing point, a smooth pointing point is obtained. Based on the smooth pointing point, the pointing dwell ratio, pointing distance and gesture direction are calculated. This makes the subsequent judgment of target screen elements not directly dependent on a single instantaneous pointing point, which can reduce misselection and mis-triggering caused by the user's hand shaking and improve the stability of the motion interaction process.

[0017] 3. By constructing a screen element relationship structure based on the pointing proximity relationship, focus path relationship, same operation relationship, and same area relationship of screen elements, and combining the pointing dwell ratio, gesture direction consistency value, pointing distance, jump steps, operable value, and display area to calculate the target judgment value, the judgment of target screen elements not only considers the pointing position of the gesture, but also the logical relationship between screen elements in the TV interface, thereby improving the accuracy of target recognition in complex interfaces.

[0018] 4. The path selection value is calculated by the number of times the direct pointing reasoning method and the action recognition reasoning method are selected and the number of successful selections. Based on this, the selected reasoning method is selected. This allows the system to adjust the reasoning method according to the actual interaction results during the current boot-up period. Under the condition of limited computing power on the edge, it can balance response speed and control accuracy and reduce the impact of continuous misidentification on the user interaction experience. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the end-side intelligent interactive system structure of the present invention applied to ultra-high-definition television; Figure 2 This is a schematic diagram of the end-side intelligent interaction method of the present invention applied to ultra-high-definition television. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 This invention provides an edge-side intelligent interactive system for ultra-high-definition televisions, comprising an image information acquisition module, a gesture processing module, an element feature generation module, an element relationship judgment module, a reasoning and selection module, and a control execution module. The image information acquisition module is used to acquire captured images, screen interface information, and TV terminal operation information, and output them to the gesture processing module; The gesture processing module is used to generate gesture fragments, gesture pointing points, and smooth pointing points based on the captured images; The element feature generation module is used to generate element feature records of screen elements based on the smooth pointing point and screen interface information. The element relationship judgment module is used to construct the screen element relationship structure based on the element feature record, calculate the target judgment value, determine the target screen element and generate candidate TV control instructions. The reasoning and selection module is used to generate the current TV control command based on candidate TV control commands, gesture fragments, and TV terminal operation information; The control execution module is used to execute the current television control commands.

[0022] Please see Figure 2 This invention provides an edge-side intelligent interaction method for use in ultra-high-definition televisions, comprising the following steps: S1. Acquire captured images, screen interface information, and TV operating information.

[0023] The images captured from the user in front of the television are obtained in chronological order; each captured image includes a color image and a depth image acquired at the same time, and the depth image is used to provide the depth value of the corresponding position in the color image; The color images from the acquired images are input into the MediaPipe model to obtain 21 hand joints for each hand, as well as the x and y coordinates of these 21 hand joints. The 21 hand joints include one palmar base point, and four joints each for the thumb, index finger, middle finger, ring finger, and little finger. The palmar base point represents the center position of the palm near the wrist. The four joints of the thumb include the thumb palmar base connection point, the thumb palmar finger connection point, the thumb bending point, and the thumb fingertip point. The four joints of the index finger, middle finger, ring finger, and little finger each include the palmar finger connection point, the first bending point, the second bending point, and the fingertip point. A bounding box for the hand is generated based on the coordinates of 21 hand joints in a color image. Specifically, the minimum x-coordinate and minimum y-coordinate of the 21 hand joints are taken as the coordinates of the upper left corner of the bounding box, and the maximum x-coordinate and maximum y-coordinate of the 21 hand joints are taken as the coordinates of the lower right corner of the bounding box. The width, height, and center point coordinates of the bounding box are then calculated. The palm base point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger are selected from 21 hand joint points. The palm center point is obtained based on the average coordinates of the palm base point, the palmar-finger connection point of the index finger, the palmar-finger connection point of the middle finger, the palmar-finger connection point of the ring finger, and the palmar-finger connection point of the little finger. The corresponding depth values ​​are read from the depth image of the same acquired image based on the coordinates of the palm base point, the palm center point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger in the color image. Then, based on the camera intrinsic parameters of the image acquisition device, the palm base point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger are converted into three-dimensional coordinates in the camera coordinate system. The process involves matching the bounding boxes of hands in two adjacent captured images to obtain gesture segments of the same hand. Specifically, the intersection-union ratio (IUR) and the center distance between the bounding boxes of hands in the previous and current captured images are calculated. A larger IUR results in a higher IU ranking value, while a smaller center distance results in a higher center distance ranking value. The IUR ranking value and the center distance ranking value are added together to obtain a matching score, and a Hungarian algorithm is used to complete one-to-one matching. Successfully matched bounding boxes of hands are grouped into the same gesture segment. Newly appearing bounding boxes of hands in the current captured image that have not yet been matched are used as the starting point of new gesture segments. Bounding boxes of hands that existed in the previous captured image but have not yet been matched in the current captured image are used as the ending point of the corresponding gesture segment. The system acquires screen interface information and TV operating information. The screen interface information includes the current list of screen elements, the current focused element number, the display area of ​​each screen element, TV control operations, executable operations, default operations, focus jump relationships, whether operations are allowed, and the area to which they belong. Screen elements refer to display objects on the TV interface that can be selected, clicked, switched, or adjusted, including buttons, content cards, menu items, playback progress bars, volume bars, and pop-up options. TV control operations refer to the types of operations that the TV can perform after a screen element is selected or triggered by the user, including enter, return, move left, move right, move up, move down, play or pause, increase volume, decrease volume, advance progress, and rewind progress. Executable operations refer to one or more TV control operations that the screen element is allowed to trigger. Default operations refer to a TV control operation triggered by the screen element when no matching TV control operation is found based on the gesture direction. The center point of a screen element is determined by the horizontal and vertical midpoints of its display area. The TV operating information includes processor utilization, NPU utilization, memory utilization, and the number of inference tasks queued.

[0024] S2. Generate smooth pointing points based on gesture fragments and form screen element feature records.

[0025] The hand pointing direction is generated based on the palm point, the palm-finger connection point of the index finger, and the fingertip point of the index finger in the same acquired image. Specifically, a first direction vector is generated from the palm point to the fingertip point of the index finger, and a second direction vector is generated from the palm-finger connection point of the index finger to the fingertip point of the index finger. The two direction vectors are normalized separately and then added together, and the result is normalized again to obtain the hand pointing direction. If the length of one of the direction vectors is zero, only the other direction vector is used. If the lengths of both direction vectors are zero, the acquired image is not included in the subsequent pointing calculation. Starting from the three-dimensional coordinates of the palm point and taking the direction of the hand's pointing as the direction, a hand pointing ray is generated; the intersection of the hand pointing ray and the TV screen plane is calculated as the hand pointing point; if there is no intersection point, or the intersection point is outside the display range of the TV screen, the captured image will not participate in the subsequent pointing calculation; if the intersection point is within the display range of the TV screen, the intersection point is converted into TV screen coordinates. The gesture pointing point is smoothed to obtain a smoothed pointing point. For the first gesture pointing point in the current gesture segment, this gesture pointing point is used as the first smoothed pointing point, and both the horizontal and vertical velocities are set to 0. For the second and subsequent gesture pointing points in the current gesture segment, the differences in the screen horizontal and vertical coordinates between the previous gesture pointing point and its predecessor are calculated first. The difference in the screen horizontal coordinate is used as the previous horizontal velocity, and the difference in the screen vertical coordinate is used as the previous vertical velocity. Then, the screen horizontal coordinate of the previous smoothed pointing point is added to the previous horizontal velocity to obtain the current predicted horizontal coordinate, and the screen vertical coordinate of the previous smoothed pointing point is added to the previous vertical velocity to obtain the current predicted vertical coordinate. Then, the current predicted horizontal coordinate and the screen horizontal coordinate of the current gesture pointing point are averaged to obtain the screen horizontal coordinate of the current smoothed pointing point, and the current predicted vertical coordinate and the screen vertical coordinate of the current gesture pointing point are averaged to obtain the screen vertical coordinate of the current smoothed pointing point. If there is no gesture pointing point in the current gesture segment, no target screen element and TV control command are generated. Calculate the pointing dwell ratio for each screen element; for any screen element, count the number of times the smooth pointing point in the current gesture segment falls into the display area of ​​the screen element, divide the number of times by the total number of smooth pointing points in the current gesture segment to obtain the pointing dwell ratio of the screen element. The gesture direction is calculated based on the first and last smooth pointing points in the current gesture segment. If the difference between their horizontal and vertical coordinates is zero, the gesture direction is stationary. If the absolute value of the difference between the horizontal coordinates is greater than or equal to the absolute value of the difference between the vertical coordinates, the horizontal coordinate difference is positive and the gesture direction is rightward; if the horizontal coordinate difference is negative, the gesture direction is leftward. If the absolute value of the difference between the vertical coordinates is greater than the absolute value of the difference between the horizontal coordinates, the vertical coordinate difference is positive and the gesture direction is upward; if the vertical coordinate difference is negative, the gesture direction is downward. A focus transition graph is generated based on the screen interface information. The focus transition graph includes screen elements and focus transition edges. The focus transition edges indicate that the focus can be switched between two screen elements by moving left, right, up, or down. A breadth-first search algorithm is used to calculate the focus transition path from the current focus element to other screen elements. If a screen element does not have a focus transition path and is not the current focus element, then the screen element will not participate in the subsequent target judgment. For each screen element involved in target judgment, an element feature record is generated. The element feature record includes the pointing dwell ratio, gesture direction consistency value, pointing distance, number of jump steps, operable value, display area, default operation number, and interface area number. The gesture direction consistency value is calculated as follows: if the screen element is the current focus element, the gesture direction consistency value is 1; if the screen element is not the current focus element, and the gesture direction is left, right, up, or down, the gesture direction consistency value is obtained by dividing the number of focus jump edges in the focus jump path with the same direction as the gesture direction by the total number of focus jump edges in that path; if the gesture direction... If the focus is on a specific element, the gesture direction of the currently focused element is 1, and the gesture direction of other screen elements is 0. The pointing distance is the distance from the last smooth pointing point to the center point of the screen element. The jump step count is the number of focus jump edges in the focus jump path. The operable value indicates whether the screen element is allowed to be operated; allowed operation is recorded as 1, and disallowed operation is recorded as 0. The display area is the product of the horizontal length and vertical length of the display area. The default operation number is assigned by the TV to the default operation. The interface area number is assigned by the TV to the menu bar area, content recommendation area, playback control area, pop-up area, and settings area.

[0026] S3. Calculate the target judgment value based on the screen element relationship structure and generate candidate TV control instructions.

[0027] Based on the screen elements and element feature records involved in the target determination, a screen element relationship structure is constructed. The screen element relationship structure includes element nodes and element groups. Each element node corresponds to a screen element involved in the target determination, and the element group is used to connect one or more screen elements with common relationships. If the number of screen elements involved in the target determination is zero, no target screen element and TV control instructions are generated. Element groups include elements pointing to nearby elements, elements along the focus path, elements with the same operation, and elements in the same region. The proximity element group is generated as follows: If the number of screen elements participating in the target judgment is 1, then that screen element forms the proximity element group; if the number of screen elements participating in the target judgment is greater than 1, then the number of screen elements participating in the target judgment is denoted as N, where N is a positive integer greater than 1, and the N screen elements are sorted in ascending order of pointing distance to obtain the 1st to Nth screen elements; the position index between two adjacent screen elements is denoted as i, where i is a positive integer not greater than N-1, and the pointing distance difference between the i-th screen element and the (i+1)-th screen element is calculated respectively; the i with the largest pointing distance difference is used as the separator index, and the 1st to i-th screen elements are selected to form the proximity element group; if there are multiple i with the largest pointing distance difference, then the i with the smallest value is selected as the separator index; if all pointing distance differences are zero, then all N screen elements form the proximity element group. The focus path element group consists of all screen elements in the focus jump path from the current focus element to a certain screen element; for each TV control operation, screen elements that contain the TV control operation in the executable operation are grouped into a group of elements with the same operation; for each interface area number, screen elements with that interface area number are grouped into a group of elements with the same area number; if an element group does not contain any screen elements, then the element group is not constructed. Based on the screen element relationship structure and element feature records, calculate the target judgment value for each screen element; let M be the number of screen elements participating in the target judgment, where M is a positive integer; for any screen element, calculate the pointing dwell ranking value, direction consistency ranking value, pointing distance ranking value, jump step ranking value, operable ranking value, and element group affiliation ranking value respectively; among them, the pointing dwell ranking value is assigned from M to 1 according to the pointing dwell ratio from large to small, the gesture direction consistency ranking value is assigned from M to 1 according to the gesture direction consistency value from large to small, the pointing distance ranking value is assigned from M to 1 according to the pointing distance from small to large, the jump step ranking value is assigned from M to 1 according to the jump step from small to large, the operable ranking value is assigned from M to 1 according to the operable value from large to small, and the element group affiliation ranking value is assigned from M to 1 according to the number of element group affiliations from large to small. For any screen element, if a larger value is more conducive to identifying the screen element as the target screen element, then the ranking value corresponding to that value is 1 plus the number of other screen elements whose value is less than that screen element; if a smaller value is more conducive to identifying the screen element as the target screen element, then the ranking value corresponding to that value is 1 plus the number of other screen elements whose value is greater than that screen element; the larger the ranking value, the higher the priority of the screen element in the corresponding item; the value includes the pointing dwell ratio, gesture direction consistency value, pointing distance, jump steps, operable value, and number of element group affiliations; The number of element groups is equal to the number of element group types to which the screen element belongs; if the screen element belongs to the nearest element group, the number of element groups is incremented by 1; if the screen element belongs to at least one focus path element group, the number of element groups is incremented by 1; if the screen element belongs to at least one same operation element group, the number of element groups is incremented by 1; if the screen element belongs to at least one same area element group, the number of element groups is incremented by 1. Add the following sort values ​​for the same screen element: the sort value for pointing and staying, the sort value for consistent direction, the sort value for pointing distance, the sort value for jump steps, the sort value for operability, and the sort value for element group affiliation, to obtain the target judgment value for that screen element. The screen element with the highest target judgment value is determined as the target screen element; if two or more screen elements have the same target judgment value, the screen element with the highest pointing dwell ratio, the smallest pointing distance, and the fewest jump steps is selected as the target screen element in sequence. Candidate TV control instructions are generated based on the executable operation, default operation, and gesture direction of the target screen element. If there is a TV control operation among the executable operations of the target screen element that is the same as the gesture direction, then the TV control operation is converted into a candidate TV control instruction. If there is no TV control operation that is the same as the gesture direction, then the default operation of the target screen element is converted into a candidate TV control instruction. If there is no default operation of the target screen element, then no candidate TV control instructions are generated.

[0028] S4. Select the reasoning method based on the number of selections and the number of successes, and generate the current TV control command.

[0029] Two inference methods are established: direct pointing inference and action recognition inference. Direct pointing inference directly generates TV control commands based on target screen elements and candidate TV control commands. Action recognition inference inputs the current gesture segment into a temporal convolutional network model, outputs dynamic gesture categories, and then combines them with the executable operations of the target screen elements to generate TV control commands. The dynamic gesture categories include left waving, right waving, up waving, down waving, hovering pointing, and returning waving. For each inference method, the TV records the number of times that inference method is selected and the number of times it is successfully selected during this power-on period; the number of selections is the number of times that inference method is selected as the current inference method; the number of successes is the number of times that the user does not correct the TV control command generated by that inference method. If there is a reasoning method with 0 selection counts, the reasoning method with 0 selection counts is selected first; if both reasoning methods have 0 selection counts, the direct pointing reasoning method is selected. When the number of selections for both reasoning methods is greater than 0, calculate the path selection value for each reasoning method. The path selection value is obtained by adding the average success value and the exploration compensation value. The average success value is the number of successes for that reasoning method divided by the number of selections for that reasoning method. The exploration compensation value is calculated by first calculating the natural logarithm of the total number of selections for both reasoning methods during this startup period, then multiplying the natural logarithm by 2, then dividing by the number of selections for that reasoning method, and finally taking the square root of the result. Select the reasoning method with the largest path selection value as the selected reasoning method. If the two path selection values ​​are the same, select the reasoning method with fewer selections as the selected reasoning method. If the number of selections is also the same, select the direct-pointing reasoning method as the selected reasoning method. If the selected reasoning method is the direct-pointing reasoning method and a candidate television control command has already been generated, then the candidate television control command will be used as the current television control command; if no candidate television control command has been generated, then the current television control command will not be generated. If the selected reasoning method is action recognition reasoning, the current gesture segment is input into the temporal convolutional network model, and the dynamic gesture category is output. The training data of the temporal convolutional network model includes the sequence of the center point of the hand bounding box, the sequence of smooth pointing points, the sequence of hand joints, and the label of the dynamic gesture category. If the TV control operation corresponding to the dynamic gesture category is an executable operation of the target screen element, then the TV control operation is used as the current TV control instruction; otherwise, no current TV control instruction is generated. After the TV executes the current TV control command, it updates the selection count and success count of the selected inference method. Specifically: each time the selected inference method is executed, the selection count of that inference method is incremented by 1; if the user does not perform the opposite control operation on the same target screen element in the next gesture segment, the success count of that inference method is incremented by 1; if the user performs the opposite control operation on the same target screen element in the next gesture segment, or if no current TV control command is generated this time, the success count of that inference method does not increase; opposite control operations include left and right movement, up and down movement, and entering and returning. After generating the current TV control command, the inference selection module sends the current TV control command to the control execution module; the control execution module controls the interface display, playback status, channel status or volume status of the UHD TV according to the TV control operation corresponding to the current TV control command, so that the UHD TV performs the corresponding interactive operation.

[0030] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0031] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0032] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An edge-side intelligent interactive system for ultra-high-definition televisions, comprising an image information acquisition module, a gesture processing module, an element feature generation module, an element relationship judgment module, a reasoning and selection module, and a control execution module: The image information acquisition module is used to acquire captured images, screen interface information, and TV terminal operation information, and output them to the gesture processing module; The gesture processing module is used to generate gesture fragments, gesture pointing points, and smooth pointing points based on the captured images; The element feature generation module is used to generate element feature records of screen elements based on the smooth pointing point and screen interface information. The element relationship judgment module is used to construct the screen element relationship structure based on the element feature record, calculate the target judgment value, determine the target screen element and generate candidate TV control instructions. The reasoning and selection module is used to generate the current TV control command based on candidate TV control commands, gesture fragments, and TV terminal operation information; The control execution module is used to execute the current television control commands.

2. End-side intelligent interaction methods applied to ultra-high-definition televisions, including: S1. Acquire captured images, screen interface information, and TV terminal operation information; S2. Generate smooth pointing points based on gesture fragments and form screen element feature records; S3. Calculate the target judgment value and generate candidate TV control instructions based on the screen element relationship structure; S4. Select the reasoning method based on the number of selections and the number of successes, and generate the current TV control command.

3. The end-side intelligent interaction method for ultra-high-definition television according to claim 2, characterized in that: S1 acquires the captured image, screen interface information, and TV operating information as follows: The images captured from the user in front of the television are obtained in chronological order; each captured image includes a color image and a depth image acquired at the same time. The color images from the acquired images are input into the MediaPipe model to obtain 21 hand joints for each hand, as well as the x and y coordinates of these 21 hand joints. The 21 hand joints include one palmar base point, and four joints each for the thumb, index finger, middle finger, ring finger, and little finger. The palmar base point represents the center position of the palm near the wrist. The four joints of the thumb include the thumb palmar base connection point, the thumb palmar finger connection point, the thumb bending point, and the thumb fingertip point. The four joints of the index finger, middle finger, ring finger, and little finger each include the palmar finger connection point, the first bending point, the second bending point, and the fingertip point. A bounding box for the hand is generated based on the coordinates of 21 hand joints in a color image. Specifically, the minimum x-coordinate and minimum y-coordinate of the 21 hand joints are taken as the coordinates of the upper left corner of the bounding box, and the maximum x-coordinate and maximum y-coordinate of the 21 hand joints are taken as the coordinates of the lower right corner of the bounding box. The width, height, and center point coordinates of the bounding box are then calculated. The palm base point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger are selected from 21 hand joint points. The palm center point is obtained based on the average coordinates of the palm base point, the palmar-finger connection point of the index finger, the palmar-finger connection point of the middle finger, the palmar-finger connection point of the ring finger, and the palmar-finger connection point of the little finger. The corresponding depth values ​​are read from the depth image of the same acquired image based on the coordinates of the palm base point, the palm center point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger in the color image. Then, based on the camera intrinsic parameters of the image acquisition device, the palm base point, the palmar-finger connection point of the index finger, and the fingertip point of the index finger are converted into three-dimensional coordinates in the camera coordinate system. The process involves matching the bounding boxes of hands in two adjacent captured images to obtain gesture segments of the same hand. Specifically, the intersection-union ratio (IUR) and center distance between the bounding boxes of hands in the previous and current captured images are calculated. The IUR and distance ranking values ​​are then added together to obtain a matching score, and a Hungarian algorithm is used to complete one-to-one matching. Successfully matched bounding boxes of hands are assigned to the same gesture segment. Newly appearing bounding boxes of hands in the current captured image that have not yet been matched are used as the starting point of new gesture segments. Bounding boxes of hands that existed in the previous captured image but have not yet been matched in the current captured image are used as the ending point of the corresponding gesture segment. The system acquires screen interface information and TV operating information. The screen interface information includes the current list of screen elements, the current focused element number, the display area of ​​each screen element, TV control operations, executable operations, default operations, focus jump relationships, whether operations are allowed, and the area to which they belong. Screen elements refer to display objects on the TV interface that can be selected, clicked, switched, or adjusted, including buttons, content cards, menu items, playback progress bars, volume bars, and pop-up options. TV control operations refer to the types of operations that the TV can perform after a screen element is selected or triggered by the user, including enter, return, move left, move right, move up, move down, play or pause, increase volume, decrease volume, advance progress, and rewind progress. Executable operations refer to one or more TV control operations that the screen element is allowed to trigger. Default operations refer to a TV control operation triggered by the screen element when no matching TV control operation is found based on the gesture direction. The center point of a screen element is determined by the horizontal and vertical midpoints of its display area. The TV operating information includes processor utilization, NPU utilization, memory utilization, and the number of inference tasks queued.

4. The end-side intelligent interaction method for ultra-high-definition television according to claim 2, characterized in that: In S2, smooth pointing points are generated based on gesture fragments, and screen element feature records are formed as follows: The hand pointing direction is generated based on the palm point, the palm-finger connection point of the index finger, and the fingertip point of the index finger in the same acquired image. Specifically, a first direction vector is generated from the palm point to the fingertip point of the index finger, and a second direction vector is generated from the palm-finger connection point of the index finger to the fingertip point of the index finger. The two direction vectors are normalized separately and then added together, and the result is normalized again to obtain the hand pointing direction. If the length of one of the direction vectors is zero, only the other direction vector is used. If the lengths of both direction vectors are zero, the acquired image is not included in the subsequent pointing calculation. Using the three-dimensional coordinates of the palm point as the starting point and the direction of the hand pointing as the direction, a hand pointing ray is generated; the intersection of the hand pointing ray and the TV screen plane is calculated as the gesture pointing point; If there is no intersection point, or the intersection point is outside the display area of ​​the TV screen, the captured image will not participate in the subsequent pointing calculation; if the intersection point is within the display area of ​​the TV screen, the intersection point will be converted into TV screen coordinates. The gesture pointing point is smoothed to obtain a smooth pointing point; for the first gesture pointing point in the current gesture segment, the gesture pointing point is taken as the first smooth pointing point, and the horizontal velocity and vertical velocity are both set to 0; for the second and subsequent gesture pointing points in the current gesture segment, the difference between the screen horizontal coordinate and the screen vertical coordinate between the previous gesture pointing point and the previous gesture pointing point is calculated first, and the difference between the screen horizontal coordinate is taken as the previous horizontal velocity, and the difference between the screen vertical coordinate is taken as the previous vertical velocity; Add the previous horizontal velocity to the screen horizontal coordinate of the previous smoothed pointing point to obtain the current predicted horizontal coordinate; add the previous vertical velocity to the screen vertical coordinate of the previous smoothed pointing point to obtain the current predicted vertical coordinate. Then, the screen x-coordinate of the current predicted x-coordinate is averaged with the screen x-coordinate of the current gesture pointing point to obtain the screen x-coordinate of the current smooth pointing point. The screen y-coordinate of the current predicted y-coordinate is averaged with the screen y-coordinate of the current gesture pointing point to obtain the screen y-coordinate of the current smooth pointing point. If there is no gesture pointing point in the current gesture fragment, no target screen element and TV control command will be generated. Calculate the pointing dwell ratio for each screen element; for any screen element, count the number of times the smooth pointing point in the current gesture segment falls into the display area of ​​the screen element, divide the number of times by the total number of smooth pointing points in the current gesture segment to obtain the pointing dwell ratio of the screen element. The gesture direction is calculated based on the first and last smooth pointing points in the current gesture segment; if the difference between their horizontal and vertical coordinates is zero, the gesture direction is stationary. If the absolute value of the difference in the horizontal coordinates is greater than or equal to the absolute value of the difference in the vertical coordinates, the difference in the horizontal coordinates is positive and the gesture direction is to move to the right; if the difference in the horizontal coordinates is negative, the gesture direction is to move to the left. If the absolute value of the difference in the vertical coordinates is greater than the absolute value of the difference in the horizontal coordinates, the vertical coordinate difference is positive and the gesture direction is upward; if the vertical coordinate difference is negative, the gesture direction is downward. A focus transition graph is generated based on the screen interface information. The focus transition graph includes screen elements and focus transition edges. The focus transition edges indicate that the focus can be switched between two screen elements by moving left, right, up, or down. A breadth-first search algorithm is used to calculate the focus transition path from the current focus element to other screen elements. If a screen element does not have a focus transition path and is not the current focus element, then the screen element will not participate in the subsequent target judgment. Generate an element feature record for each screen element involved in the target determination; The element feature record includes the pointing dwell ratio, gesture direction consistency value, pointing distance, number of jump steps, operable value, display area, default operation number, and interface area number; the gesture direction consistency value is calculated as follows: if the screen element is the currently focused element, the gesture direction consistency value is 1; If the screen element is not the currently focused element, and the gesture direction is left, right, up, or down, then the number of focus jump edges in the focus jump path with the same direction as the gesture direction is divided by the total number of focus jump edges in that path to obtain the gesture direction consistency value. If the gesture direction is a pause, the gesture direction value of the currently focused element is 1, and the gesture direction value of other screen elements is 0. The pointing distance is the distance from the last smooth pointing point to the center point of the screen element; The number of jump steps is the number of focus jump edges in the focus jump path; The operable value indicates whether an element on the screen is allowed to be operated; 1 indicates allowed operation and 0 indicates disallowed operation. The display area is the product of the horizontal and vertical lengths of the display area. The default operation number is assigned by the TV to the default operation. The interface area number is assigned by the TV to the menu bar area, content recommendation area, playback control area, pop-up area, and settings area.

5. The end-side intelligent interaction method for ultra-high-definition television according to claim 2, characterized in that: In S3, the target judgment value is calculated based on the screen element relationship structure, and candidate TV control instructions are generated as follows: Based on the screen elements and element feature records involved in the target judgment, a screen element relationship structure is constructed. The screen element relationship structure includes element nodes and element groups. Each element node corresponds to a screen element involved in the target judgment, and element groups are used to connect one or more screen elements with common relationships. If the number of screen elements participating in the target determination is zero, then no target screen element and TV control instructions will be generated. Element groups include elements pointing to nearby elements, elements along the focus path, elements with the same operation, and elements in the same region. The nearest element group is generated as follows: if the number of screen elements participating in the target judgment is 1, then the nearest element group is formed by that screen element. If the number of screen elements participating in the target judgment is greater than 1, then the number of screen elements participating in the target judgment is recorded as N, where N is a positive integer greater than 1, and the N screen elements are sorted in ascending order of pointing distance to obtain the 1st to Nth screen elements. Let i be the position index between two adjacent screen elements, where i is a positive integer not greater than N-1. Calculate the difference in pointing distance between the i-th screen element and the (i+1)-th screen element. Use i, which has the largest difference in pointing distance, as the separator index. Select the 1st to the i-th screen elements to form a group of elements with close pointing distances. If there are multiple i with the largest difference in pointing distance, then the i with the smallest value is selected as the separator number; If the difference between all pointing distances is zero, then all N screen elements form a group pointing to the nearest element. A focus path element group consists of all screen elements in the focus jump path from the current focus element to a certain screen element; for each type of TV control operation, screen elements that contain the TV control operation in the executable operation are grouped into a group of elements with the same operation; for each type of interface area number, screen elements with that interface area number are grouped into a group of elements with the same area number; if an element group does not contain any screen elements, then the element group is not constructed.

6. The end-side intelligent interaction method for ultra-high-definition television according to claim 2, characterized in that: In S3, the target judgment value is calculated based on the screen element relationship structure, and candidate TV control instructions are generated as follows: Based on the screen element relationship structure and element feature records, calculate the target judgment value for each screen element; let M be the number of screen elements participating in the target judgment, where M is a positive integer; for any screen element, calculate the pointing dwell ranking value, direction consistency ranking value, pointing distance ranking value, jump step ranking value, operable ranking value, and element group affiliation ranking value respectively; among them, the pointing dwell ranking value is assigned from M to 1 according to the pointing dwell ratio from large to small, the gesture direction consistency ranking value is assigned from M to 1 according to the gesture direction consistency value from large to small, the pointing distance ranking value is assigned from M to 1 according to the pointing distance from small to large, the jump step ranking value is assigned from M to 1 according to the jump step from small to large, the operable ranking value is assigned from M to 1 according to the operable value from large to small, and the element group affiliation ranking value is assigned from M to 1 according to the number of element group affiliations from large to small. For any screen element, if a larger value is more conducive to identifying the screen element as the target screen element, then the ranking value corresponding to that value is 1 plus the number of other screen elements whose value is less than that screen element; if a smaller value is more conducive to identifying the screen element as the target screen element, then the ranking value corresponding to that value is 1 plus the number of other screen elements whose value is greater than that screen element; the larger the ranking value, the higher the priority of the screen element in the corresponding item; the value includes the pointing dwell ratio, gesture direction consistency value, pointing distance, jump steps, operable value, and number of element group affiliations; The number of element groups is equal to the number of element group types to which the screen element belongs; if the screen element belongs to the nearest element group, the number of element groups is incremented by 1; if the screen element belongs to at least one focus path element group, the number of element groups is incremented by 1; if the screen element belongs to at least one same operation element group, the number of element groups is incremented by 1; if the screen element belongs to at least one same area element group, the number of element groups is incremented by 1. Add the following sort values ​​for the same screen element: the sort value for pointing and staying, the sort value for consistent direction, the sort value for pointing distance, the sort value for jump steps, the sort value for operability, and the sort value for element group affiliation, to obtain the target judgment value for that screen element. The screen element with the highest target judgment value is determined as the target screen element; if two or more screen elements have the same target judgment value, the screen element with the highest pointing dwell ratio, the smallest pointing distance, and the fewest jump steps is selected as the target screen element in sequence. Candidate TV control instructions are generated based on the executable operation, default operation, and gesture direction of the target screen element. If there is a TV control operation among the executable operations of the target screen element that is the same as the gesture direction, then the TV control operation is converted into a candidate TV control instruction. If there is no TV control operation that is the same as the gesture direction, then the default operation of the target screen element is converted into a candidate TV control instruction. If there is no default operation of the target screen element, then no candidate TV control instructions are generated.

7. The end-side intelligent interaction method for ultra-high-definition television according to claim 2, characterized in that: In S4, the inference method is selected based on the number of selections and the number of successes, and the current TV control command is generated as follows: Two inference methods are established: direct pointing inference and action recognition inference. Direct pointing inference directly generates TV control commands based on target screen elements and candidate TV control commands. Action recognition inference inputs the current gesture segment into a temporal convolutional network model, outputs dynamic gesture categories, and then combines them with the executable operations of the target screen elements to generate TV control commands. The dynamic gesture categories include left waving, right waving, up waving, down waving, hovering pointing, and returning waving. For each reasoning method, the TV records the number of times that reasoning method was selected and the number of times it was successfully selected during this broadcast. The selection count is the number of times that reasoning method is selected as the current reasoning method; The number of successful attempts is the number of times that the TV control commands generated by this reasoning method were not corrected by the user. If there is a reasoning method with 0 selection counts, the reasoning method with 0 selection counts is selected first; if both reasoning methods have 0 selection counts, the direct pointing reasoning method is selected. When the number of selections for both reasoning methods is greater than 0, the path selection value for each reasoning method is calculated separately; the path selection value is obtained by adding the average success value and the exploration compensation value. The average success value is the number of successful inferences for that reasoning method divided by the number of times that reasoning method was selected. The exploration compensation value is calculated as follows: first, the natural logarithm of the total number of times the two reasoning methods were selected during the current startup period is calculated; then, the natural logarithm is multiplied by 2; then, it is divided by the number of times that reasoning method was selected; finally, the square root of the result is taken. The reasoning method with the largest path selection value is selected as the chosen reasoning method. If the two path selection values ​​are the same, the reasoning method with fewer selections is selected as the chosen reasoning method. If the number of selections is also the same, the direct-pointing reasoning method is selected as the chosen reasoning method.

8. The end-side intelligent interaction method for ultra-high-definition television according to claim 2, characterized in that: In S4, the inference method is selected based on the number of selections and the number of successes, and the current TV control command is generated as follows: If the selected reasoning method is the direct-pointing reasoning method and a candidate television control command has already been generated, then the candidate television control command will be used as the current television control command; if no candidate television control command has been generated, then the current television control command will not be generated. If the selected reasoning method is action recognition reasoning, the current gesture segment is input into the temporal convolutional network model, and the dynamic gesture category is output. The training data of the temporal convolutional network model includes the sequence of the center point of the hand bounding box, the sequence of smooth pointing points, the sequence of hand joints, and the label of the dynamic gesture category. If the TV control operation corresponding to the dynamic gesture category is an executable operation of the target screen element, then the TV control operation is used as the current TV control instruction; otherwise, no current TV control instruction is generated. After the TV executes the current TV control command, it updates the selection count and success count of the selected inference method. Specifically: each time the selected inference method is executed, the selection count of that inference method is incremented by 1; if the user does not perform the opposite control operation on the same target screen element in the next gesture segment, the success count of that inference method is incremented by 1; if the user performs the opposite control operation on the same target screen element in the next gesture segment, or if no current TV control command is generated this time, the success count of that inference method does not increase; opposite control operations include left and right movement, up and down movement, and entering and returning. After generating the current TV control command, the inference selection module sends the current TV control command to the control execution module; The control execution module controls the interface display, playback status, channel status, or volume status of the ultra-high-definition television according to the television control operation corresponding to the current television control command, so that the ultra-high-definition television performs the corresponding interactive operation.

Images