Gesture positioning method and device applied to intelligent remote controller and electronic equipment

Abstract

The invention provides a gesture positioning method and device applied to an intelligent remote controller and electronic equipment, and relates to the field of data processing. According to the method, in the process that a user uses the intelligent remote controller, position change data and posture change data of the intelligent remote controller are obtained; inputting the position change data and the attitude change data into a preset LSTM model to obtain a dynamic pointing area; performing gesture intention reasoning on the dynamic pointing area, and determining a pointing probability corresponding to the dynamic pointing area; if it is determined that the pointing probability meets a preset condition, obtaining a target object from the dynamic pointing region; and sending the control instruction to the target object to control the target object to complete the control operation of the intelligent remote controller through the control instruction. By implementing the technical scheme provided by the invention, the positioning problem of the intelligent remote controller is solved, so that the intelligent remote controller realizes a control effect.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a gesture positioning method, device and electronic device applied to a smart remote controller. Background Art

[0002] With the development of smart home technology, smart remote controls, as an important interactive tool between users and home appliances, are increasingly used.

[0003] At present, in the process of users using smart remote controls, related technologies usually rely on a single pointing positioning method, such as infrared, ultrasonic or single wireless radio frequency positioning, to determine the pointing intention of the remote control. However, the above single pointing positioning method relies on direct line of sight transmission between the remote control and the target device, which is affected by factors such as ambient lighting, object occlusion and reflection, resulting in a significant decrease in positioning accuracy. This unreliable positioning problem makes it easy for users to make mistakes or fail to achieve the expected control effect when using the smart remote control to operate the target object.

[0004] Therefore, there is an urgent need for a gesture positioning method, device and electronic device applied to a smart remote controller to solve the positioning problem of the smart remote controller so as to achieve a control effect. Summary of the invention

[0005] The present application provides a gesture positioning method, device and electronic device applied to a smart remote control, which solves the positioning problem of the smart remote control so that the smart remote control can achieve a control effect.

[0006] In a first aspect of the present application, a gesture positioning method for an intelligent remote controller is provided, the method comprising: obtaining position change data and posture change data of the intelligent remote controller during the user's use of the intelligent remote controller; inputting the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area; performing gesture intention inference on the dynamic pointing area to determine a pointing probability corresponding to the dynamic pointing area; if it is determined that the pointing probability meets a preset condition, obtaining a target object from the dynamic pointing area; and sending a control instruction to the target object to control the target object through the control instruction to complete the control operation of the intelligent remote controller.

[0007] By adopting the above technical solution, by simultaneously acquiring the position change data and posture change data of the remote control, the motion trajectory of the remote control in three-dimensional space can be more comprehensively understood. This combination can significantly improve the pointing accuracy and avoid misjudgment caused by relying only on a single parameter. The LSTM model can learn and memorize the dynamic characteristics of the position and posture changes of the remote control, which enables the system to adapt to the user's operating habits and environmental changes and achieve more flexible pointing recognition. The acquisition of the dynamic pointing area can achieve fast feedback, so that when the user uses the remote control, the device can respond to the user's pointing intention in real time. This real-time performance greatly enhances the user experience and reduces latency. By reasoning about the gesture intention of the dynamic pointing area, the system can effectively judge the user's true intention. The target object is identified only after the pointing probability meets the preset conditions, avoiding misoperation. This mechanism ensures that the remote control will only perform the control operation when the user's intention is clear, improving the safety and accuracy of intelligent control. The user only needs to point to the target object naturally, and the system will automatically complete the judgment of the pointing area and the sending of the control instruction, simplifying the operation process and improving the friendliness and convenience of the intelligent remote control. Therefore, the above solution solves the positioning problem of the intelligent remote control, so that the intelligent remote control can achieve the control effect.

[0008] Optionally, the acquiring of the position change data and the attitude change data of the intelligent remote control specifically includes: receiving the absolute position data of the intelligent remote control sent by the UWB module; generating three-dimensional coordinate data based on the absolute position data; receiving the acceleration data, angular velocity data and direction data of the intelligent remote control sent by the IMU module; generating multi-axis data based on the acceleration data, the angular velocity data and the direction data; performing data processing on the three-dimensional coordinate data and the multi-axis data to obtain the position change data and the attitude change data, and the data processing includes denoising, filtering, feature extraction and normalization.

[0009] By adopting the above technical solutions, the absolute position data provided by the UWB module has high accuracy and can effectively eliminate the errors caused by environmental interference of other positioning technologies, thereby ensuring that the position information of the remote control is accurate and reliable. By converting the absolute position data into three-dimensional coordinates, the system can fully understand the specific position of the remote control in three-dimensional space. This spatial positioning capability provides a solid foundation for subsequent dynamic pointing area recognition. The acceleration, angular velocity and direction data provided by the IMU module can fully capture the posture changes of the remote control. This multi-dimensional data helps to monitor the motion state of the remote control in real time and provides an important reference for intelligent control. By combining the acceleration, angular velocity and direction data to generate multi-axis data, the dynamic changes of the remote control in space can be more comprehensively reflected. This comprehensive information can enhance the system's ability to understand and judge the user's intentions. During the data transmission process, the signal may be interfered, resulting in inaccurate data. Denoising and filtering processing can effectively improve data quality, remove irrelevant interference, and ensure the accuracy of subsequent analysis. Through feature extraction, the system can identify key motion patterns and features, which helps to further understand the user's gestures and intentions. This process provides basic data support for subsequent gesture intention reasoning. Normalization can unify the scale of data from different sources, so that the influence of each data source can be reasonably balanced in subsequent data analysis, ensuring the effectiveness of model training and reasoning. By acquiring and processing position change and posture change data in real time, the system can quickly respond to the user's pointing behavior and improve the user interaction experience. Every action of the user can be quickly identified and processed to reduce latency.

[0010] Optionally, the position change data and the posture change data are input into a preset LSTM model to obtain a dynamic pointing area, specifically comprising: fusing the position change data and the posture change data to generate a trajectory vector, wherein the trajectory vector comprises a spatial position sequence, a posture quaternion sequence and a gesture dynamic feature sequence; predicting the spatial position sequence, the posture quaternion sequence and the gesture dynamic feature sequence through the preset LSTM model to generate a pointing area sequence; and generating the dynamic pointing area according to the pointing area sequence.

[0011] By adopting the above technical solution, the position change data, posture change data and gesture dynamic feature sequence are integrated to generate a trajectory vector, which can fully capture the user's operation behavior. This multi-dimensional data integration improves the system's ability to understand the user's intention. The spatial position sequence provides the position information of the remote control in three-dimensional space, helping the system understand the movement trajectory of the remote control. The posture quaternion sequence records the direction and tilt state of the remote control during use, and can accurately capture the changes in the user's gestures. The gesture dynamic feature sequence includes dynamic features such as the speed and acceleration of the gesture, which provides an important basis for identifying the type and intention of the gesture. The LSTM model can remember long-term dependencies. This enables the model to effectively capture the dynamic characteristics of the user's gestures and position changes and improve the accuracy of prediction. The LSTM model can learn the user's operation mode based on historical data, thereby improving the system's ability to predict future pointing behaviors. This learning ability enables the smart remote control to adapt to the individual differences of different users. By predicting and generating a pointing area sequence, the system can provide real-time feedback on the user's operation intention. This immediacy enhances the interactive experience between the user and the smart remote control. The generation of the pointing area is based on the user's current operation trajectory and posture, ensuring that the system can adapt to the user's changes in real time and improve the flexibility and accuracy of control. The user's gesture changes and the dynamic position changes of the remote control can be recognized in time, and the system can perform corresponding operations according to the user's actual pointing intention, reducing the possibility of misoperation. By generating dynamic pointing areas, users can get more intelligent feedback when using the smart remote control, improving overall user satisfaction.

[0012] Optionally, the gesture intention inference is performed on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area, specifically using the following calculation formula: ; in, is the pointing probability, is the first target area, is the pointing vector, For historical data, To control the importance of the directionality of the pointing vector in the likelihood function, To control the importance of the distance of the pointing vector in the likelihood function, is the angle between the pointing vector and the vector corresponding to the first target area, is the distance from the starting point of the pointing vector to the first target area, is the number of times the user points to the first target area in historical operations, is the angle between the pointing vector and the vector corresponding to the second target area, is the distance from the starting point of the pointing vector to the second target area, is the number of times the user points to the second target area in historical operations, the dynamic pointing area includes multiple target areas, and the first target area and the second target area are any two target areas among the multiple target areas.

[0013] By adopting the above technical solution, the user's actual pointing intention is quantified by calculating the angle between the pointing vector and the target area. The accuracy of the judgment is further refined by measuring the distance between the remote control and the target area. By introducing the user's historical preferences using probability, the user's intention can be better predicted, and the personalization and intelligence of the system can be improved. When the user's behavior and environment change, the probability distribution can be updated by new observation data. This flexibility enables the system to adapt to the user's changes in real time and maintain high accuracy. This method can effectively handle uncertainty. In practical applications, there may be uncertainty in the relationship between the pointing vector and the target area due to environmental noise and sensor errors. Through probabilistic calculation, the model is allowed to make the best decision under uncertain conditions, rather than relying on absolute deterministic judgments. By utilizing the user's historical behavior data, the formula provides a personalized gesture recognition experience for different users, and the model can calculate the corresponding probability based on real-time preferences, thereby improving its accuracy.

[0014] Optionally, if it is determined that the pointing probability meets a preset condition, the target object is obtained from the dynamic pointing area, specifically including: comparing the pointing probability with a preset threshold, and the preset threshold is obtained based on the historical operation statistics of the user; if it is determined that the pointing probability is greater than or equal to the preset threshold, target detection is performed on the dynamic pointing area to obtain the target object.

[0015] By adopting the above technical solution, by comparing the pointing probability with the preset threshold value obtained according to the user's historical operation statistics, the user's pointing intention can be judged according to the user's actual behavior habits. This personalized threshold setting enables the system to understand the user's operation more accurately and avoids the inaccuracy caused by the fixed threshold. The system will only perform target detection when the pointing probability is greater than or equal to the preset threshold. This mechanism effectively avoids misoperation caused by accidental or unclear pointing and improves the reliability of the system. By performing target detection on the dynamic pointing area, the system can quickly identify and locate the target object when the user's intention is clear. This not only improves the recognition speed, but also ensures the system's sensitivity to the user's intention. By analyzing the user's historical operation data, the system can customize personalized operation thresholds for different users. This personalized experience allows each user to feel more intimate and efficient service when using the smart remote control. By comparing the pointing probability with the preset threshold, the system can respond in a very short time and quickly perform target detection. This fast decision-making ability significantly improves the user's interactive experience and makes the operation smoother.

[0016] Optionally, the method further includes: inputting the position change data and the posture change data into the preset LSTM model to obtain a gesture pattern; performing gesture recognition on the gesture pattern to generate the operation instruction.

[0017] By adopting the above technical solution, the system can capture the user's gesture pattern in real time by inputting the position change and posture change data into the LSTM model. This instant response capability improves the interaction efficiency between the system and the user, making the operation smoother. The LSTM model can identify and learn the user's gesture change trend. This feature enables the system to effectively distinguish different gestures and maintain high accuracy even under complex or fast operations. The fusion of position change data and posture change data makes the gesture pattern richer. By integrating this information, the system can fully understand the user's intention and reduce misjudgment caused by insufficient information. By learning from the user's historical gesture data, the LSTM model can optimize the gesture recognition algorithm, improve recognition accuracy, and reduce the occurrence of misoperation. This data-driven learning process enables the system to adapt to the operating habits of different users. Once the gesture pattern is recognized, the system can quickly generate the corresponding operation instructions to ensure that the user's intention can be realized immediately. This fast response mechanism significantly improves the user experience and reduces waiting time.

[0018] Optionally, the dynamic pointing area includes a smart device that is communicatively connected to the smart remote controller, an operation position on the smart device, and a target point on the operation position.

[0019] By adopting the above technical solution, by clarifying the components of the dynamic pointing area, users can clearly understand the range that the smart remote control can control. This clear interactive definition helps to improve the intuitiveness of the user's operation and reduce confusion. Integrating smart devices that communicate with the remote control into the dynamic pointing area can simplify the user's management of different smart devices. Users only need to pay attention to the devices in the dynamic pointing area, which improves the convenience of operation. By marking the operation position and target point in the dynamic pointing area, users can operate more quickly and reduce the time to find the target. This efficient operation process enhances the user experience and makes device control smoother. The composition of the dynamic pointing area enables the system to perceive the status of the smart device in real time, and then provide feedback based on the user's pointing intention. This intelligent interaction improves the device's understanding of the user's intention and enhances the system's responsiveness.

[0020] In a second aspect of the present application, a gesture positioning device for an intelligent remote controller is provided, the gesture positioning device comprising an acquisition module and a processing module, wherein the acquisition module is used to acquire position change data and posture change data of the intelligent remote controller during the user's use of the intelligent remote controller; the processing module is used to input the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area; the processing module is also used to perform gesture intention inference on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area; the processing module is also used to acquire a target object from the dynamic pointing area if it is determined that the pointing probability meets a preset condition; the processing module is also used to send a control instruction to the target object to control the target object through the control instruction to complete the control operation of the intelligent remote controller.

[0021] In the third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By simultaneously acquiring the position change data and posture change data of the remote control, the motion trajectory of the remote control in three-dimensional space can be more comprehensively understood. This combination can significantly improve the pointing accuracy and avoid misjudgment caused by relying only on a single parameter. The LSTM model can learn and memorize the dynamic characteristics of the position and posture changes of the remote control, which enables the system to adapt to the user's operating habits and environmental changes and achieve more flexible pointing recognition. The acquisition of the dynamic pointing area can achieve fast feedback, so that when the user uses the remote control, the device can respond to the user's pointing intention in real time. This real-time performance greatly enhances the user experience and reduces latency. By reasoning about the gesture intention of the dynamic pointing area, the system can effectively judge the user's true intention. The target object is identified only after the pointing probability meets the preset conditions, avoiding misoperation. This mechanism ensures that the remote control will only perform control operations when the user's intention is clear, improving the safety and accuracy of intelligent control. The user only needs to naturally point to the target object, and the system will automatically complete the judgment of the pointing area and the sending of control instructions, simplifying the operation process and improving the friendliness and convenience of the intelligent remote control. Therefore, the above solution solves the problem of positioning the intelligent remote control, so that the intelligent remote control can achieve the control effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a gesture positioning method applied to a smart remote controller provided in an embodiment of the present application.

[0025] Figure 2 Another flowchart of a gesture positioning method applied to a smart remote controller provided in an embodiment of the present application.

[0026] Figure 3 A schematic diagram of a module of a gesture positioning device applied to a smart remote controller provided in an embodiment of the present application.

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] Explanation of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. DETAILED DESCRIPTION

[0029] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0030] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0031] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0032] With the continuous advancement and popularization of smart home technology, smart remote controls have become an indispensable and important interactive tool between users and home appliances, and their frequency of use in daily life has also increased significantly. However, current related technologies usually rely on a single pointing positioning method when users use smart remote controls. These methods may include relying on infrared, ultrasonic or a single wireless radio frequency signal for positioning to determine the pointing intention of the remote control.

[0033] However, there are some obvious limitations to the single pointing positioning method. First, these technologies usually require direct line of sight between the remote control and the target device, which means that once there is any object blocking the way, or due to factors such as ambient lighting and object reflection, the positioning accuracy will be significantly reduced. This lack of reliability often causes users to encounter misoperations when operating the smart remote control, or fail to effectively control the target device as expected.

[0034] In addition, with the increase of various obstacles and furniture in the home environment, the traditional single positioning method has caused great trouble to the user experience. When the user is operating, he may not be able to accurately find the device due to obstructed vision, resulting in the remote control's pointing intention not being effectively recognized. Such inconvenience not only reduces the efficiency of the smart home system, but also affects the overall user satisfaction.

[0035] In order to solve the above technical problems, the present application provides a gesture positioning method applied to a smart remote control. Figure 1 , Figure 1A flowchart of a gesture positioning method applied to a smart remote controller provided in an embodiment of the present application is provided. The gesture positioning method is applied to a server and includes steps S110 to S150. The steps are as follows: S110. When the user uses the smart remote controller, position change data and posture change data of the smart remote controller are obtained.

[0036] Specifically, in a smart home system, the server is responsible for processing and storing data from different devices. It can be a cloud server or a local server, designed to provide more powerful computing power and data management capabilities. The server is able to receive and analyze data from the smart remote control to better understand the user's operating intentions, thereby improving the system's responsiveness and accuracy. Position change data refers to the movement trajectory of the remote control in three-dimensional space. This includes the absolute position of the remote control, such as the position information obtained through GPS or UWB technology, and the change in relative position, such as the movement of the user holding the remote control during use. Obtaining position change data can help the system understand the physical relationship between the user and the target device, thereby improving the remote control's precise control of the target. Posture change data refers to information such as the tilt angle and rotation state of the remote control during use. This data can be obtained through the inertial measurement module, and the IMU can provide information such as acceleration, angular velocity and direction. Posture change data helps to identify the user's gesture intention. For example, a specific tilt or rotation action may correspond to a specific control instruction.

[0037] In a possible implementation, the position change data and attitude change data of the smart remote control are obtained, specifically including: receiving the absolute position data of the smart remote control sent by the UWB module; generating three-dimensional coordinate data based on the absolute position data; receiving the acceleration data, angular velocity data and direction data of the smart remote control sent by the IMU module; generating multi-axis data based on the acceleration data, angular velocity data and direction data; performing data processing on the three-dimensional coordinate data and the multi-axis data to obtain the position change data and the attitude change data, and the data processing includes denoising, filtering, feature extraction and normalization processing.

[0038] Specifically, UWB technology is a short-range, high-precision positioning technology that can provide more accurate location information in indoor environments. Absolute position data refers to the specific position coordinates of the remote control at a certain moment, for example, the coordinates (x, y, z) of the remote control in a three-dimensional space. This data can help the server determine the specific position of the remote control in the room. Based on the absolute position data received, the server converts it into three-dimensional coordinate data. This step may involve converting the original position information into a format suitable for subsequent processing to ensure that the position of the remote control can be accurately described in three-dimensional space. Inertial measurement units (IMUs) can provide data such as acceleration, angular velocity, and direction, and are usually used to track the motion state of an object. Acceleration data represents the acceleration change of the remote control in each axis (x-axis, y-axis, z-axis). Angular velocity data represents the speed at which the remote control rotates around each axis. Direction data refers to the orientation of the remote control relative to a reference direction, expressed in the form of quaternions or Euler angles. Combining acceleration, angular velocity, and direction data, the system will generate multi-axis data that describes the dynamic behavior of the remote control in space. This data can help the system understand the movement and rotation state of the remote control. Denoising can filter the noise of the received data to eliminate possible interference signals and ensure the accuracy of subsequent data. Filtering uses algorithms such as low-pass filters to smooth the data, reduce the impact of sudden changes or instantaneous fluctuations, and obtain a more stable signal. Feature extraction extracts useful features from the processed data, such as identifying specific gesture patterns or motion trajectories. Normalization ensures the comparability of different features by adjusting the data to the same standard range (such as between 0 and 1), which facilitates subsequent analysis. Ultimately, the data processed by the above steps will form position change data, such as the movement of the remote control in space, and posture change data, such as the tilt and rotation of the remote control, which will be used for subsequent gesture recognition and pointing intention analysis.

[0039] For example, suppose a user uses a smart remote control to control the TV at home. The specific process is as follows: the user places the remote control on the coffee table, and the UWB module sends absolute position data in real time, such as the coordinates (2,3,1.5), indicating the position of the remote control in three-dimensional space. The server converts the received absolute position data into a three-dimensional coordinate that can be used for calculation, perhaps converting it into a floating-point array for subsequent processing. When the user picks up the remote control and moves it to the left, the IMU module records acceleration data, such as 2m / s² to the left, angular velocity data, such as 0.5rad / s around the vertical axis, and direction data, such as a quaternion pointing to the north. The server combines the acceleration and angular velocity data to form multi-axis data that represents the overall dynamic state of the remote control. For example, the user may be shaking the remote control quickly, and the multi-axis data recorded by the system can reflect this movement.

[0040] S120, inputting the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area.

[0041] Specifically, LSTM is a recursive neural network that is particularly good at processing sequence data. It can remember long-term dependencies and is therefore very effective in processing time series data. In an embodiment of the present application, position change data and posture change data are input into the LSTM model so that the model can analyze these time series data to learn the user's gestures and pointing intentions. The dynamic pointing area refers to multiple possible control target areas predicted by the LSTM model based on input data during the user operation process. It represents the device or location that the user is currently paying attention to, and the server will perform subsequent operations based on this area. The LSTM model will determine the user's pointing intention and generate a corresponding dynamic pointing area by analyzing historical data patterns to facilitate the execution of subsequent operations.

[0042] In a possible implementation, the position change data and the posture change data are input into a preset LSTM model to obtain a dynamic pointing area, specifically including: fusing the position change data and the posture change data to generate a trajectory vector, the trajectory vector includes a spatial position sequence, a posture quaternion sequence and a gesture dynamic feature sequence; predicting the spatial position sequence, the posture quaternion sequence and the gesture dynamic feature sequence through a preset LSTM model to generate a pointing area sequence; generating a dynamic pointing area according to the pointing area sequence.

[0043] Specifically, the server first fuses the position change data from the UWB module and the attitude change data from the IMU module. This step combines the two types of data into a comprehensive trajectory vector. The trajectory vector contains the following three sequences: spatial position sequence: describes the position change of the remote control in the time series, such as (x1, y1, z1), (x2, y2, z2), etc.; attitude quaternion sequence: uses quaternions to represent the direction and rotation state of the remote control, providing information about how the remote control tilts in space. Quaternions avoid the gimbal lock problem that may occur in Euler angles and can represent rotation more smoothly; gesture dynamic feature sequence: describes the dynamic features of the user's gesture, such as speed, acceleration, etc., reflecting the user's operation intention. Secondly, the server inputs the generated trajectory vector, including the spatial position sequence, attitude quaternion sequence, and gesture dynamic feature sequence, into the preset LSTM model. The LSTM model is suitable for processing time series data and can capture the temporal features in the data. The LSTM model analyzes and predicts the input data to generate a pointing area sequence. These sequences reflect the user's pointing intention at a specific time point and may contain multiple candidate target areas. Based on the pointing area sequence output by the LSTM model, the server will identify the dynamic pointing area. This area represents the device or location that the user may focus on or try to control at a specific time. The dynamic pointing area can be a specific device, such as a TV, a lamp, or a specific part of a space. The server performs subsequent operations based on this information.

[0044] For example, suppose a user uses a smart remote control to control smart lights at home. The specific process is as follows: the user raises the remote control, and the UWB module provides position change data, for example: t1: position (2,3,1.5), t2: position (2,4,1.5), t3: position (2,5,1.5). The IMU module provides attitude change data, for example: t1: attitude quaternion (0.707,0,0.707,0), indicating an upward tilt of 45 degrees; t2: attitude quaternion (0.5,0.5,0.5,0.5), indicating a gradual turn to the right. After the trajectory vector is generated, the trajectory vector is input into the LSTM model. The model analyzes these sequence data, learns the temporal relationship therein, and finally generates a pointing area sequence. For example, the pointing area sequence is [(2,5,2.0), (2,5,1.8), (2,5,1.6)]. These outputs indicate that the user is approaching the target area, which may be a smart light located near the coordinate (2,5,2.0). Based on the pointing area sequence, the server finally determines the dynamic pointing area that the user is concerned about. For example, for a specific smart lamp, the server will determine that the user's intention is to adjust the brightness or switch the light. By fusing the position change data and posture change data and inputting them into the LSTM model, the server can effectively analyze the user's gestures and dynamic behaviors and generate an accurate dynamic pointing area. This process not only improves the accuracy of user operations, but also provides the necessary basis for further control of the smart remote control, thereby achieving a smoother smart home interaction experience.

[0045] S130: Perform gesture intention inference on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area.

[0046] Specifically, gesture intention reasoning means that the server analyzes the user's gesture pattern and behavior characteristics based on the data of the dynamic pointing area to infer the operation the user wants to perform or the target to be controlled. For example, whether the user wants to turn on a device, adjust the volume, or adjust the brightness of the light. The pointing probability is the server's quantitative assessment of the user's pointing intention, usually expressed as a number between 0 and 1, reflecting the possibility that the dynamic pointing area corresponds to the target. This probability can be calculated by a classifier or regression model. The model analyzes the user's historical behavior, real-time data, gesture characteristics and other information to predict the user's intention.

[0047] In a possible implementation, gesture intention inference is performed on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area, specifically using the following calculation formula: ; in, is the pointing probability, is the first target area, is the pointing vector, For historical data, To control the importance of the directionality of the pointing vector in the likelihood function, To control the importance of the distance of the pointing vector in the likelihood function, is the angle between the pointing vector and the vector corresponding to the first target area, is the distance from the starting point of the pointing vector to the first target area, is the number of times the user points to the first target area in historical operations, is the angle between the pointing vector and the vector corresponding to the second target area, is the distance from the starting point of the pointing vector to the second target area, is the number of times the user points to the second target area in historical operations, the dynamic pointing area includes multiple target areas, and the first target area and the second target area are any two target areas among the multiple target areas.

[0048] Specifically, the first target area and the second target area are any two target areas in the dynamic pointing area. The dynamic pointing area is an area inferred based on the user's gesture and position change, and may contain multiple target devices. The pointing vector is a vector dynamically generated by the user's gesture, which indicates the user's pointing intention and can be understood as pointing from the position of the remote control to the target area that the user wants to control. Historical data is the data of the user in past operations, including the number of times and behavior patterns of the user pointing to each target area in history. These data are used to evaluate the user's pointing habits. The importance coefficient α controls the importance coefficient of the directionality of the pointing vector in the likelihood function. It reflects the degree of match between the direction of the pointing vector and the target area vector. The better the direction match, the higher the pointing probability. β controls the importance coefficient of the distance of the pointing vector in the likelihood function. It indicates the influence of the distance from the starting point of the pointing vector to the target area on the pointing probability. The shorter the distance, the higher the pointing probability. The angle θi is the angle between the pointing vector and the vector corresponding to the first target area. The smaller the angle, the clearer the user's pointing intention. θj is the angle between the pointing vector and the vector corresponding to the second target area. di is the distance from the starting point of the pointing vector to the first target area. The closer the distance, the higher the pointing probability. dj is the distance from the starting point of the pointing vector to the second target area. The more times the user points to the first target area in historical operations, the higher the user's attention to the area.

[0049] Among them, the user's actual pointing intention is quantified by calculating the angle between the pointing vector and the target area. The accuracy of the judgment is further refined by measuring the distance between the remote control and the target area. The use of probability to introduce the user's historical preferences can better predict the user's intentions and improve the personalization and intelligence of the server. When the user's behavior and environment change, the probability distribution can be updated with new observation data. This flexibility enables the server to adapt to the changes of users in real time and maintain high accuracy. This method can effectively handle uncertainty. In practical applications, there may be uncertainty in the relationship between the pointing vector and the target area due to environmental noise and sensor errors. Through probabilistic calculation, the model is allowed to make the best decision under uncertain conditions instead of relying on absolute deterministic judgments. By utilizing the user's historical behavior data, the formula provides a personalized gesture recognition experience for different users, and the model can calculate the corresponding probability based on real-time preferences, thereby improving its accuracy.

[0050] S140: If it is determined that the pointing probability meets a preset condition, the target object is acquired from the dynamic pointing area.

[0051] Specifically, the pointing probability is a value obtained by the server through inference, indicating the possibility that the user may point to a specific target area. Meeting the preset conditions means that the pointing probability has reached the level recognized by the server, and it is considered that the user's intention is very clear. The dynamic pointing area may include multiple target objects, such as lamps, televisions, and speakers. Once it is determined that the pointing probability meets the preset conditions, the server needs to accurately identify the specific target object that the user wants to control in the area. Obtaining the target object may involve further analysis based on probability, historical data, or other auxiliary information.

[0052] In a possible implementation, if it is determined that the pointing probability meets a preset condition, the target object is obtained from the dynamic pointing area, specifically including: comparing the pointing probability with a preset threshold, the preset threshold is obtained based on the user's historical operation statistics; if it is determined that the pointing probability is greater than or equal to the preset threshold, target detection is performed on the dynamic pointing area to obtain the target object.

[0053] Specifically, the server first calculates a pointing probability, which represents the probability that the user points to a specific target area. Then, the server compares this probability with a preset threshold. This preset threshold is not fixed, but is dynamically adjusted according to the user's historical operation data. By analyzing the user's past operation behavior, such as the frequency of pointing to the target, the situation of misoperation, etc., a reasonable threshold can be set to adapt to individual differences. The server will record the user's selection habits for different target areas, the success rate of operations, and the number of misoperations in past operations. These data help the server understand the user's behavior pattern. For example, if a user is accustomed to waving the remote control in a large range to select a target, then his threshold may be set lower, while for those users with very precise gestures, the threshold may be higher. If the pointing probability is greater than or equal to the preset threshold, the server believes that the user's intention is relatively clear. Next, the server will perform target detection on all potential target objects in the dynamic pointing area. Target detection can be judged based on information such as position, direction, gesture, etc. to accurately identify the user's target object. The result of target detection is the target object that the server believes the user wants to control. For example, the server may identify the smart light that the user wants to turn on or the speaker that adjusts the volume.

[0054] Therefore, by setting a preset threshold based on the user's historical operations, the server can dynamically adjust the sensitivity to the user's pointing intention. The process of comparing the pointing probability with the threshold helps the server select the correct target object when the user's intention is clear. Specific target detection ensures that the server can accurately identify the user's control target within the dynamic pointing area, thereby reducing the possibility of misoperation. This method can enable the smart remote control to respond to user operations more intelligently in a multi-target environment, improving the user experience and control accuracy.

[0055] S150: Send a control instruction to the target object, so as to control the target object through the control instruction to complete the control operation of the intelligent remote controller.

[0056] Specifically, after determining the target object, such as smart lights, speakers, TVs, etc., the server needs to generate an appropriate control instruction, such as turning on, off, adjusting the volume, switching channels, etc. This control instruction is based on the user's operation intention, such as gesture recognition, button pressing, or voice command. Subsequently, the server sends the control instruction to the target object through a network, such as Wi-Fi, Zigbee, Bluetooth, etc. After the control instruction is delivered to the target object, the target object will parse the instruction content and perform the corresponding operation. Smart devices usually have the ability to parse server control instructions. The instructions sent by the server can be operations such as turning on, off, adjusting brightness or volume. Through this process, the smart remote control does not directly control the target device, but completes the control operation through the server. This architectural design makes the system more scalable and flexible, and the server can handle multi-device collaborative control more comprehensively. In this way, during the operation, the user does not need to point precisely or communicate directly with the device, but relies on the server's intelligent analysis and decision-making, which can make the operation of the smart remote control more intuitive and intelligent.

[0057] In one possible implementation, refer to Figure 2 , Figure 2 Another flow chart of a gesture positioning method applied to an intelligent remote control provided in an embodiment of the present application includes steps S210 to S220, and the above steps are as follows: S210, input the position change data and the posture change data into a preset LSTM model to obtain a gesture pattern; S220, perform gesture recognition on the gesture pattern to generate an operation instruction.

[0058] Specifically, the server obtains the position information and posture data of the smart remote control. The time series of these data reflects the changes in the user's gestures over a period of time. The server inputs these time series data into a trained LSTM model. The LSTM model uses the position change data and posture change data to identify the trajectory and posture changes of the remote control in the user's hand, and analyzes these changes to infer the user's gesture pattern. For example, the LSTM model may recognize a user's "push up" or "swing to the left" gesture. After obtaining the gesture pattern, the server compares the pattern with the preset gesture library to identify the user's specific gesture. The gesture library contains gesture types that have been defined and trained in advance, such as "turn on the TV", "adjust the volume", "turn off the lights", etc. By matching the gesture pattern with the gesture library, the server can determine the user's operation intention. According to the recognized gesture, the server will generate a corresponding operation instruction. The operation instruction is a control command for a specific target device, such as "turn on the TV", "turn up the volume", etc. The instruction can contain the identification of the target device and the specific operation content.

[0059] In a possible implementation, the dynamic pointing area includes a smart device that is communicatively connected to the smart remote controller, an operation position on the smart device, and a target point on the operation position.

[0060] Specifically, the dynamic pointing area includes the target smart devices that the user wants to control, such as smart lights, TVs, air conditioners, and speakers. These devices can establish a communication connection with the smart remote control and transmit data through wireless protocols. The communication connection between the smart remote control and the smart device is the basis for the server to accurately determine the user's pointing. Each smart device has a different "operation position". These operation positions refer to the points or areas where users can perform specific operations on the device. For example, for a smart TV, the operation position can be a volume adjustment button, a channel switching area, a switch button, etc.; for a smart light, the operation position may be an area for adjusting the brightness, a color temperature adjustment area, etc. The target point is the specific location pointed by the user, and the server can lock these points by combining the user's posture change data and position change data. For example, for a smart light, the target point may be the brightness level that the user wants to adjust; for a smart speaker, the target point may be a specific volume on the volume adjustment area.

[0061] The present application also provides a gesture positioning device for an intelligent remote controller, referring to Figure 3 , Figure 3 A schematic diagram of a module of a gesture positioning device applied to an intelligent remote controller provided in an embodiment of the present application. The gesture positioning device includes an acquisition module 31 and a processing module 32, wherein the acquisition module 31 acquires position change data and posture change data of the intelligent remote controller during the user's use of the intelligent remote controller; the processing module 32 inputs the position change data and posture change data into a preset LSTM model to obtain a dynamic pointing area; the processing module 32 performs gesture intention inference on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area; if the processing module 32 determines that the pointing probability meets the preset conditions, the target object is obtained from the dynamic pointing area; the processing module 32 sends a control instruction to the target object to control the target object through the control instruction to complete the control operation of the intelligent remote controller.

[0062] In a possible implementation, the acquisition module 31 acquires the position change data and attitude change data of the intelligent remote control, specifically including: the acquisition module 31 receives the absolute position data of the intelligent remote control sent by the UWB module; the processing module 32 generates three-dimensional coordinate data based on the absolute position data; the acquisition module 31 receives the acceleration data, angular velocity data and direction data of the intelligent remote control sent by the IMU module; the processing module 32 generates multi-axis data based on the acceleration data, angular velocity data and direction data; the processing module 32 processes the three-dimensional coordinate data and multi-axis data to obtain position change data and attitude change data, and the data processing includes denoising, filtering, feature extraction and normalization.

[0063] In a possible implementation, the processing module 32 inputs the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area, specifically including: the processing module 32 fuses the position change data and the posture change data to generate a trajectory vector, the trajectory vector includes a spatial position sequence, a posture quaternion sequence and a gesture dynamic feature sequence; the processing module 32 predicts the spatial position sequence, the posture quaternion sequence and the gesture dynamic feature sequence through a preset LSTM model to generate a pointing area sequence; the processing module 32 generates a dynamic pointing area according to the pointing area sequence.

[0064] In a possible implementation, the processing module 32 performs gesture intention inference on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area, specifically using the following calculation formula: ; in, is the pointing probability, is the first target area, is the pointing vector, For historical data, To control the importance of the directionality of the pointing vector in the likelihood function, To control the importance of the distance of the pointing vector in the likelihood function, is the angle between the pointing vector and the vector corresponding to the first target area, is the distance from the starting point of the pointing vector to the first target area, is the number of times the user points to the first target area in historical operations, is the angle between the pointing vector and the vector corresponding to the second target area, is the distance from the starting point of the pointing vector to the second target area, is the number of times the user points to the second target area in historical operations, the dynamic pointing area includes multiple target areas, and the first target area and the second target area are any two target areas among the multiple target areas.

[0065] In a possible implementation, if the processing module 32 determines that the pointing probability meets a preset condition, the target object is obtained from the dynamic pointing area, specifically including: the processing module 32 compares the pointing probability with a preset threshold, and the preset threshold is obtained based on the user's historical operation statistics; if the processing module 32 determines that the pointing probability is greater than or equal to the preset threshold, target detection is performed on the dynamic pointing area to obtain the target object.

[0066] In a possible implementation, the processing module 32 inputs the position change data and the posture change data into a preset LSTM model to obtain a gesture pattern; the processing module 32 performs gesture recognition on the gesture pattern to generate an operation instruction.

[0067] In a possible implementation, the dynamic pointing area includes a smart device that is communicatively connected to the smart remote controller, an operation position on the smart device, and a target point on the operation position.

[0068] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0069] The present application also provides an electronic device, referring to Figure 4 , Figure 4 The electronic device may include: at least one processor 41 , at least one network interface 44 , a user interface 43 , a memory 45 , and at least one communication bus 42 .

[0070] The communication bus 42 is used to realize the connection and communication between these components.

[0071] The user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may also include a standard wired interface and a wireless interface.

[0072] The network interface 44 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0073] Among them, the processor 41 may include one or more processing cores. The processor 41 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 45, and calling data stored in the memory 45. Optionally, the processor 41 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 41 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 41, and it can be implemented separately through a chip.

[0074] Among them, the memory 45 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 45 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 45 may also be optionally at least one storage device located away from the aforementioned processor 41. As Figure 4 As shown, the memory 45 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a gesture positioning method applied to an intelligent remote controller.

[0075] exist Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 41 can be used to call an application program stored in the memory 45 for a gesture positioning method applied to a smart remote control. When executed by one or more processors, the electronic device executes one or more methods in the above-mentioned embodiments.

[0076] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0077] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0078] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0083] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, it will be easy for those skilled in the art to think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A gesture positioning method applied to an intelligent remote controller, characterized in that: The method comprises: When the user uses the intelligent remote controller, the position change data and the posture change data of the intelligent remote controller are obtained; Inputting the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area; Performing gesture intention inference on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area; If it is determined that the pointing probability meets a preset condition, obtaining a target object from the dynamic pointing area; A control instruction is sent to the target object, so as to control the target object through the control instruction to complete the control operation of the intelligent remote controller.

2. The gesture positioning method for intelligent remote controller according to claim 1, characterized in that: The obtaining of the position change data and the posture change data of the intelligent remote controller specifically includes: Receiving the absolute position data of the intelligent remote controller sent by the UWB module; Generate three-dimensional coordinate data according to the absolute position data; Receive acceleration data, angular velocity data and direction data of the intelligent remote controller sent by the IMU module; Generate multi-axis data according to the acceleration data, the angular velocity data and the direction data; The three-dimensional coordinate data and the multi-axis data are processed to obtain the position change data and the posture change data, wherein the data processing includes denoising, filtering, feature extraction and normalization processing.

3. The gesture positioning method for intelligent remote controller according to claim 1, characterized in that: The step of inputting the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area specifically includes: The position change data and the posture change data are merged to generate a trajectory vector, wherein the trajectory vector includes a spatial position sequence, a posture quaternion sequence, and a gesture dynamic feature sequence; The spatial position sequence, the posture quaternion sequence, and the gesture dynamic feature sequence are predicted by the preset LSTM model to generate a pointing area sequence; The dynamic pointing area is generated according to the pointing area sequence.

4. The gesture positioning method for an intelligent remote controller according to claim 1, characterized in that: The gesture intention inference is performed on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area, and specifically adopts the following calculation formula: ； in, is the pointing probability, is the first target area, is the pointing vector, For historical data, To control the importance of the directionality of the pointing vector in the likelihood function, To control the importance of the distance of the pointing vector in the likelihood function, is the angle between the pointing vector and the vector corresponding to the first target area, is the distance from the starting point of the pointing vector to the first target area, is the number of times the user points to the first target area in historical operations, is the angle between the pointing vector and the vector corresponding to the second target area, is the distance from the starting point of the pointing vector to the second target area, is the number of times the user points to the second target area in historical operations, the dynamic pointing area includes multiple target areas, and the first target area and the second target area are any two target areas among the multiple target areas.

5. The gesture positioning method for intelligent remote controller according to claim 1, characterized in that: If it is determined that the pointing probability meets a preset condition, obtaining the target object from the dynamic pointing area specifically includes: Comparing the pointing probability with a preset threshold, where the preset threshold is obtained based on historical operation statistics of the user; If it is determined that the pointing probability is greater than or equal to a preset threshold, target detection is performed on the dynamic pointing area to obtain the target object.

6. The gesture positioning method for an intelligent remote controller according to claim 1, characterized in that: The method further comprises: Inputting the position change data and the posture change data into the preset LSTM model to obtain a gesture pattern; Perform gesture recognition on the gesture pattern to generate the operation instruction.

7. The gesture positioning method for an intelligent remote controller according to claim 1, characterized in that: The dynamic pointing area includes a smart device that is communicatively connected to the smart remote controller, an operation position on the smart device, and a target point on the operation position.

8. A gesture positioning device applied to an intelligent remote controller, characterized in that: The gesture positioning device comprises an acquisition module (31) and a processing module (32), wherein: The acquisition module (31) is used to acquire the position change data and posture change data of the intelligent remote controller when the user uses the intelligent remote controller; The processing module (32) is used to input the position change data and the posture change data into a preset LSTM model to obtain a dynamic pointing area; The processing module (32) is further used to perform gesture intention inference on the dynamic pointing area to determine the pointing probability corresponding to the dynamic pointing area; The processing module (32) is further configured to obtain a target object from the dynamic pointing area if it is determined that the pointing probability meets a preset condition; The processing module (32) is further used to send a control instruction to the target object, so as to control the target object through the control instruction to complete the control operation of the intelligent remote controller.

9. An electronic device, characterized in that: The electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), wherein the memory (45) is used to store instructions, the user interface (43) and the network interface (44) are both used to communicate with other devices, and the processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images