Remote control method and electronic device

Abstract

The application provides a remote control method and an electronic device, and relates to the technical field of terminals. The application can determine a controlled device pointed by a control device by controlling a shooting picture of the control device, implement intelligent control of the control device on the controlled device, and thus meet user interaction requirements while reducing hardware cost. The method comprises the following steps: the control device acquires a shooting picture through a camera, and the shooting picture comprises a picture obtained by shooting a region where the controlled device is located. The device analyzes features in the shooting picture to determine that an electronic device required to execute a user interaction event is the controlled device. In response to the user interaction event, an operation indicated by the user interaction event is executed on the controlled device. The user interaction event comprises an interaction event of pointing the controlled device by using the control device.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a remote control method and electronic device. Background Technology

[0002] With the development of terminal technology, remote control devices are becoming increasingly feature-rich. For example, based on pointing positioning technology, smart screens can display a cursor corresponding to the direction the remote control is pointing. Thus, during remote control operations, the smart screen can move the cursor position according to the direction the user is pointing the remote control, helping the user understand the current operating position and enabling a richer remote interactive experience.

[0003] However, in the above scenario, both the remote control device and the smart screen need to be equipped with wireless positioning modules. This is necessary to calculate the incident angle of the signal based on information such as the time difference between the signal transmission and reception by the wireless positioning module. Only then, based on this incident angle, can the direction of the remote control device relative to the smart screen be determined, achieving pointing positioning.

[0004] It can be seen that the current pointing and positioning technology relies on hardware configuration, which increases the hardware cost of the device. Furthermore, the pre-installation process requires the installation and adjustment of the supporting hardware, resulting in poor overall flexibility. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a remote control method and an electronic device. The technical solution provided by this application can determine the controlled device by the image captured by the control device, thereby achieving intelligent control of the controlled device by the control device, thus reducing hardware costs while meeting user interaction needs.

[0006] To achieve the above-mentioned technical objectives, this application provides the following technical solution:

[0007] Firstly, a remote control method is provided. The method includes: a control device acquiring a captured image via a camera, the captured image including an image captured towards the area where the controlled device is located; a parsing device determining, based on features in the captured image, that an electronic device requiring a user interaction event is the controlled device; and, in response to a user interaction event, performing an operation indicated by the user interaction event on the controlled device; wherein the user interaction event includes an interaction event where the control device points to the controlled device.

[0008] In this way, the control device can use the images captured by the camera to control the direction of the controlled device. Compared to configuring separate wireless positioning modules in the control device and the controlled device, this method, which relies on image processing of the captured images to determine the controlled device, can effectively reduce the hardware cost of implementing direction control.

[0009] Furthermore, it eliminates the need for complete hardware installation and adjustment of the control and controlled equipment, increasing the flexibility of directional control.

[0010] According to the first aspect, the operation of executing the user interaction event indication on the controlled device includes: displaying a first element on the controlled device at a display position corresponding to the pointing position in response to the pointing position of the control device. And / or, the user interaction event further includes: a selection operation on a control component on the control device; the operation of executing the user interaction event indication on the controlled device includes: triggering the controlled device to execute an operation in response to the selection operation.

[0011] Optionally, control components on the control device may include controls displayed on the control device and / or physical buttons on the control device.

[0012] In this way, by identifying the controlled device through the captured images, the control device can achieve more flexible control over the controlled device.

[0013] According to the first aspect, or any implementation of the first aspect above, the camera includes a vision sensor, and the captured image includes identifying data of the controlled device captured using the vision sensor and / or data generated by capturing the display image of the controlled device.

[0014] According to the first aspect, or any implementation of the first aspect above, the identifying data includes the outline data of the controlled device, and / or data used to locate the position of the controlled device in the captured image.

[0015] For example, the captured image obtained by the control device includes the image of the controlled device and the image outside the controlled device's image. Changes in the direction the control device points towards the area where the controlled device is located will cause changes in the positional relationship between the controlled device's image and the captured image. Therefore, by confirming the positional relationship between the controlled device's image and the captured image, the pointing position of the control device can be obtained. Optionally, the controlled device's image included in the captured image can be identified using the controlled device's identifying data.

[0016] Optionally, the identifying data may include data generated by an external flashing light source configured to capture images of the controlled device, data generated by capturing the flashing feature image displayed by the controlled device, or data generated by capturing identifying objects on the peripherals of the controlled device. For example, contour data may be generated by capturing images of an external flashing light source configured to capture images of the controlled device, which is positioned at the outer edge of the controlled device, thereby identifying the contour of the controlled device. Another example is data used to locate the position of the controlled device in the captured image, such as data generated by capturing the flashing feature image displayed by the controlled device.

[0017] In this way, by using the identifying data of the controlled device or the displayed screen, the corresponding data of the controlled device can be identified in the image captured towards the controlled device, and then the pointing position of the control device can be determined.

[0018] According to the first aspect, or any implementation of the first aspect above, after the parsing device determines that the electronic device that needs to perform the user interaction event is the controlled device through the features in the captured image, the method further includes: the parsing device obtains the display position on the controlled device corresponding to the pointing position of the control device based on the positional relationship between the identification data and the captured image, or based on the data generated by capturing the display image of the controlled device.

[0019] For example, after the analysis device confirms the controlled device captured by the control device, it can determine the positional relationship between the controlled device image and the captured image based on the identifying data in the captured image. Then, based on this positional relationship, it can perform coordinate transformation on the center position of the captured image to determine the pointing position of the control device.

[0020] For example, if the controlled device has a display function, the analysis device can also determine the positional relationship between the controlled device's image and the captured image based on the data generated by the control device capturing the image of the controlled device's display. Based on this positional relationship, the center position of the captured image can be transformed to determine the pointing position of the control device.

[0021] In this way, after identifying the controlled device, the parsing device can obtain the display position on the controlled device corresponding to the pointing position of the control device, which makes it easier to trigger the controlled device to display the corresponding element at that display position.

[0022] According to the first aspect, or any implementation of the first aspect above, after the parsing device determines that the electronic device that needs to perform the user interaction event is the controlled device through the features in the captured image, the method further includes: the control device establishing a communication connection with the controlled device based on the determined controlled device, so as to perform the user interaction event.

[0023] Thus, by identifying the controlled device through image capture, the remote control device automatically establishes a wireless communication connection with the controlled device, enabling remote control of the controlled device. Compared to solutions where the user needs to pair and connect the control device with the controlled device before controlling the controlled device, the remote control method provided in this application effectively simplifies user operation and improves the user experience.

[0024] According to the first aspect, or any implementation of the first aspect above, the camera includes an event-based visual sensor (EVS) camera; the control device acquires a captured image through the camera, including: generating an EVS event through the EVS camera, wherein the EVS event is generated based on a light source configured by the controlled device and / or a feature image displayed by the controlled device, and the EVS event is used to generate the captured image.

[0025] According to the first aspect, or any implementation of the first aspect above, the light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

[0026] In this way, the EVS camera allows the control device to acquire the captured image, which facilitates the subsequent determination of the pointing position based on the captured image.

[0027] According to the first aspect, or any implementation of the first aspect above, the method further includes: the controlled device displays the feature image at a preset position on the display screen, or displays the feature image in full screen.

[0028] According to the first aspect, or any implementation of the first aspect above, the method further includes: the control device performing filtering processing and / or lens parameter correction on the EVS event to generate a captured image. The parsing device, based on the captured image, obtains the image captured in the captured image that is directed towards the area where the controlled device is located.

[0029] In this way, by processing the EVS event, we can output images of acceptable quality, thereby ensuring that we can obtain the correct pointing position of the control device later.

[0030] According to the first aspect, or any implementation of the first aspect above, the camera includes an active pixel sensor (APS) camera.

[0031] Optionally, the images captured by the control device through the APS camera may include data generated from capturing the display screen of the controlled device.

[0032] According to the first aspect, or any implementation thereof, the parsing device determines the electronic device that needs to perform a user interaction event as the controlled device by using features in the captured image, including: the parsing device comparing feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image. Based on the first position, the parsing device obtains the positional relationship between the image of the area where the controlled device is located and the captured image.

[0033] In this way, by comparing feature points, the amount of data processed can be reduced while processing the data generated by the captured and displayed images, thereby improving efficiency.

[0034] According to the first aspect, or any implementation of the first aspect above, the camera includes EVS camera function and APS camera function, and the control device acquires the shooting image through the camera, including: the control device acquires the shooting image through the cooperation of the EVS camera function and APS camera function of the camera.

[0035] Thus, when the camera of the control device includes multiple camera functions, the control device can flexibly use multiple camera functions in combination to acquire shooting images.

[0036] According to the first aspect, or any implementation of the first aspect above, the captured image is a first captured image obtained through the APS camera function of the camera, or a second captured image obtained through the EVS camera function of the camera. The control device acquires the first captured image through the APS camera function of the camera at preset intervals, and the first captured image is used to determine a first pointing position of the control device. The controlled device displays a first element at the display position corresponding to the first pointing position. During the intervals of the preset period, the control device acquires the second captured image through the EVS camera function of the camera. The control device obtains the relative displacement of the change in the first pointing position based on the second captured image. The control device obtains the second pointing position based on the relative displacement. The controlled device displays the first element at the display position corresponding to the second pointing position.

[0037] In this way, by using the APS camera function and the EVS camera function to acquire shooting images in a preset cycle, the power consumption of using the APS camera function can be reduced, and the shooting images can be effectively output to determine the pointing position of the control device.

[0038] According to the first aspect, or any implementation of the first aspect above, the method further includes: during the process of performing the operation indicated by the user interaction event on the controlled device, the control device displays a control interface, which is used to receive the user's operation to control the controlled device.

[0039] In this way, the control device can receive user interaction operations on the controlled device through the control interface, such as selecting a control, thereby providing users with a more flexible and richer remote control experience for the controlled device.

[0040] According to the first aspect, or any implementation of the first aspect above, the method further includes: the control device sequentially acquiring a first captured image and a second captured image, wherein the first captured image includes an image captured towards the area where the first controlled device is located, and the second captured image includes an image captured towards the area where the second controlled device is located. The control device controls the first controlled device and the second controlled device to work together.

[0041] In this way, the control device can recognize the user's continuous pointing operations to realize the functional linkage between multiple controlled devices, enriching the user's remote control interaction experience.

[0042] According to the first aspect, or any implementation of the first aspect above, the parsing device is a control device, or the parsing device is a controlled device, or the parsing device is a device other than a control device and a controlled device.

[0043] For example, a parsing device can be a control device. Similarly, a remote control device can be both a control device and a parsing device.

[0044] For example, the parsing device can be the controlled device. Similarly, a remote control device can be the controlling device, while a smart screen can be both a controlled device and a parsing device.

[0045] For example, the parsing device can be any device other than the control device and the controlled device. For instance, a remote control device can be the control device, a smart screen can be the parsing device, and an air conditioner can be the controlled device.

[0046] In this way, each electronic device can perform corresponding operations as different roles based on factors such as device capabilities and interaction scenarios, thereby providing users with a remote pointing and control experience.

[0047] Secondly, a remote control method is provided. This method is applied to a control device and includes: acquiring a captured image via a camera, the captured image including an image captured towards the area where the controlled device is located; determining, based on features in the captured image, that an electronic device requiring a user interaction event is the controlled device; and, in response to a user interaction event, instructing the controlled device to perform the operation indicated by the user interaction event; wherein the user interaction event includes an interaction event where the control device points at the controlled device.

[0048] In this way, the control device can not only acquire the captured image through the camera, but also process the captured image as a parsing device to identify the controlled device and realize the pointing control of the controlled device.

[0049] Furthermore, by using images captured by a camera to control the direction of the controlled device, the hardware cost of implementing direction control can be effectively reduced.

[0050] According to the second aspect, the operation of instructing the controlled device to perform the user interaction event instruction includes: in response to the pointing position of the control device, instructing the controlled device to display a first element at a display position on the controlled device corresponding to the pointing position. And / or, the user interaction event further includes: a selection operation on a control component on the control device; the operation of instructing the controlled device to perform the user interaction event instruction includes: in response to the selection operation, triggering the controlled device to perform the operation.

[0051] According to the second aspect, or any implementation of the second aspect above, the camera includes a vision sensor, and the captured image includes identifying data of the controlled device captured using the vision sensor and / or data generated by capturing the display image of the controlled device.

[0052] According to the second aspect, or any implementation of the second aspect above, the identifying data includes the outline data of the controlled device, and / or data used to locate the position of the controlled device in the captured image.

[0053] According to the second aspect, or any implementation of the second aspect above, after determining that the electronic device that needs to perform the user interaction event is the controlled device by the features in the captured image, the method further includes: obtaining the display position on the controlled device corresponding to the pointing position of the control device based on the positional relationship between the identification data and the captured image, or based on the data generated by capturing the display image of the controlled device.

[0054] According to the second aspect, or any implementation of the second aspect above, after determining the electronic device that needs to perform the user interaction event as the controlled device by the features in the captured image, the method further includes: establishing a communication connection with the controlled device based on the determined controlled device, so as to perform the user interaction event.

[0055] According to the second aspect, or any implementation of the second aspect above, the camera includes an event-based visual sensor (EVS) camera; acquiring a captured image through the camera includes: generating an EVS event through the EVS camera, wherein the EVS event is generated based on a light source configured for the controlled device and / or a feature image displayed by the controlled device, and the EVS event is used to generate the captured image.

[0056] According to the second aspect, or any implementation of the second aspect above, the method further includes: filtering the EVS event and / or correcting lens parameters to generate a captured image. Based on the captured image, acquiring the image captured in the captured image that is directed towards the area where the controlled device is located.

[0057] According to the second aspect, or any implementation of the second aspect above, the light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

[0058] According to the second aspect, or any implementation of the second aspect above, the camera includes an active pixel sensor (APS) camera.

[0059] According to the second aspect, or any implementation thereof, determining the electronic device that needs to perform the user interaction event as the controlled device by using features in the captured image includes: comparing feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image; and based on the first position, obtaining the positional relationship between the image of the area where the controlled device is located and the captured image.

[0060] According to the second aspect, or any implementation of the second aspect above, the camera includes EVS camera function and APS camera function, and the camera acquires the shooting image, including: acquiring the shooting image by combining the EVS camera function and APS camera function of the camera.

[0061] According to the second aspect, or any implementation of the second aspect above, the captured image is a first captured image obtained through the APS camera function of the camera, or a second captured image obtained through the EVS camera function of the camera. The first captured image is acquired through the APS camera function of the camera at a preset period, and the first captured image is used to determine a first pointing position of the control device. The controlled device is instructed to display a first element at the display position corresponding to the first pointing position. During the interval of the preset period, the second captured image is acquired through the EVS camera function of the camera. Based on the second captured image, the relative displacement of the change in the first pointing position is obtained. Based on the relative displacement, the second pointing position is obtained. The controlled device is instructed to display the first element at the display position corresponding to the second pointing position.

[0062] According to the second aspect, or any implementation of the second aspect above, the method further includes: displaying a control interface, the control interface being used to receive user operations to control the controlled device.

[0063] According to the second aspect, or any implementation of the second aspect above, the method further includes: sequentially acquiring a first captured image and a second captured image, wherein the first captured image includes an image captured towards the area where the first controlled device is located, and the second captured image includes an image captured towards the area where the second controlled device is located. The first controlled device and the second controlled device are then controlled to work together.

[0064] Thirdly, this application provides a remote control method applied to a parsing device. The method includes: receiving a captured image from a camera sent by a control device, the captured image including an image captured towards the area where the controlled device is located; determining, based on features in the captured image, that an electronic device requiring a user interaction event is the controlled device; and, assuming the controlled device is a parsing device, performing an operation indicated by the user interaction event in response to the user interaction event, wherein the user interaction event includes an interaction event where the control device points to the controlled device; or, assuming the controlled device is a device other than a parsing device, sending information to the control device to instruct the controlled device.

[0065] Thus, if the controlled device is a parsing device, it can also identify the controlled device based on the received captured image, enabling the control device to control the direction of the controlled device. Alternatively, if the controlled device is a device other than a parsing device, the parsing device can also identify the controlled device based on the captured or received captured image, enabling the control device to control the direction of the controlled device.

[0066] Furthermore, by using images captured by a camera to control the direction of the controlled device, the hardware cost of implementing direction control can be effectively reduced.

[0067] According to the third aspect, the operation of executing the user interaction event indication includes: in response to the pointing position of the control device, displaying a first element at the display position corresponding to the pointing position on the controlled device. And / or, the user interaction event further includes: a selection operation on a control component on the control device; the operation of executing the user interaction event indication includes: performing the operation corresponding to the selection operation according to the selection operation.

[0068] According to the third aspect, or any implementation of the third aspect above, the camera includes a vision sensor, and the captured image includes identifying data of the controlled device captured using the vision sensor and / or data generated by capturing the display image of the controlled device.

[0069] According to the third aspect, or any implementation of the third aspect above, the identifying data includes the outline data of the controlled device, and / or data used to locate the position of the controlled device in the captured image.

[0070] According to the third aspect, or any implementation of the third aspect above, after determining that the electronic device that needs to perform the user interaction event is the controlled device by the features in the captured image, the method further includes: obtaining the display position on the controlled device corresponding to the pointing position of the control device based on the positional relationship between the identification data and the captured image, or based on the data generated by capturing the display image of the controlled device.

[0071] According to the third aspect, or any implementation of the third aspect above, the information sent to the control device to instruct the controlled device is used by the control device to establish a communication connection with the controlled device based on the determined controlled device, so as to execute user interaction events.

[0072] According to the third aspect, or any implementation of the third aspect above, the camera includes an event-based visual sensor (EVS) camera; receiving the captured image obtained by the control device through the camera sent by the control device includes: receiving the EVS event generated by the control device through the EVS camera sent by the control device, wherein the EVS event is generated based on the light source configured by the controlled device and / or the feature image displayed by the controlled device, and the EVS event is used to generate the captured image.

[0073] According to the third aspect, or any implementation of the third aspect above, the light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

[0074] According to the third aspect, or any of the above implementations of the third aspect, the method further includes: displaying the feature image at a preset position on the display screen, or displaying the feature image in full screen, depending on the controlled device being a parsing device.

[0075] According to the third aspect, or any implementation of the third aspect above, the method further includes: filtering the EVS event and / or correcting lens parameters to generate a captured image. Based on the captured image, acquiring the image captured in the captured image that is directed towards the area where the controlled device is located.

[0076] According to the third aspect, or any implementation of the third aspect above, the camera includes an active pixel sensor (APS) camera.

[0077] According to the third aspect, or any implementation thereof, determining the electronic device that needs to perform the user interaction event as the controlled device by using features in the captured image includes: comparing feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image; and based on the first position, obtaining the positional relationship between the image of the area where the controlled device is located and the captured image.

[0078] According to the third aspect, or any of the above implementations of the third aspect, the camera includes EVS camera function and APS camera function, which are used together to acquire the captured image.

[0079] Fourthly, an electronic device is provided, which is a control device. The control device includes a processor, a memory, and a camera, the memory and camera being coupled to the processor. The memory stores computer program code, including computer instructions. When the processor reads the computer instructions from the memory, the control device executes: acquiring a captured image through the camera, the captured image including an image captured towards the area where the controlled device is located; determining, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device; and, in response to a user interaction event, instructing the controlled device to perform the operation indicated by the user interaction event; wherein the user interaction event includes an interaction event where the control device points to the controlled device.

[0080] According to the fourth aspect, the operation of instructing the controlled device to perform the user interaction event instruction includes: in response to the pointing position of the control device, instructing the controlled device to display a first element at the display position corresponding to the pointing position on the controlled device. And / or, the user interaction event further includes: a selection operation on a control component on the control device; the operation of instructing the controlled device to perform the user interaction event instruction includes: in response to the selection operation, triggering the controlled device to perform the operation.

[0081] According to the fourth aspect, or any implementation of the fourth aspect above, the camera includes a vision sensor, and the captured image includes identifying data of the controlled device captured using the vision sensor and / or data generated by capturing the display image of the controlled device.

[0082] According to the fourth aspect, or any implementation of the fourth aspect above, the identifying data includes the outline data of the controlled device, and / or data used to locate the position of the controlled device in the captured image.

[0083] According to the fourth aspect, or any implementation of the fourth aspect above, when the processor reads computer instructions from memory, it also causes the control device to perform: based on the positional relationship between the identification data and the captured image, or based on the data generated from capturing the display image of the controlled device, to obtain the display position on the controlled device corresponding to the pointing position of the control device.

[0084] According to the fourth aspect, or any implementation of the fourth aspect above, when the processor reads computer instructions from memory, it also causes the control device to perform: establishing a communication connection with the controlled device based on the determined controlled device, for the purpose of executing user interaction events.

[0085] According to the fourth aspect, or any implementation of the fourth aspect above, the camera includes an event-based visual sensor (EVS) camera; acquiring a captured image through the camera includes: generating an EVS event through the EVS camera, wherein the EVS event is generated based on a light source configured for the controlled device and / or a feature image displayed by the controlled device, and the EVS event is used to generate the captured image.

[0086] According to the fourth aspect, or any implementation of the fourth aspect above, when the processor reads computer instructions from memory, it also causes the control device to perform: filtering and / or lens parameter correction on the EVS event, generating a captured image. Based on the captured image, it acquires the image captured in the captured image that is directed towards the area where the controlled device is located.

[0087] According to the fourth aspect, or any of the above implementations of the fourth aspect, the light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

[0088] According to the fourth aspect, or any implementation of the fourth aspect above, the camera includes an active pixel sensor (APS) camera.

[0089] According to the fourth aspect, or any implementation of the fourth aspect above, determining the electronic device that needs to perform the user interaction event as the controlled device by using features in the captured image includes: comparing feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image. Based on the first position, obtaining the positional relationship between the image of the area where the controlled device is located and the captured image.

[0090] According to the fourth aspect, or any of the above implementations of the fourth aspect, the camera includes EVS camera function and APS camera function, and the camera acquires the shooting image, including: acquiring the shooting image by combining the EVS camera function and APS camera function of the camera.

[0091] According to the fourth aspect, or any implementation of the fourth aspect above, the captured image is a first captured image obtained through the APS camera function of the camera, or a second captured image obtained through the EVS camera function of the camera. The first captured image is acquired through the APS camera function of the camera at a preset period, and the first captured image is used to determine a first pointing position of the control device. The controlled device is instructed to display a first element at the display position corresponding to the first pointing position. During the interval of the preset period, the second captured image is acquired through the EVS camera function of the camera. Based on the second captured image, the relative displacement of the change in the first pointing position is obtained. Based on the relative displacement, the second pointing position is obtained. The controlled device is instructed to display the first element at the display position corresponding to the second pointing position.

[0092] According to the fourth aspect, or any implementation of the fourth aspect above, when the processor reads computer instructions from memory, it also causes the control device to execute: displaying a control interface, the control interface being used to receive user operations to control the controlled device.

[0093] According to the fourth aspect, or any implementation of the fourth aspect above, when the processor reads computer instructions from memory, it also causes the control device to perform the following actions: sequentially acquiring a first captured image and a second captured image, wherein the first captured image includes an image captured towards the area where the first controlled device is located, and the second captured image includes an image captured towards the area where the second controlled device is located. The first controlled device and the second controlled device are then controlled to work together.

[0094] Fifthly, an electronic device is provided, which is a parsing device. The parsing device includes a processor, a memory, and a display screen, coupled to the processor. The memory stores computer program code, including computer instructions. When the processor reads the computer instructions from the memory, the parsing device performs the following actions: receiving a captured image from a camera sent by a control device, the captured image including an image captured towards the area where the controlled device is located; determining, based on features in the captured image, that the electronic device requiring a user interaction event is a controlled device; and, based on the controlled device being a parsing device, performing an operation indicated by the user interaction event in response to the user interaction event, wherein the user interaction event includes an interaction event where the control device points to the controlled device; or, based on the controlled device being a controlled device other than the parsing device, sending information to the control device to instruct the controlled device.

[0095] According to the fifth aspect, the operation of executing the user interaction event indication includes: in response to the pointing position of the control device, displaying a first element at a display position corresponding to the pointing position on the controlled device. And / or, the user interaction event further includes: a selection operation on a control component on the control device; the operation of executing the user interaction event indication includes: performing the operation corresponding to the selection operation based on the selection operation.

[0096] According to the fifth aspect, or any implementation of the fifth aspect above, the camera includes a vision sensor, and the captured image includes identifying data of the controlled device captured using the vision sensor and / or data generated by capturing the display image of the controlled device.

[0097] According to the fifth aspect, or any implementation thereof, the identifiable data includes the outline data of the controlled device, and / or data used to locate the position of the controlled device in the captured image.

[0098] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the parsing device to perform: based on the positional relationship between the identifying data and the captured image, or based on the data generated from capturing the display image of the controlled device, to obtain the display position on the controlled device corresponding to the pointing position of the control device.

[0099] According to the fifth aspect, or any implementation of the fifth aspect above, the information sent to the control device to instruct the controlled device is used by the control device to establish a communication connection with the controlled device based on the determined controlled device, so as to execute user interaction events.

[0100] According to the fifth aspect, or any implementation of the fifth aspect above, the camera includes an event-based visual sensor (EVS) camera; receiving the captured image obtained by the control device through the camera sent by the control device includes: receiving the EVS event generated by the control device through the EVS camera sent by the control device, wherein the EVS event is generated based on the light source configured by the controlled device and / or the feature image displayed by the controlled device, and the EVS event is used to generate the captured image.

[0101] According to the fifth aspect, or any implementation of the fifth aspect above, the light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

[0102] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the parsing device to perform: depending on the controlled device being the parsing device, displaying the feature image at a preset position on the display screen, or displaying the feature image in full screen.

[0103] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the parsing device to perform: filtering and / or lens parameter correction on the EVS event, generating a captured image. Based on the captured image, it acquires the image captured in the captured image that is directed towards the area where the controlled device is located.

[0104] According to the fifth aspect, or any implementation of the fifth aspect above, the camera includes an active pixel sensor (APS) camera.

[0105] According to the fifth aspect, or any implementation thereof, determining the electronic device that needs to perform the user interaction event as the controlled device by identifying features in the captured image includes: comparing feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image; and based on the first position, obtaining the positional relationship between the image of the area where the controlled device is located and the captured image.

[0106] According to the fifth aspect, or any implementation of the fifth aspect above, the camera includes an EVS camera function and an APS camera function, which are used together to acquire the captured image.

[0107] A sixth aspect provides an electronic device. The electronic device includes a processor and a memory coupled to the processor. The memory stores computer program code, which includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device performs the method of the third aspect or any embodiment of the third aspect.

[0108] According to the sixth aspect, the electronic device is a parsing device.

[0109] A seventh aspect provides an electronic device. The electronic device includes: a processor, a memory, and a display screen, the memory and the display screen being coupled to the processor. The memory stores computer program code, the computer program code including computer instructions. When the processor reads the computer instructions from the memory, it causes the electronic device to perform the method of the second aspect or any embodiment of the second aspect, or causes the electronic device to perform the method of the third aspect or any embodiment of the third aspect.

[0110] Eighthly, an electronic device is provided that has the function of implementing the remote control method as described in the second aspect and any of its possible implementations, or the electronic device has the function of implementing the remote control method as described in the third aspect and any of its possible implementations. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0111] Ninthly, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program (also referred to as instructions or code) that, when executed by an electronic device, causes the electronic device to perform the method of the second aspect or any embodiment of the second aspect, or causes the electronic device to perform the method of the third aspect or any embodiment of the third aspect.

[0112] In a tenth aspect, a computer program product is provided that, when the computer program product is run on an electronic device, causes the electronic device to perform the method of the second aspect or any one of the embodiments of the second aspect, or causes the electronic device to perform the method of the third aspect or any one of the embodiments of the third aspect.

[0113] Eleventhly, a circuit system is provided, the circuit system including a processing circuit, the processing circuit being configured to perform the method of the second aspect or any embodiment of the second aspect, or the processing circuit being configured to perform the method of the third aspect or any embodiment of the third aspect.

[0114] In a twelfth aspect, a chip system is provided, including at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor, and when the at least one processor executes the instructions, the at least one processor performs the method of the second aspect or any embodiment of the second aspect, or the at least one processor performs the method of the third aspect or any embodiment of the third aspect.

[0115] The technical effects of the aforementioned aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0116] Figure 1 This application provides a schematic diagram of a remote control scenario. Figure 1 ;

[0117] Figure 2 A schematic diagram of the communication system used in the remote control method provided in the embodiments of this application;

[0118] Figure 3 A schematic diagram of the hardware structure of the first electronic device provided in an embodiment of this application;

[0119] Figure 4 A schematic diagram of the hardware structure of the second electronic device provided in an embodiment of this application;

[0120] Figure 5 This is a schematic diagram of the captured image provided in an embodiment of this application;

[0121] Figure 6A Schematic diagram of light source position provided for embodiments of this application Figure 1 ;

[0122] Figure 6B This is a schematic diagram of a focus position recognition scenario provided in an embodiment of this application;

[0123] Figure 7 Schematic diagram of light source position provided for embodiments of this application Figure 2 ;

[0124] Figure 8 This is a schematic diagram of the feature screen position provided in the embodiments of this application;

[0125] Figure 9 Module interaction diagram provided for embodiments of this application Figure 1 ;

[0126] Figure 10 A schematic diagram of feature points provided for embodiments of this application;

[0127] Figure 11 This is a schematic diagram illustrating the correspondence between the display screen and the captured image provided in the embodiments of this application;

[0128] Figure 12 This is a schematic diagram of a relative displacement determination scenario provided in an embodiment of this application;

[0129] Figure 13 Module interaction diagram provided for embodiments of this application Figure 2 ;

[0130] Figure 14 This application provides a schematic diagram of a remote control scenario. Figure 2 ;

[0131] Figure 15 This is a schematic diagram of a remote control device display scene provided in an embodiment of this application;

[0132] Figure 16 This application provides a schematic diagram of a remote control scenario. Figure 3 ;

[0133] Figure 17 This is a schematic flowchart of the remote control method provided in the embodiments of this application;

[0134] Figure 18 This is a schematic diagram of the structure of the control device provided in the embodiments of this application;

[0135] Figure 19 A schematic diagram of the structure of the parsing device provided in an embodiment of this application. Detailed Implementation

[0136] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two).

[0137] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0138] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0139] In some embodiments, users can control the smart screen by pointing it via a remote control device. For example, such as... Figure 1 As shown, both the remote control device and the smart screen are equipped with wireless positioning modules. The remote control device sends signals through the wireless positioning module. When the remote control device is pointed at the smart screen, the smart screen can receive the signals sent by the remote control device through the wireless positioning module. Then, the smart screen can calculate the angle of incidence of the signal based on information such as the phase difference or time difference of the signals received by the multiple antennas included in the wireless positioning module, thereby determining the pointing direction of the remote control device relative to the smart screen. In this way, the smart screen can display the corresponding cursor position based on the pointing direction of the remote control device.

[0140] However, current pointing and positioning technologies rely on hardware configuration, increasing the hardware cost of the device. Furthermore, the pre-installation process requires the installation and adjustment of the supporting hardware, resulting in poor overall flexibility.

[0141] Therefore, this application provides a remote control method that can determine the controlled device by the camera image captured by the control device, thereby enabling the control device to intelligently control the controlled device, thus reducing hardware costs while meeting user interaction needs.

[0142] Figure 2 This is a schematic diagram of the communication system used in the remote control method provided in the embodiments of this application. Figure 2 As shown, the communication system includes a first electronic device 100 and a second electronic device 200.

[0143] Optionally, the first electronic device 100 may be a remote control handle, wearable device, mobile phone or other terminal device with remote control function. This application does not limit the specific type of the first electronic device 100.

[0144] Optionally, the second electronic device 200 can be, for example, a smart screen, projector, smart home device (such as smart light, smart speaker, etc.), laptop, mobile phone, tablet, in-vehicle terminal, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), artificial intelligence (AI) device, etc. The operating system installed on the second electronic device 200 includes, but is not limited to, Alternatively, other operating systems may be used. This application does not limit the specific type of the second electronic device 200 or the operating system installed on it.

[0145] In some embodiments, a wireless communication connection is established between the first electronic device 100 and the second electronic device 200. The wireless communication technology used to establish this connection includes, but is not limited to, at least one of the following: Bluetooth (BT) (e.g., classic Bluetooth or Bluetooth Low Energy (BLE)), wireless local area networks (WLAN) (such as Wi-Fi), near field communication (NFC), Zigbee, frequency modulation (FM), infrared (IR), and starlight.

[0146] Optionally, the first electronic device 100 and the second electronic device 200 can also establish a communication connection through a third-party device in the local area network, such as a router, gateway, smart device controller, server, etc.

[0147] In other embodiments, a wired communication connection may also be established between the first electronic device 100 and the second electronic device 200. For example, the first electronic device 100 and the second electronic device 200 may be connected via a universal serial bus (USB) interface.

[0148] Optionally, the first electronic device 100 is, for example, a remote control device, and the second electronic device 200 is, for example, a remotely controlled smart device. Optionally, in a smart home scenario, the second electronic device 200 is, for example, a smart home device. In a smart office scenario, the second electronic device 200 is, for example, a smart office device. This application embodiment does not limit the application scenarios of the first electronic device 100 and the second electronic device 200; for example, the remote control method provided in this application embodiment can also be applied to other remote control scenarios.

[0149] Figure 3 This is a schematic diagram of the hardware structure of the first electronic device 100 provided in an embodiment of this application. Figure 3 As shown, the first electronic device 100 includes at least one processor 301, a communication line 302, a memory 303, at least one communication interface 304, a camera 305, and a sensor module 306. The memory 303 may also be included within the processor 301.

[0150] The processor 301 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0151] Communication line 302 may include a path for transmitting information between the aforementioned components.

[0152] Communication interface 304 is used for communication with other devices. In this embodiment, the communication interface can be a module, circuit, bus, interface, transceiver, or other device capable of communication functions, used for communication with other devices. Optionally, when the communication interface is a transceiver, the transceiver can be a separately configured transmitter used to send information to other devices, or it can be a separately configured receiver used to receive information from other devices. The transceiver can also be a component that integrates sending and receiving information functions; this embodiment does not limit the specific implementation of the transceiver.

[0153] Memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processor via communication line 302. Memory may also be integrated with the processor.

[0154] The memory 303 stores computer execution instructions for implementing the solutions of this application, and its execution is controlled by the processor 301. The processor 301 executes the computer execution instructions stored in the memory 303, thereby implementing the data processing method provided in the following embodiments of this application.

[0155] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, instructions, computer program or other names, and the embodiments of this application do not specifically limit them.

[0156] In a specific implementation, as one example, processor 301 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 in the CPU.

[0157] In a specific implementation, as one example, the first electronic device 100 may include multiple processors, such as... Figure 3 Processors 301 and 307 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0158] Camera 305 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an image signal processor (ISP) to be converted into a digital image signal. The ISP outputs the digital image signal to a digital signal processor (DSP) for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the first electronic device 100 may include one or N cameras 305, where N is a positive integer greater than 1.

[0159] In some embodiments, the camera 305 includes, for example, an event-based vision sensor (EVS) camera, or simply an event sensor or event camera. The first electronic device 100 determines the outline data of the second electronic device 200 (such as a smart screen) captured in the EVS camera's image. The center position of the current captured image is the pointing position of the first electronic device 100. Therefore, based on the positional relationship between the outline data of the second electronic device 200 and the captured image, the first electronic device 100 can determine the pointing position of itself on the display screen of the second electronic device 200. Optionally, the EVS camera can also capture other features on the second electronic device 200 used for positioning and determine the pointing position through analysis and calculation.

[0160] Optionally, a light source that flashes at a preset frequency is installed at a preset position on the second electronic device 200, or the second electronic device 200 flashes an image at a preset position on the display screen at a preset frequency. In this way, the EVS camera can generate a captured image by capturing these flashing events. Optionally, the flashing light source or flashing image is invisible to the user.

[0161] In other embodiments, camera 305 also includes an active pixel sensor (APS) camera. The first electronic device 100 captures an image using the APS camera. By comparing the captured image with the feature points included in the display image of the second electronic device 200, the position of the data generated from capturing the display image in the captured image can be determined, thereby determining the pointing position of the first electronic device 100 in the display image of the second electronic device 200.

[0162] Sensor module 306 is used to detect the attitude information of the first electronic device 100. For example, sensor module 306 includes an inertial measurement unit (IMU). Optionally, the IMU may include sensors such as accelerometers and gyroscopes, which can be used to measure motion parameters such as acceleration, angular velocity, and orientation change of the first electronic device 100, thereby determining the attitude of the first electronic device 100. In this way, based on the measurement results of sensor module 306, the first electronic device 100 can correct the image captured by camera 305 to make its orientation consistent with reality.

[0163] It is understood that the embodiments illustrated in this application are as follows: Figure 3 The structure does not constitute the only limitation on the structural implementation of the first electronic device 100. In other embodiments of this application, the first electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0164] Figure 4 This is a schematic diagram of the hardware structure of the second electronic device 200 provided in an embodiment of this application. Figure 4 As shown, the second electronic device 200 may include a processor 410, a memory 420, a wireless communication module 430, an antenna, a power management module 440, an audio module 450, a display screen 460, a camera 470, a sensor module 480, and buttons 490, etc.

[0165] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the second electronic device 200. In other embodiments of this application, the second electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0166] Processor 410 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0167] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0168] The processor 410 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. This memory can store instructions or data that the processor 410 has just used or that are used repeatedly. If the processor 410 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0169] In some embodiments, the processor 410 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0170] The MIPI interface can be used to connect the processor 410 to peripheral devices such as the display screen 460 and the camera 470. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 410 and the camera 470 communicate via the CSI interface to enable the shooting function of the second electronic device 200. The processor 410 and the display screen 460 communicate via the DSI interface to enable the display function of the second electronic device 200.

[0171] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the second electronic device 200. In other embodiments of this application, the second electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0172] The wireless communication function of the second electronic device 200 can be realized through an antenna, a wireless communication module 430, a modem processor, and a baseband processor.

[0173] Antennas are used to transmit and receive electromagnetic wave signals. Each antenna in the second electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0174] The wireless communication module 430 can provide solutions for wireless communication applications on the second electronic device 200, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 430 can be one or more devices integrating at least one communication processing module. The wireless communication module 430 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 410. The wireless communication module 430 can also receive signals to be transmitted from the processor 410, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via the antenna.

[0175] In some embodiments, the second electronic device 200 establishes a wireless communication connection with the first electronic device 100 via the wireless communication module 430. Optionally, the second electronic device 200 can receive a captured image sent by the first electronic device 100 through this wireless communication connection. This captured image is an image including the second electronic device 200 captured by the first electronic device 100 through its camera. Then, based on the captured image and the displayed image, the second electronic device 200 can determine the orientation of the first electronic device 100 within the displayed image of the first electronic device 100.

[0176] The second electronic device 200 implements display functions through a GPU, a display screen 460, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 460 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0177] Display screen 460 is used to display images, videos, etc. Display screen 460 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), such as an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the second electronic device 200 may include one or N displays 460, where N is a positive integer greater than 1.

[0178] The following section uses the example of the first electronic device 100 as the remote control device and the second electronic device 200 as the remotely controlled smart device (such as a smart screen or other smart home device) to provide a detailed description of the remote control method provided in the embodiments of this application.

[0179] In some embodiments, a user holds a remote control device and points it at a smart device, intending to remotely control the smart device. During this process, the remote control device can capture an image using a camera, including an image taken from the area where the smart device is located. For simplicity, the image taken from the area where the smart device is located will be described below as the smart device image. It should be understood that this smart device image is a captured image and should not be simply interpreted as an image that reflects the appearance of the smart device in a 1:1 ratio in the real environment, or a 1:1 ratio of the smart device's display. Thus, the remote control device or the smart device can determine the change in the focus position of the remote control device based on the change in the position of the smart device image within the captured image.

[0180] Optionally, the remote control device analyzes the captured image to determine the change in the focus position. Optionally, the remote control device sends the captured image to a smart device, which then analyzes the captured image to determine the change in the focus position of the remote control device.

[0181] Optionally, the smart device is a display device such as a smart screen. After determining the focus position pointed to by the remote control device, the smart device can display the corresponding display element at the display position corresponding to the focus position. As the focus position changes, the display position of the display element also changes, thereby helping the user understand the corresponding pointing position. Optionally, the display element can be, for example, a cursor, a trajectory, etc. The cursor can be, for example, a circle, an arrow, a game controller shape, a virtual character, or other display forms.

[0182] Optionally, the smart device is a device without a display function, and the focus position is used to determine the smart device being pointed to, or a certain position on the smart device being pointed to.

[0183] For example, such as Figure 5 As shown, the remote control device captures a shot via a camera, which includes the smart screen image. As the focus point changes, the position of the smart screen image within the captured shot also changes. Therefore, if the position of the smart screen image is fixed, the captured shot will move accordingly. Based on the positional relationship between the captured shot and the smart screen image, coordinate transformation can be performed to obtain the focus point. Optionally, the center of the captured shot is the focus point. Based on the positional relationship between the center of the captured shot and the center of the smart screen image, coordinate transformation can be performed to obtain the display position of the focus point within the smart screen image, thus enabling the smart screen to display elements at that position. This display position can be the focus point itself or a reference point, where the focus point is an absolute or relative coordinate within the smart screen's display.

[0184] Based on the above, it is clear that remote control devices or smart devices need to obtain the position of the smart device's image within the captured frame in order to determine the focus point. The following section details the specific process of determining the position of the smart device's image within the captured frame.

[0185] In some embodiments, users hold the remote control device in different postures, resulting in the device exhibiting different postures during shooting. Optionally, the remote control device can acquire its posture information through a posture sensing module. Subsequently, based on this posture information, the captured image can be corrected to ensure that the image orientation is consistent with reality.

[0186] For example, the attitude perception module includes an IMU (Induction Unit). Based on the IMU's detection data, the remote control device can understand its motion state in real time and estimate its attitude, outputting attitude information. Thus, during the recording process, if the remote control device's angle deviates from a predetermined direction based on its attitude information, the device can correct the captured image to ensure the output image's orientation matches reality. For instance, if the user holds the remote control but it's not horizontal or nearly horizontal, but at an angle to the horizontal, the captured image will rotate accordingly, and the smart device's image within the captured image will also rotate. However, smart devices are typically placed on a horizontal surface; without correction, the final output focus position will deviate from the user's desired pointing position.

[0187] In this way, by correcting the orientation of the captured image, it is easier to output the correct focus position later.

[0188] In some embodiments, the remote control device may also perform blind spot compensation (also described as blind interpolation) and frame interpolation on the captured image based on IMU detection data, thereby improving the accuracy and stability of the captured image and ensuring the smoothness of the remote control process.

[0189] Optionally, blind zone compensation refers to situations where, under certain circumstances, the IMU may be unable to provide accurate data due to external interference or physical limitations (e.g., sensor failure for a short period, or the sensor entering a blind zone). In such cases, the system needs to use intelligent algorithms to compensate for this lost or inaccurate data to ensure the overall stability and accuracy of the system.

[0190] For example, during remote control operation, situations such as other people passing by might obstruct the smart device, resulting in missing footage. In such cases, the smart device can supplement the missing information based on IMU detection data through interpolation, prediction, or other estimation methods. This avoids missing focus points and prevents remote control malfunctions or other anomalies.

[0191] Optionally, frame interpolation is used when the IMU has limited or no continuous angle data. It involves using interpolation techniques to calculate "intermediate frames," essentially inserting new data points between two frames. This helps improve the smoothness of device movement, especially in applications requiring smooth transitions or precise control, such as smart devices controlled by remote controls.

[0192] For example, an IMU can only provide a certain amount of detection data per second, while a higher frame rate is needed in practical applications to ensure a smoother control experience. Frame interpolation technology uses algorithms to calculate the angle value between two frames, thereby providing higher motion accuracy and a better user experience. Another example is that frame interpolation can also reduce power consumption and extend the usage time of remote control devices.

[0193] In some embodiments, the camera in the remote control device includes, for example, an EVS camera. The EVS camera can capture brightness changes of each pixel. When the brightness change of a pixel in the scene exceeds a preset threshold, that pixel records an "event" and transmits the timestamp and pixel position of the event. In high-speed, dynamic, or low-light environments, the EVS camera offers higher response speed, lower latency, better dynamic range, and lower power consumption. Therefore, the remote control device can use the EVS camera to take pictures and output the captured images to determine information such as the location of the smart device.

[0194] Optionally, as mentioned above, the remote control device or smart device needs to determine the focus position based on the position of the smart device's image within the captured image, while the EVS camera is used to capture events where brightness changes exceed a preset threshold. Therefore, by configuring a flashing light source that matches the external contour of the smart device, the smart device's image within the captured image can be identified, thereby determining the smart device's position and outputting the correct focus position. Optionally, this light source flashes at a preset frequency and is invisible to the user, thus avoiding any impact on the user. Optionally, this light source may be, for example, an infrared light source.

[0195] Optionally, the remote control device may have pre-set information such as the outline shape, size, and device ID of the smart device, which will facilitate the identification of the smart device included in the captured image based on this information.

[0196] In some examples, a fixed light source is added outside the smart device to locate the smart device's position.

[0197] For example, such as Figure 6AAs shown, the display surface of a smart screen is generally rectangular, so light sources can be installed at each of the four corners. After the smart screen is powered on, these four light sources can flash at a preset frequency. Alternatively, the smart screen can trigger these four light sources to flash at a preset frequency after receiving a pointing signal from a remote control device. Thus, as... Figure 6B In the captured image shown in (a), the remote control device can determine the outline of the smart screen in the captured image based on the acquired EVS events. Subsequently, the remote control device can perform coordinate transformation on the center of the captured image based on the location of the smart screen's outline to obtain the focal point position pointed to by the remote control device. This focal point position is, for example, the coordinates in the transformed display screen of the smart screen. Then, as... Figure 6B As shown in (b), the remote control device can send the focus position to the smart screen to trigger the smart screen to display cursor 61 at the focus position. Thus, as the user points the remote control device at the smart screen, the remote control device can determine the focus position through the image captured by the EVS camera, thereby displaying the cursor at the corresponding position on the smart screen. This helps the user locate the direction of the remote control device to trigger the required response event. For example, by changing the cursor position, the user can trigger a click operation on a control displayed on the smart screen, enabling the smart screen to execute the event corresponding to that control.

[0198] Optionally, the remote control device can also send the captured image to a smart device, which then processes the image to obtain the focus position. That is, in this embodiment, the device performing data processing is not displayed; it can be either a remote control device or a smart device. Further details will not be provided below.

[0199] Optionally, based on the external outline shape of the smart device and the pattern of light emission, one or more light sources (such as infrared light sources) can be configured on the smart device to identify information such as the device's outline and position. For example, as in the scenario described above, a light source can be installed at each of the four corners of the smart screen. Alternatively, in addition to installing light sources at the four corners of the smart screen, light sources can also be installed along each edge of the smart screen. Optionally, light sources capable of emitting multiple beams can be installed near or inside devices such as cameras, speakers, and screen decorations on the smart screen. For example, a light source capable of emitting four beams can be installed inside the camera of the smart screen, with the positional relationship between these four beams corresponding to the four corners of the smart screen. In this way, as the light source flashes, the remote control device can also identify the external outline of the smart screen in the image captured by the EVS, thereby determining the focus position.

[0200] For example, such as Figure 7 As shown in (a), this illustrates the four-point arrangement of light sources on the camera of the smart screen. Figure 7As shown in (b), there is a two-point layout of light sources on the camera of the smart screen, and a single-point layout of light sources in the screen deco. Subsequently, based on the layout and position of these light sources, geometric positioning can be used to identify the external outline of the smart screen in the captured image corresponding to the EVS event, thereby determining the focus position.

[0201] In other examples, smart devices with display capabilities can also locate the smart device in the captured image by displaying a flashing feature image. Optionally, the smart device displays a feature image that is invisible to the user at a preset frequency, thereby avoiding the display of the feature image from affecting the user's viewing of the displayed image.

[0202] Optionally, the feature image may include information such as a device identifier (ID) or coordinate information to facilitate the determination of the smart device's location from the captured image. The device identifier is used to uniquely identify the corresponding smart device, and may include information such as the smart device's device code, name, and model. Optionally, the feature image may be presented in an encoded manner to facilitate the remote control device or smart device to read the information carried in the feature image. Optionally, the feature image may not need to carry information; it may be captured by the EVS camera through flashing to determine the smart device's location.

[0203] Optionally, the feature image is displayed at a preset position on the smart device's display screen to determine the smart device's outline. For example, if the smart device is rectangular, the feature image is displayed at the edges of the four corners of the smart device's display screen. Alternatively, displaying the feature image in full screen can also determine the smart device's outline, such as the outline of the smart device's display screen.

[0204] Optionally, the number of feature screens displayed can be one or more.

[0205] For example, such as Figure 8As shown, the smart screen displays feature images at the four corner edges, carrying device identification information. After the smart screen is activated, or receives a pointing signal from the remote control device, it can display these feature images at a preset frequency. This allows the remote control device to determine the outline of the smart screen in the captured image based on acquired EVS events. Subsequently, the remote control device can perform coordinate transformation on the center of the captured image based on the smart screen's outline location, obtaining the focal point position where the remote control is pointing. This focal point position is, for example, the coordinates in the transformed smart screen display. Then, the remote control device can send this focal point position to the smart screen to trigger the display of a cursor at the focal point. Thus, as the user points at the smart screen with the remote control device, the remote control device can determine the focal point position through the EVS camera's captured image, thereby displaying a cursor at the corresponding location on the smart screen. This helps the user locate the remote control device's direction to trigger the required response event.

[0206] Optionally, the remote control device may be pre-configured with information such as the size of the smart device and the display position of the featured image, or it may be able to receive this information sent by the smart device. Based on this information, the remote control device can more accurately determine the position of the smart device's image in the captured frame. For example, if the featured image is some distance from the edge of the smart screen's display, then if the remote control device can obtain this distance information in advance, it can more accurately identify the outer contour of the smart screen in the captured frame.

[0207] In this way, smart devices can provide flashing light sources to determine their location in various ways, allowing remote control devices to determine the focus point by capturing images. Compared to configuring separate wireless positioning modules in the remote control and smart devices, relying solely on image analysis to determine the focus point effectively reduces hardware costs.

[0208] In some embodiments, the smart device can be accessed via, for example... Figure 8The illustrated method displays a characteristic image, which serves as identifying data for the smart device. This image can be captured by the remote control device to determine the smart device's location. In other examples, the smart device can also configure this identifying data in other ways; this identifying data is inherent to the smart device. For example, the smart device may have an external light source, as described above, whose position indicates the smart device's external outline. Alternatively, the smart device may have an external light source or other identifying device (such as a sticker attached to the outside of the smart device). The data corresponding to this device can also carry data for locating the smart device, which can be acquired by the remote control device while it is pointing towards the area where the smart device is located. This data includes, for example, data indicating the positional relationship between the device's location and the smart device's location. Thus, based on this data, the remote control device can locate the smart device and achieve directional control.

[0209] In some embodiments, events captured by the remote control device via the EVS camera may be affected by the shooting environment, resulting in abnormalities such as noise and irrelevant changes, which may affect the output captured images. Therefore, after acquiring the event stream, the remote control device can first filter the event stream to remove unnecessary noise or erroneous data and retain important information.

[0210] Optionally, the remote control device can process the event stream captured by the EVS camera through temporal filtering, two-dimensional filtering, etc., to optimize and sharpen the EVS camera's shooting results. The event stream typically refers to the data stream captured by the EVS camera. The EVS camera captures changes in each pixel (e.g., changes in light intensity), rather than traditional frame images. In this case, temporal filtering refers to processing these events sequentially, filtering out noise or irrelevant changes while retaining meaningful motion or change information. Temporal filtering can generally smooth signals, remove random noise, or remove irrelevant rapid changes. Two-dimensional filtering refers to processing image data in the spatial domain (i.e., pixel locations in space). Common two-dimensional filtering operations include smoothing, sharpening, and edge detection. This processing adjusts the visual effect of the image, removes noise, or highlights certain features by operating on each pixel and its neighboring pixels. Spatial filtering focuses on local or global spatial processing of static images.

[0211] In some embodiments, events captured by the remote-controlled device via the EVS camera are affected by intrinsic parameters calibrated by the EVS camera, which may introduce some distortions. For example, common distortions include barrel distortion (e.g., the image center is relatively normal, but the corners are stretched) or pincushion distortion (e.g., the corners are shrunken). By correcting the lens parameters of the EVS camera's calibrated intrinsic parameters, the remote-controlled device can make the image closer to the actual scene, reducing or eliminating these visual distortions.

[0212] Optionally, each camera has some inherent parameters (such as focal length, principal point position, etc.), which are obtained through the camera calibration process. Calibration is a calculation process used to determine the intrinsic characteristics of the camera; these parameters typically include the camera's focal length, optical center, distortion coefficient, etc. Lens distortion refers to the bending or distortion of the image at the edges due to the physical characteristics of the lens. The lens parameter correction process corrects the image by using intrinsic parameters (such as distortion coefficients) obtained from camera calibration to eliminate or reduce this distortion.

[0213] For example, after acquiring an event stream by capturing images with an EVS camera, the remote control device filters the event stream. Then, the remote control device performs lens parameter correction on the filtered image using the intrinsic parameters calibrated by the EVS camera to eliminate distortion introduced by the EVS camera.

[0214] The preceding text described a scenario where the focus position is determined based on the event stream captured by the EVS camera within the remote control device. The following text uses a smart screen as an example to illustrate the module interaction process between the remote control device and the smart device in this scenario.

[0215] For example, such as Figure 9As shown, the remote control device includes an EVS camera, an IMU, a processor, and a wireless communication module. The remote control device captures an event stream using the EVS camera and sends this stream to the processor. The processor receives the event stream and also receives detection data from the IMU. The processor then performs orientation calibration and other processing on the event stream based on the IMU detection data. The processor can also filter and correct the calibrated event stream before outputting the captured image. In some examples, the remote control device can further process the captured image through the processor to obtain the position of the smart device within the captured image. Based on the positional relationship between the smart device image and the captured image, coordinate transformation is performed to obtain the focus position, such as coordinates. The remote control device then sends this focus position to the smart screen via the wireless communication module. In other examples, the remote control device can also directly send the captured image to the smart screen via the wireless communication module, where the smart screen processes the image to obtain the focus position. Optionally, the remote control device and the smart screen can establish various types of wireless communication connections, such as Bluetooth, WiFi, and Wi-Fi, via the wireless communication module to achieve data transmission. Optionally, the events sent by the remote control device to the smart screen may include not only the focus position or the captured image, but also information such as pressure-sensitive data and timestamps, so that the smart screen can determine the corresponding response event.

[0216] Subsequently, after receiving the event sent by the remote control device via the wireless communication module, the smart screen can forward the event to the input adaptation module. Optionally, if the event received by the smart screen includes a captured image to be processed, the captured image can be processed to obtain the focus position before sending the focus position and other information in the event to the input adaptation module. After receiving the event, the input adaptation module can parse and combine the information in the event, converting the event into an interactive event recognizable by the operating system, and report the interactive event to the operating system. After receiving the interactive event, the operating system can instruct the display module to display the element at the corresponding focus position on the screen.

[0217] The above-described module interaction process utilizes the flickering of an external light source on the smart screen to output the corresponding captured image from the EVS camera. In some examples, the smart screen can also display feature images through a display module to achieve positioning of the smart screen and obtain the focus position.

[0218] Optionally, during the display of the feature image in the display module, if the image resolution is performed on the smart screen side, the smart screen can obtain the position of the smart screen image in the captured image based on the feature information of the feature image to obtain the focus position. The feature information includes, for example, the position information of the feature image and the encoded information it carries, facilitating the identification of the feature image and determining the positional relationship between the feature image and the smart screen image. If the image resolution is performed on the remote control device side, the smart screen can obtain the feature information through the feature point processing module and send this feature information to the remote control device through the wireless communication module. This allows the remote control device to perform image processing on the captured image based on the feature information and output the focus position.

[0219] Optionally, smart devices with display functions, such as smart screens, can achieve positioning by using an external light source and / or displaying characteristic images.

[0220] The following section describes other ways to implement a camera.

[0221] In some embodiments, the camera in the remote control device includes, for example, an APS camera. An APS camera contains active elements (such as transistors) in each pixel unit, enabling each pixel to actively read and amplify photoelectric signals, thereby enhancing signal reading and processing and reducing reliance on external circuitry. APS cameras effectively improve the sensitivity and dynamic range of each pixel, performing particularly well in low-light environments. Therefore, they can respond more quickly to changes in light, making them suitable for high frame rate and fast image capture applications, such as rapid motion capture.

[0222] Optionally, a smart device with a display function can display an image. Then, after the remote control device acquires a captured image through the APS camera, it can determine the position of the displayed image within the captured image by comparing the captured image with the image displayed on the smart device. Based on this position, the position of the smart device's image within the captured image can be determined, thereby establishing the focus point.

[0223] Optionally, comparing the entire content of the displayed screen with the captured image would be difficult in data processing, consume a lot of power, and may also affect the smoothness of focus position determination. Therefore, by pre-setting feature points in the displayed screen, comparing a limited number of feature points in the displayed screen can not only ensure the accuracy of the smart device's position determination in the captured image, but also reduce data processing difficulty and power consumption, and improve the output efficiency of focus position. Optionally, the feature points include a certain range of the displayed screen, or a certain area of ​​the displayed screen at a fixed position.

[0224] For example, such as Figure 10As shown, the display screen of the smart screen includes multiple feature points. When a remote control is pointed at the smart screen, the remote control can take a picture using an APS camera. The captured image includes the smart screen's display, and therefore, the captured image will also include feature points. For example, as... Figure 11 The display screen shown in (a) is as follows: Figure 11 The image captured by the remote control device is shown in (b). The remote control device can then match feature points in the display screen with those captured by the APS camera to determine the feature points included in the captured image and their specific locations. The center of the APS camera's captured image is its focal point, as indicated by the direction the remote control device is pointing. By determining the positional relationship between the display screen and the captured image through the feature point locations and performing coordinate transformation, the focal point of the APS camera can be determined. This focal point is, for example, the focal point of the display screen that the remote control device is pointing at.

[0225] Alternatively, in order to reduce the data processing pressure on the remote control device, the remote control device can also send the captured image from the APS camera to the smart device (such as a smart screen) for processing to obtain the focus position.

[0226] In this way, by comparing the displayed images, the methods for obtaining the focus position are enriched. Furthermore, in scenarios where smart devices with display functions are pointing, the pointing position of the remote control device can be determined more accurately.

[0227] In some embodiments, the direction of the remote control device can be directly located based on the image captured by the APS camera. However, the processing of the APS camera image involves a large amount of data and high power consumption. Therefore, the direction of the remote control device can be located by combining the images captured by the APS camera and the EVS camera, thus saving power consumption. For example, the camera of the remote control device includes a hybrid event-based camera (HVS), which includes EVS camera functionality and APS camera functionality, enabling the fusion of EVS and APS data. Optionally, the remote control device includes both an EVS camera and an APS camera. Alternatively, the remote control device includes an HVS camera. For the sake of simplicity, the following description uses the example of a remote control device including both an EVS camera and an APS camera to illustrate the cooperative implementation process of the EVS camera functionality and the APS camera functionality. The inclusion of an HVS camera in the remote control device can be referred to in the relevant embodiments below, which will not be repeated in this application.

[0228] Optionally, the remote control device acquires images via an APS camera at preset intervals, while during these preset intervals, it can acquire images via an EVS camera. For example, after initially detecting a user's pointing action, the remote control device acquires an image via the APS camera and obtains the focus position to locate the direction of the remote control device. Subsequently, the remote control device acquires images via the EVS camera and uses EVS data (such as images captured by the EVS camera) to correct the displacement of the smart device's image, achieving continuous positioning of the smart device's image.

[0229] For example, a remote control device or smart device compares the EVS data of the current frame with that of the previous frame to obtain the displacement value. By calculating the relative displacement of the smart device's image within the EVS data, the specific position of the smart device's image in the current frame can be determined based on its position and relative position in the previous frame. During this process, the remote control device again captures images from the APS camera at preset intervals, such as 1 second or 2 seconds. By matching the APS camera's captured images with the smart device's displayed images, the position of the smart device's image is output. This also calibrates the EVS data, facilitating more accurate positioning of the smart device's image based on the relative displacement of the EVS data in subsequent iterations. By repeating these steps, combining images from both the APS and EVS cameras, the device's direction can be determined while reducing data processing power consumption and improving positioning efficiency.

[0230] For example, the remote control device obtains the reference focus position within the current period using APS data (such as the image captured by an APS camera). Then, the remote control device obtains the EVS data of the next frame, acquires the relative displacement, and can determine a new focus position based on the reference focus position and the relative displacement. Subsequently, in each subsequent frame within the current period, the remote control device can obtain a new focus position based on the EVS data corresponding to the previous frame. For example, when the remote control device obtains data such as... Figure 12 The current frame EVS data in (a), and as shown in Figure 12 After the previous frame of EVS data shown in (b), the relative displacement of the feature points, including direction and distance, can be obtained by matching the feature points in the two frames of EVS data, thereby outputting the new focus position corresponding to the current frame.

[0231] In this way, by combining APS data and EVS data, the pointing of the remote control device can be achieved while reducing data processing power consumption.

[0232] Optionally, the remote control device can detect the user's pointing gestures using sensors such as an IMU. For example, when a user holds the remote control device, it changes the device's motion. Therefore, the remote control device can determine that it has detected the user's pointing gesture based on the IMU detection data. Alternatively, the remote control device can determine the user's pointing intention and confirm that it has detected the user's pointing gesture by detecting the user's operation on preset buttons or controls, or by detecting the user's voice command.

[0233] The preceding text described a scenario where the camera in the remote control device also includes an APS camera, and the focus position is determined based on the image captured by the APS camera. The following text uses a smart screen as an example to illustrate the module interaction process between the remote control device and the smart device in this scenario.

[0234] For example, such as Figure 13 As shown, the remote control device includes an EVS camera, an APS camera, a processor, and a wireless communication module. After detecting a user's pointing operation, the remote control device first acquires a captured image A through the APS camera and sets a preset period, instructing the APS camera to acquire captured image A according to the preset period. The processor acquires captured image A and sends it to the smart screen via the wireless communication module. Optionally, the remote control device can send a pre-processed captured image A to the smart screen. For example, the remote control device performs pre-processing on the captured image, such as orientation correction, filtering, and lens parameter correction, through the processor. Correspondingly, after receiving captured image A or the pre-processed captured image A, the smart screen can acquire the displayed image from the display module and the feature point information output by the feature point processing module to perform image matching and determine the focus position. Optionally, the smart screen can also send the focus position (or the position of the smart screen image in the captured image) to the remote control device via the wireless communication module so that the remote control device can subsequently obtain the relative displacement based on the focus position. During this process, the remote control device can also acquire a captured image B via the EVS camera and process this image B through a processor to obtain relative displacement based on EVS data. This allows for the continuous acquisition of new focus positions during the intervals between APS data acquisition. Optionally, the events sent by the remote control device to the smart screen include events corresponding to both APS data and EVS data. Upon receiving these events, the smart screen, through its input adapter module and operating system, can trigger continuous changes in the display positions of elements on the display module, helping the user understand the changes in the remote control device's pointing position.

[0235] Optionally, Figure 13 For the interaction between the remote control device and other modules in the smart screen, please refer to the above description. Figure 9 The relevant content will not be repeated here.

[0236] In some embodiments, the camera of the remote control device includes an EVS camera and an APS camera. The above description illustrates the process by which the EVS camera and the APS camera cooperate to acquire images at a preset cycle. It should be understood that the EVS camera and the APS camera can also cooperate to acquire images in other ways.

[0237] For example, the remote control device takes a picture of the area where the remotely controlled smart device is located using an APS camera to acquire the captured image. Then, based on this captured image, the remote control device identifies the remotely controlled smart device and can acquire subsequent images using an EVS camera. This achieves accurate identification of the smart device using an APS camera while saving power by using an EVS camera.

[0238] For example, the remote control device takes a picture of the area where the remotely controlled smart device is located using an APS camera to obtain the captured image. Then, based on the captured image, the remote control device identifies the remotely controlled smart device. Subsequently, if the remotely controlled smart device is a smart device with a display function (such as a smart screen), the remote control device can subsequently obtain the captured image using an EVS camera. If the remotely controlled smart device is a smart device without a display function (such as an air conditioner), the remote control device can subsequently obtain the captured image using an APS camera.

[0239] Thus, when the camera of the remote control device includes multiple camera functions, the remote control device can flexibly use multiple camera functions to acquire shooting images.

[0240] In some embodiments, the smart device is not configured with a display function. After receiving the focus position sent by the remote control device or calculating the focus position based on the captured image, the smart device can determine the physical button where the focus position is located, thereby triggering an operation event on that physical button. In this way, the user can trigger the smart device to perform the function required by the user by pointing the remote control device at the smart device.

[0241] The above text introduces the process of implementing pointing in positioning and remote control devices. The following text introduces more application scenarios of pointing in positioning and remote control devices.

[0242] In some embodiments, the remote control device can also identify the smart device using images captured by an EVS camera and / or an APS camera. Once the smart device is identified, the remote control device can then establish a wireless communication connection with it, enabling remote control of the smart device.

[0243] Optionally, the smart device is equipped with a flashing light source to enable the EVS camera of the remotely controlled device to capture the corresponding smart device image.

[0244] Optionally, after recognizing a smart device, the remote control device can select a communication protocol compatible with that smart device. Then, the remote control device can establish a communication connection with the smart device through this protocol. Subsequently, in response to the user's remote control operation, the remote control device can send corresponding control commands to the smart device to trigger the smart device to execute corresponding events. In this way, remote control operations such as switching the smart device on and off and changing modes can be achieved, enabling "point-and-click control."

[0245] Optionally, different types of smart devices may support the same or different wireless communication protocols, and smart devices of the same type but different models may support the same or different wireless communication protocols. Therefore, the smart device identified by the remote control device through image recognition can include the type and / or model of the smart device to ultimately determine the wireless communication protocol supported by the smart device. For example, the types of smart devices include smart lights, smart speakers, air purifiers, air conditioners, projectors, etc.

[0246] For example, such as Figure 14 As shown in (a), the remote control device responds to the user's pointing operation by taking a picture using the EVS camera. Then, through image recognition of the captured image, the remote control device identifies the smart device in the picture as a smart light. Next, the remote control device determines that the smart light supports the WiFi communication protocol. Therefore, the remote control device can establish a WiFi connection with the smart light based on the WiFi communication protocol. After the WiFi connection is established, the remote control device can send commands such as turning the smart light on or off via the WiFi connection, thereby meeting the user's remote control needs for the smart light.

[0247] For example, such as Figure 14 As shown in (b), the remote control device responds to the user's pointing operation by taking a picture using the EVS camera. Then, through image recognition of the captured image, the remote control device identifies the smart device in the picture as a smart speaker. Next, the remote control device determines that the smart speaker supports the Bluetooth communication protocol. Therefore, the remote control device can establish a Bluetooth connection with the smart speaker based on the Bluetooth communication protocol. After the Bluetooth connection is established, the remote control device can send commands such as turning the smart speaker on / off and adjusting the volume to the smart speaker, thereby meeting the user's remote control needs for the smart speaker.

[0248] In this way, the remote control device automatically establishes a wireless communication connection with the smart device by recognizing the device through image recognition, thereby enabling remote control of the smart device. Compared to the previous method where the user needed to pair the remote control device with the smart device before controlling it, the remote control method provided in this application effectively simplifies user operation and improves the user experience.

[0249] Optionally, after a smart device is identified, the remote control device or the smart device being remotely controlled can further determine the focus position pointed to by the remote control device through the methods described in the above embodiments, so as to trigger the smart device being remotely controlled to display elements at the corresponding display position or perform corresponding operations.

[0250] In some embodiments, the remote control device is equipped with a display screen. After establishing a communication connection with the smart device, the remote control device can display the control interface of the smart device on the display screen, thereby providing users with a more flexible and richer remote control experience for the smart device based on the control interface. Optionally, the displayed content of the control interface may be the same as or similar to the displayed content of the control interface after adding the smart device in a smart living application, or it may be a pre-configured control interface adapted to the remote control device. This application embodiment does not limit the displayed content of the control interface, which includes at least one functional control for realizing intelligent control of the smart device.

[0251] For example, such as Figure 14 As shown in (a), after the remote control device identifies the smart device through image recognition, it can display the following: Figure 15 The control interface 151 corresponding to the smart light is shown. After establishing a wireless communication connection with the smart device, the remote control device can generate remote control commands based on the user's operations on the control interface 151, and send these commands to the smart light via the wireless communication connection to control the smart light. For example, in response to the user's operation of the light-on control displayed on the control interface 151, the remote control device sends a light-on control command to the smart light via the wireless communication connection to trigger the smart light to turn on.

[0252] In some solutions, multiple embodiments of this application can be combined, and the combined solution can be implemented. Optionally, some operations in the processes of each method embodiment may be combined, and / or the order of some operations may be changed. Furthermore, the execution order between the steps of each process is merely exemplary and does not constitute a limitation on the execution order between steps; other execution orders are also possible. It is not intended to indicate that the execution order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. In addition, it should be noted that the process details involved in one embodiment of this document are similarly applicable to other embodiments, or different embodiments may be combined.

[0253] Furthermore, some steps in the method embodiments can be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and can be deleted in certain use cases. Or, other possible steps may be added to the method embodiments.

[0254] Furthermore, the various method embodiments can be implemented individually or in combination.

[0255] For example, after the remote control device identifies the smart device through the captured image, it automatically establishes a wireless communication connection with the smart device. During this process, the remote control device determines the focus position by processing the captured image, and in conjunction with the content described in the above embodiments, sends an event to the smart device based on the established wireless communication connection to trigger the smart device to respond to the event, such as displaying display elements.

[0256] In some embodiments, the remote control device can also enable functional linkage between different smart devices. Based on the user's pointing gesture, the remote control device identifies smart device A by recognizing the captured image. Then, it detects the user pointing in another direction. Based on this new pointing gesture, the remote control device identifies smart device B by recognizing the captured image. Thus, the remote control device can determine that it has detected the user's intention to link smart devices A and B. The remote control device can establish wireless communication connections with both smart devices A and B, and based on these connections, it acts as a relay device for data transmission between smart devices A and B, thereby enabling functional linkage between smart devices A and B.

[0257] Optionally, the remote control device can serve as a relay device between at least two smart devices that the user continuously points to.

[0258] Optionally, the remote control device is pre-configured with information about smart devices that support data relay functions. Optionally, the remote control device and / or smart device are pre-configured with supported linkage functions. For example, linkage functions include image streaming, etc.

[0259] For example, such as Figure 16As shown, the remote control device responds to the user's pointing operation by taking pictures using an EVS camera and / or an APS camera to acquire the captured image. The remote control device then processes the captured image to obtain the image of the smart device included within it. By recognizing the smart device image, it identifies the currently pointed smart device as the PC and establishes a wireless communication connection with the PC. If the remote control device detects the user pointing in another direction after detecting the user's pointing operation towards the PC, the smart device can determine that it has detected continuous pointing operations from the user. The remote control device takes pictures using an EVS camera and / or an APS camera to acquire the captured image. The remote control device then processes the captured image to obtain the image of the smart device included within it. By recognizing the smart device image, it identifies the currently pointed smart device as the smart screen and establishes a wireless communication connection with the smart screen. The remote control device can then send a data relay instruction to the PC and / or the smart screen. In response to this data relay instruction, the PC can send image data to the remote control device. After receiving the image data, the remote control device can forward the image data to the smart screen, thereby realizing screen mirroring from the PC to the smart screen.

[0260] Optionally, the remote control device, acting as a relay device, can also send device information of other targeted smart devices to the targeted smart device, thereby triggering the establishment of direct wireless communication connections between the various targeted smart devices. In this way, the remote control device no longer needs to perform data relay, but can automatically establish connections between different smart devices, simplifying the connection process for users and improving the user experience.

[0261] For example, such as Figure 16 In the scenario shown, after recognizing the PC and the smart screen, the remote control device can receive device information from both the PC and the smart screen via wireless communication, and send the smart screen's device information to the PC, and vice versa. Based on the received device information, the PC and the smart screen can establish a wireless communication connection. Using this connection, the PC can directly send image data to the smart screen, enabling screen mirroring.

[0262] In this way, the remote control device can recognize the user's continuous pointing operations to realize the functional linkage between multiple smart devices, enriching the user's remote control interaction experience.

[0263] It should be understood that the above description uses the example of a remote control device or a remotely controlled smart device processing the captured image to illustrate the image processing process. In other embodiments, the remote control device may also send the captured image to a third-party device for processing, and the remote control device can obtain the corresponding processing result. Optionally, the third-party device may be, for example, a smart screen, in which case the third-party device and the remotely controlled smart device may be the same device. Alternatively, the third-party device may be other electronic devices, such as a central control screen or a smart speaker. Optionally, the third-party device may be a dedicated device for processing captured images, or a device with the function of processing captured images. For example, as described above. Figure 2 The communication system shown may further include a third electronic device. After acquiring the captured image, the first electronic device can send the captured image to the third electronic device. Subsequently, the first electronic device can receive the image processing result sent by the third electronic device. Then, based on the processing result, the first electronic device can control the second electronic device.

[0264] Figure 17 This is a flowchart illustrating a remote control method provided in an embodiment of this application. It should be noted that this method does not rely on... Figure 17 The specific order described below is a limitation. It should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0265] S1701, The control device acquires a captured image through a camera, which includes an image captured towards the area where the controlled device is located.

[0266] In some embodiments, a user holds a control device and points it at the controlled device, intending to control the controlled device. The control device captures an image through a configured camera, and the captured image includes the view corresponding to the pointing direction. The controlled device is located in the pointing direction of the control device. Therefore, the image captured by the control device includes the image captured towards the area where the controlled device is located. For simplicity, the image captured towards the area where the controlled device is located will be described hereafter as the controlled device image.

[0267] Optionally, the camera includes a vision sensor. Optionally, the vision sensor is, for example, an EVS camera. Alternatively, the vision sensor is, for example, an APS camera. Alternatively, the vision sensor is, for example, an HVS camera that has both EVS and APS camera functions. Alternatively, the camera may include, for example, both an EVS camera and an APS camera.

[0268] In some embodiments, the controlled device is configured with a display function, or the controlled device is not configured with a display function.

[0269] In some embodiments, the captured images include identifying data of the controlled device captured using a vision sensor and / or data generated from capturing the display screen of the controlled device.

[0270] Optionally, the identifiable data includes contour data of the controlled device, and / or data used to locate the position of the controlled device in the captured image.

[0271] For example, the captured image obtained by the control device includes the image of the controlled device and the image outside the controlled device's image. Changes in the direction the control device points towards the area where the controlled device is located will cause changes in the positional relationship between the controlled device's image and the captured image. Therefore, by confirming the positional relationship between the controlled device's image and the captured image, the pointing position of the control device can be obtained. Optionally, the controlled device's image included in the captured image can be identified using the controlled device's identifying data.

[0272] Optionally, the identifying data may include data generated by an external flashing light source configured to capture images of the controlled device, data generated by capturing the flashing feature image displayed by the controlled device, or data generated by capturing identifying objects on the peripherals of the controlled device. For example, contour data may be generated by capturing images of an external flashing light source configured to capture images of the controlled device, which is positioned at the outer edge of the controlled device, thereby identifying the contour of the controlled device. Another example is data used to locate the position of the controlled device in the captured image, such as data generated by capturing the flashing feature image displayed by the controlled device.

[0273] For example, such as Figure 6A As shown, by configuring a flashing light source outside the controlled device (such as a smart screen), the outline data of the controlled device can be included in the captured image, thereby confirming the positional relationship between the controlled device image and the captured image.

[0274] For example, such as Figure 8 As shown, the controlled device (such as a smart screen) displays a feature image. In some examples, this feature image is used to identify the outline of the controlled device; for example, the controlled device displays the feature image at the edge of the display screen. Thus, by displaying the feature image, the outline data of the controlled device is included in the captured image, thereby confirming the positional relationship between the controlled device image and the captured image. In other examples, the feature image is used to locate the position of the controlled device in the captured image; for example, the feature image carries position data, which includes data indicating the positional relationship between the feature image and the controlled device. Thus, by displaying the feature image, the controlled device image is located in the captured image. Optionally, the controlled device displays the feature image at a preset position on the display screen, or the feature image is displayed in full screen.

[0275] As another example, the controlled device carries location data via external stickers or other means. This location data includes information indicating the positional relationship between the stickers / other items and the controlled device. In this way, this location data can be acquired by capturing an image, thereby enabling the location of the controlled device within the captured image.

[0276] As another example, the controlled device is equipped with a display function. In this case, the captured image will also include data generated from capturing the display screen of the controlled device, so that the controlled device's image can be located in the captured image based on this data.

[0277] In this way, by using the identifying data of the controlled device or the displayed screen, the corresponding data of the controlled device can be identified in the image captured towards the controlled device, and then the pointing position of the control device can be determined.

[0278] In some embodiments, the camera includes an EVS camera. The control device acquires a captured image via the camera, including: generating an EVS event via the EVS camera, wherein the EVS event is generated based on a light source configured for the controlled device and / or a characteristic image displayed by the controlled device, and the EVS event is used to generate the captured image.

[0279] Optionally, the light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

[0280] In this way, the EVS camera allows the control device to acquire the captured image, which facilitates the subsequent determination of the pointing position based on the captured image.

[0281] In some examples, the EVS events captured by the control device via the EVS camera may exhibit anomalies such as noise, irrelevant variations, and influence from the intrinsic parameters calibrated by the EVS camera. Therefore, the control device can perform filtering and / or lens parameter correction on the EVS events to generate the captured image.

[0282] In this way, by processing the EVS event, we can output images of acceptable quality, thereby ensuring that we can obtain the correct pointing position of the control device later.

[0283] In other examples, after receiving an EVS event, the control device may also send the EVS event to a parsing device or a controlled device, which will then process the EVS event and acquire the captured image. That is, the embodiments of this application do not limit the electronic device performing the data processing procedure.

[0284] In other embodiments, the camera includes an APS camera. Optionally, the image captured by the control device through the APS camera includes data generated from capturing the display screen of the controlled device.

[0285] In some other embodiments, the camera includes EVS camera functionality and APS camera functionality, and the control device acquires the captured image through the camera, including: the control device acquires the captured image through the cooperation of the camera's EVS camera functionality and APS camera functionality.

[0286] For example, the captured image is a first captured image obtained through the APS camera function of the camera, or a second captured image obtained through the EVS camera function of the camera. The control device acquires the first captured image through the APS camera function at preset intervals, and the first captured image is used to determine a first pointing position of the control device. During the intervals of the preset period, the control device acquires the second captured image through the EVS camera function of the camera. Based on the second captured image, the control device obtains the relative displacement of the change in the first pointing position. Based on the relative displacement, the control device obtains the second pointing position. The first and second pointing positions can be used by the controlled device to subsequently display the first element at the corresponding display position.

[0287] In this way, by using the APS camera function and the EVS camera function to acquire shooting images in a preset cycle, the power consumption of using the APS camera function can be reduced, and the shooting images can be effectively output to determine the pointing position of the control device.

[0288] For another example, the control device uses an APS camera to take a picture of the area where the controlled device is located, and then captures the image. After determining the identity of the controlled device based on the captured image, the control device can then acquire subsequent images using an EVS camera to save power consumption.

[0289] For another example, the control device uses an APS camera to take a picture of the area where the controlled device is located, acquiring the captured image. Then, based on this captured image, the control device identifies the controlled device. Subsequently, if the controlled device is a smart device with a display function (such as a smart screen), the control device can subsequently acquire the captured image using an EVS camera. If the controlled device is a smart device without a display function (such as an air conditioner), the control device can subsequently acquire the captured image using an APS camera.

[0290] Thus, when the camera of the control device includes multiple camera functions, the control device can flexibly use multiple camera functions in combination to acquire shooting images.

[0291] Optionally, the above description provides examples of possible camera implementations. It should be understood that the camera in the control device can also be implemented as other types. For example, the camera can also be an RGB camera, etc.

[0292] S1702, The control device sends the captured image to the analysis device.

[0293] In some embodiments, after acquiring the captured image, the control device can send the captured image to the analysis device for processing, so as to determine the relevant information of the controlled device.

[0294] S1703. The analysis device determines the electronic device that needs to perform user interaction events as the controlled device by the features in the captured image.

[0295] User interaction events include interaction events where the control device points to the controlled device.

[0296] For example, a user holds a control device and points it at the controlled device, intending to interact with the controlled device. For instance, controlling the focus movement of the controlled device's display. Or, for example, controlling the controlled device to start or stop.

[0297] In some embodiments, the captured image includes identifying data of the controlled device captured using a visual sensor and / or data generated from capturing the display screen of the controlled device. These data constitute features in the captured image, used to identify the identity of any electronic device that may be captured and requires user interaction events. For example, the parsing device may currently identify the electronic device as a controlled device based on these features.

[0298] For example, the parsing device identifies the controlled device by recognizing contour data in the captured image.

[0299] For example, the controlled device may be equipped with a display function, such as a smart screen. The analysis device can identify the controlled device by recognizing characteristic images in the captured image.

[0300] In this way, the analysis device can identify the device that needs to perform user interaction events based on the captured image, and then perform user interaction events based on the identified device.

[0301] In some embodiments, after the parsing device determines that the electronic device that needs to perform the user interaction event is the controlled device by the features in the captured image, the parsing device obtains the display position on the controlled device corresponding to the pointing position of the control device based on the positional relationship between the identification data and the captured image, or based on the data generated by capturing the display image of the controlled device.

[0302] For example, after the analysis device confirms the controlled device captured by the control device, it can determine the positional relationship between the controlled device image and the captured image based on the identifying data in the captured image. Then, based on this positional relationship, it can perform coordinate transformation on the center position of the captured image to determine the pointing position of the control device.

[0303] For example, if the controlled device has a display function, the analysis device can also determine the positional relationship between the controlled device's image and the captured image based on the data generated by the control device capturing the image of the controlled device's display. Based on this positional relationship, the center position of the captured image can be transformed to determine the pointing position of the control device.

[0304] For example, the analysis device compares feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image. Based on the first position, the analysis device obtains the positional relationship between the image of the area where the controlled device is located and the captured image.

[0305] In this way, by comparing feature points, the amount of data processed can be reduced while processing the data generated by the captured and displayed images, thereby improving efficiency.

[0306] Optionally, the pointing position of the control device is the user's intended interaction position on the controlled device. Optionally, the controlled device displays the corresponding element at the display position corresponding to the intended interaction position, which can help the user understand the current pointing position of the control device.

[0307] In this way, after identifying the controlled device, the parsing device can obtain the display position on the controlled device corresponding to the pointing position of the control device, which makes it easier to trigger the controlled device to display the corresponding element at that display position.

[0308] S1704. The parsing device sends relevant information about the controlled device to the control device.

[0309] The relevant information of the controlled device includes, for example, the identification of the controlled device and / or the display position of the pointing position of the control device on the controlled device.

[0310] In some embodiments, after the parsing device identifies relevant information about the controlled device based on the captured image, it can send the relevant information to the control device to trigger the control device to control the controlled device.

[0311] S1705, The control device sends instruction information to the controlled device.

[0312] In some embodiments, after receiving relevant information about the controlled device sent by the parsing device, the control device can generate corresponding instruction information based on the relevant information to control the controlled device.

[0313] For example, the indication information is a display indication, which carries the display position and is used to instruct the controlled device to display the corresponding display element at the display position.

[0314] For example, the indication information is a connection indication, which is used to indicate that a communication connection is established between the controlled device and the control device, so that the control device can control the controlled device through the communication connection.

[0315] S1706, The controlled device performs the operation indicated by the user interaction event.

[0316] In some embodiments, after receiving an instruction, the controlled device can perform a corresponding operation based on the instruction.

[0317] In some examples, the operation of executing the user interaction event indication on the controlled device includes: displaying a first element on the controlled device at a display position corresponding to the pointed position in response to the pointing position on the control device. And / or, the user interaction event also includes: a selection operation on a control component on the control device; the operation of executing the user interaction event indication on the controlled device includes: triggering the controlled device to perform an operation in response to the selection operation.

[0318] Optionally, control components on the control device may include controls displayed on the control device and / or physical buttons on the control device.

[0319] Optionally, during the execution of the user interaction event indication operation on the controlled device, the control device displays a control interface, which is used to receive user operations to control the controlled device.

[0320] For example, the control device acquires relevant information about the controlled device, including the controlled device's identity information and display position. The identity information, for example, is information indicating the controlled device, used to enable the control device to send instruction information to the controlled device. Optionally, the instruction information carries the display position, used to instruct the controlled device to display the first element at that display position. For example, such as... Figure 6B As shown in (a), the parsing device determines the controlled device to be a smart screen based on the captured image, and performs coordinate transformation based on the positional relationship between the smart screen image and the captured image to obtain the display position corresponding to the position pointed to by the control device. Then, the control device can send an indication message carrying that display position to the smart screen. Thus, as shown in (a), Figure 6B As shown in (b), the smart screen can display cursor 61 at the display position according to the instruction information.

[0321] For example, the control device is equipped with a display screen. After establishing a communication connection with the controlled device, the control device can display the control interface of the controlled device on the display screen. In this way, the control device can receive user interaction operations on the controlled device through this control interface, such as selecting a control, thereby providing the user with a more flexible and richer remote control experience. For example, such as... Figure 14As shown in (a), after processing the captured image and determining that the smart device is a smart light, the control device (such as a remote control) can display the following: Figure 15 The control interface 151 shown corresponds to the smart light. The control device can then generate corresponding instruction information based on the user's operation of the controls displayed on the control interface 151, and send this instruction information to the smart light to control it.

[0322] For another example, during the process of the controlled device displaying the first element, the user moves to the display position that needs to be operated based on the first element on the controlled device. For example, after moving to a certain control displayed on the controlled device, the user can trigger the control device to instruct the controlled device to operate the control by operating the physical button of the control device.

[0323] In this way, by identifying the controlled device through the captured images, the control device can achieve more flexible control over the controlled device.

[0324] In other examples, after the parsing device determines that the electronic device that needs to perform the user interaction event is the controlled device by the features in the captured image, the method further includes: the control device establishing a communication connection with the controlled device based on the determined controlled device, so as to perform the user interaction event.

[0325] For example, the control device can also identify the controlled device by capturing images through a camera. Once the controlled device is identified, the control device can establish a wireless communication connection with the controlled device to achieve remote control of the controlled device.

[0326] Optionally, after identifying the controlled device, the control device can select a communication protocol compatible with that controlled device. Then, the control device can establish a communication connection with the controlled device through this protocol. Subsequently, in response to the user's remote control operation, the control device can send corresponding instruction information to the controlled device to trigger it to perform the corresponding operation. In this way, remote control operations such as switching on and off and changing modes of the controlled device can be achieved, enabling "point-and-click control."

[0327] Thus, by identifying the controlled device through image capture, the remote control device automatically establishes a wireless communication connection with the controlled device, enabling remote control of the controlled device. Compared to solutions where the user needs to pair and connect the control device with the controlled device before controlling the controlled device, the remote control method provided in this application effectively simplifies user operation and improves the user experience.

[0328] Based on the above, it can be seen that the controlled device can determine its pointing position through images captured by a camera, thereby enabling directional control of the device. Compared to configuring separate wireless positioning modules in both the control and controlled devices, relying solely on image analysis to determine the pointing position effectively reduces hardware costs.

[0329] Furthermore, it eliminates the need for complete hardware installation and adjustment of the control and controlled equipment, increasing the flexibility of directional control.

[0330] In some embodiments, the control device acquires a first captured image and a second captured image sequentially. The first captured image includes a view taken from the area where the first controlled device is located, and the second captured image includes a view taken from the area where the second controlled device is located. Therefore, the control device can control the first and second controlled devices to operate in conjunction.

[0331] Optionally, the control device can serve as a relay device between at least two controlled devices continuously pointed to by the user.

[0332] For example, such as Figure 16 As shown, the control device (such as a remote control) responds to the user's pointing operation by capturing an image through a camera. Then, by processing the captured image, it acquires the image of the smart device included within it. Through recognition of the smart device image, it identifies the currently pointed controlled device as a PC and establishes a wireless communication connection with the PC. If, after detecting the user's pointing operation at the PC, the control device detects the user pointing in other directions, it can determine that it has detected continuous pointing operations from the user. The control device captures an image through a camera. Then, by processing the captured image, it acquires the image of the controlled device included within it. Through recognition of the controlled device image, it identifies the currently pointed controlled device as a smart screen and establishes a wireless communication connection with the smart screen. The control device can then send a data relay instruction to the PC and / or the smart screen. In response to this data relay instruction, the PC can send image data to the control device. After receiving the image data, the control device can forward the image data to the smart screen, thereby realizing screen projection from the PC to the smart screen.

[0333] In this way, the control device can recognize the user's continuous pointing operations to realize the functional linkage between multiple controlled devices, enriching the user's remote control interaction experience.

[0334] Optionally, the control device can also execute the steps and functions performed by the first electronic device (such as a remote control device) in the above embodiments, and the controlled device can also execute the steps and functions performed by the second electronic device (such as a remotely controlled smart device) in the above embodiments, thereby realizing the remote control method provided in the above embodiments.

[0335] In some embodiments, the parsing device is a control device, or the parsing device is a controlled device, or the parsing device is a device other than a control device and a controlled device.

[0336] For example, the parsing device can be a control device. For instance, a remote control device can be both a control device and a parsing device. Then, the control device can execute steps S1701-S1704 to determine the controlled device based on the acquired captured image.

[0337] For example, the control device acquires a captured image via a camera, the captured image including a view directed towards the area where the controlled device is located. The control device determines, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device. In response to the user interaction event, the control device instructs the controlled device to perform the operation indicated by the user interaction event. The user interaction event includes an interaction event where the control device points at the controlled device.

[0338] For example, the parsing device can be the controlled device. For instance, a remote control device can be the controlling device, and a smart screen can be either the controlled device or the parsing device. In this case, the smart screen can execute steps S1702-S1704 to determine that the controlled device is the smart screen based on the received captured image.

[0339] For example, the parsing device receives a captured image from a camera sent by the control device. The captured image includes a view taken towards the area where the controlled device is located. The parsing device determines, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device. Since the controlled device is the parsing device, in response to a user interaction event, the parsing device executes the operation indicated by the user interaction event; wherein, the user interaction event includes an interaction event where the control device points to the controlled device.

[0340] For example, the parsing device could be any device other than the control device and the controlled device. For instance, a remote control could act as the control device, a smart screen as the parsing device, and an air conditioner as the controlled device. Then, the control device and the parsing device would execute steps S1701-S1704 to determine the controlled device as the air conditioner based on the acquired image, enabling the remote control device to control the air conditioner subsequently.

[0341] For example, the parsing device receives a captured image from a camera sent by the control device. The captured image includes a view taken towards the area where the controlled device is located. The parsing device determines, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device. Since the controlled device is not the parsing device itself, the parsing device sends information to the control device to indicate the controlled device. This information may include, for example, information indicating the identity of the controlled device.

[0342] In this way, each electronic device can perform corresponding operations as different roles based on factors such as device capabilities and interaction scenarios, thereby providing users with a remote pointing and control experience.

[0343] Optionally, if the parsing device is a control device, a controlled device, or a device other than a control device or a controlled device, the steps and functions to be performed can be referred to the relevant content of steps S1701-S1706 above, and will not be repeated here.

[0344] In some solutions, multiple embodiments of this application can be combined, and the combined solution can be implemented. Optionally, some operations in the processes of each method embodiment may be combined, and / or the order of some operations may be changed. Furthermore, the execution order between the steps of each process is merely exemplary and does not constitute a limitation on the execution order between steps; other execution orders are also possible. It is not intended to indicate that the execution order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. In addition, it should be noted that the process details involved in one embodiment of this document are similarly applicable to other embodiments, or different embodiments may be combined.

[0345] Furthermore, some steps in the method embodiments can be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and can be deleted in certain use cases. Or, other possible steps may be added to the method embodiments.

[0346] Furthermore, the various method embodiments can be implemented individually or in combination.

[0347] The above combination Figures 5-17 The remote control method provided in the embodiments of this application is described in detail below. Figure 18 Detailed description of the control device provided in the embodiments of this application, and in conjunction with Figure 19 The parsing device provided in the embodiments of this application is described in detail.

[0348] In one possible design, Figure 18 This is a schematic diagram of the structure of the control device provided in an embodiment of this application. Figure 18 As shown, the control device 1800 may include a processing unit 1801, a transceiver unit 1802, and a shooting unit 1803. The control device 1800 can be used to implement the functions of the first electronic device (such as a remote control device) involved in the above method embodiments. Optionally, the control device 1800 can also be used to implement the functions of the parsing device involved in the above method embodiments.

[0349] Optionally, the processing unit 1801 is used to support the control device 1800 in performing operations. Figure 17 S1701 in the middle.

[0350] Optionally, the transceiver unit 1802 is used to support the control device 1800 in performing operations. Figure 17 S1702, S1704, and S1705.

[0351] Optionally, the shooting unit 1803 is used to support the control device 1800 in performing actions. Figure 17 S1701 in the middle.

[0352] The transceiver unit may include a receiving unit and a transmitting unit, and may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver module. The operation and / or function of each unit in the control device 1800 are respectively to implement the corresponding process of the remote control method described in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit, and for the sake of brevity, it will not be repeated here.

[0353] Optionally, Figure 18 The control device 1800 shown may also include a storage unit ( Figure 18 (not shown in the image), this storage unit stores a program or instruction. When the processing unit 1801, the transceiver unit 1802, and the imaging unit 1803 execute the program or instruction, it causes... Figure 18 The control device 1800 shown can perform the remote control method described in the above method embodiments.

[0354] Figure 18 The technical effects of the control device 1800 shown can be referred to the technical effects of the remote control method described in the above method embodiments, and will not be repeated here.

[0355] In addition to being in the form of a control device 1800, the technical solution provided in this application may also be a functional unit or chip in a control device, or a device used in conjunction with a control device.

[0356] In one possible design, Figure 19 This is a schematic diagram of the analytical device provided in an embodiment of this application. Figure 19 As shown, the parsing device 1900 may include a processing unit 1901 and a transceiver unit 1902. The parsing device 1900 can be used to implement the relevant functions of the first electronic device (such as a remote control device) or the second electronic device (such as a remotely controlled smart device) involved in the above method embodiments.

[0357] Optionally, the processing unit 1901 is used to support the parsing device 1900 in performing operations. Figure 17 S1703 in the middle.

[0358] Optionally, the transceiver unit 1902 is used to support the parsing device 1900 in performing operations. Figure 17 S1702 and S1704 in the example.

[0359] Optionally, the parsing device 1900 may further include a display unit 1903. Optionally, the parsing device 1900 may be, for example, a controlled device, and the display unit 1903 is used to support the parsing device 1900 in performing operations. Figure 17 In S1706, the corresponding display element is displayed.

[0360] The transceiver unit may include a receiving unit and a transmitting unit, and may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver module. The operation and / or function of each unit in the parsing device 1900 are respectively for implementing the corresponding process of the remote control method described in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit, and will not be repeated here for the sake of brevity.

[0361] Optionally, Figure 19 The parsing device 1900 shown may also include a storage unit ( Figure 19 (not shown in the image), this storage unit stores a program or instruction. When the processing unit 1901 and the transceiver unit 1902 execute the program or instruction, it causes... Figure 19 The analytical device 1900 shown can execute the remote control method described in the above method embodiments.

[0362] Figure 19 The technical effects of the analytical device 1900 shown can be referred to the technical effects of the remote control method described in the above method embodiments, and will not be repeated here.

[0363] In addition to being in the form of a parsing device 1900, the technical solution provided in this application may also be a functional unit or chip in a parsing device, or a device used in conjunction with a parsing device.

[0364] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0365] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0366] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0367] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0368] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0369] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it causes the computer to perform the aforementioned steps to implement the remote control method described in the above embodiments.

[0370] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the remote control method described in the above embodiments.

[0371] In addition, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory stores a computer program. When the computer program is executed by one or more processors, the apparatus performs the remote control methods described in the above-described method embodiments.

[0372] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0373] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0374] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0375] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of modules or units may be electrical, mechanical or other forms.

[0376] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0377] Computer-readable storage media include, but are not limited to, any of the following: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media capable of storing program code.

[0378] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A remote control method, characterized in that, The method includes: The control device acquires images through a camera, including images captured from the area where the controlled device is located. The analysis device identifies the electronic device that needs to perform a user interaction event as the controlled device based on the features in the captured image. In response to a user interaction event, the operation indicated by the user interaction event is performed on the controlled device; wherein the user interaction event includes an interaction event in which the control device points to the controlled device.

2. The method according to claim 1, characterized in that, The operation of executing the user interaction event indication on the controlled device includes: in response to the pointing position of the control device, displaying a first element on the controlled device at a display position corresponding to the pointing position; And / or, The user interaction event further includes: a selection operation on a control component on the control device; the operation indicated by the user interaction event on the controlled device includes: in response to the selection operation, triggering the controlled device to perform an operation.

3. The method according to claim 1 or 2, characterized in that, The camera includes a vision sensor, and the captured images include identifying data of the controlled device captured using the vision sensor and / or data generated from capturing the display images of the controlled device.

4. The method according to claim 3, characterized in that, The identifying data includes the outline data of the controlled device, and / or data for locating the position of the controlled device in the captured image.

5. The method according to claim 3 or 4, characterized in that, After the parsing device determines, through features in the captured image, that the electronic device requiring a user interaction event is the controlled device, the method further includes: The analysis device obtains the display position on the controlled device corresponding to the pointing position of the control device based on the positional relationship between the identification data and the captured image, or based on the data generated from capturing the display image of the controlled device.

6. The method according to any one of claims 1-5, characterized in that, After the parsing device determines, through features in the captured image, that the electronic device requiring a user interaction event is the controlled device, the method further includes: The control device establishes a communication connection with the controlled device based on the determined controlled device, in order to execute the user interaction event.

7. The method according to any one of claims 1-6, characterized in that, The camera includes an event-based visual sensor, the EVS camera. The control device acquires captured images via a camera, including: The control device generates EVS events through the EVS camera. The EVS events are generated based on the light source configured by the controlled device and / or the characteristic images displayed by the controlled device. The EVS events are used to generate the captured images.

8. The method according to claim 7, characterized in that, The light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

9. The method according to claim 7 or 8, characterized in that, The method further includes: The controlled device displays the feature image at a preset position on the display screen, or displays the feature image in full screen.

10. The method according to any one of claims 1-6, characterized in that, The camera includes an active pixel sensor (APS) camera.

11. The method according to claim 10, characterized in that, The analysis device determines the electronic device that needs to perform a user interaction event as the controlled device based on features in the captured image, including: The analysis device compares the feature points in the display screen of the controlled device with the captured image to obtain the first position of the image corresponding to the display screen of the controlled device in the captured image; The analysis device obtains the positional relationship between the image of the area where the controlled device is located and the captured image based on the first position.

12. The method according to any one of claims 1-11, characterized in that, The camera includes EVS camera functionality and APS camera functionality. The control device acquires the captured image through the camera, including: The control device acquires the captured image through the cooperation of the EVS camera function and the APS camera function of the camera.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: During the execution of the user interaction event indicated on the controlled device, the control device displays a control interface, which is used to receive user operations to control the controlled device.

14. The method according to any one of claims 1-13, characterized in that, The method further includes: The control device acquires a first captured image and a second captured image in sequence. The first captured image includes an image captured towards the area where the first controlled device is located, and the second captured image includes an image captured towards the area where the second controlled device is located. The control device controls the first controlled device and the second controlled device to work together.

15. The method according to any one of claims 1-14, characterized in that, The parsing device is the control device, or the parsing device is the controlled device, or the parsing device is a device other than the control device and the controlled device.

16. A remote control method, characterized in that, The method is applied to a control device, and the method includes: The camera captures images, including images captured from the area where the controlled device is located. The electronic device that needs to perform the user interaction event is identified as the controlled device based on the features in the captured image. In response to a user interaction event, the controlled device is instructed to perform the operation indicated by the user interaction event; wherein the user interaction event includes an interaction event in which the control device points to the controlled device.

17. The method according to claim 16, characterized in that, The operation of instructing the controlled device to execute the user interaction event instruction includes: in response to the pointing position of the control device, instructing the controlled device to display a first element at the display position on the controlled device corresponding to the pointing position; And / or, The user interaction event further includes: a selection operation on a control component on the control device; the operation of instructing the controlled device to perform the user interaction event instruction includes: in response to the selection operation, triggering the controlled device to perform the operation.

18. The method according to claim 16 or 17, characterized in that, The camera includes a vision sensor, and the captured images include identifying data of the controlled device captured using the vision sensor and / or data generated from capturing the display images of the controlled device.

19. The method according to claim 18, characterized in that, The identifying data includes the outline data of the controlled device, and / or data for locating the position of the controlled device in the captured image.

20. The method according to claim 18 or 19, characterized in that, After determining, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device, the method further includes: Based on the positional relationship between the identification data and the captured image, or based on the data generated from capturing the display image of the controlled device, the display position of the pointing position of the control device on the controlled device is obtained.

21. The method according to any one of claims 16-20, characterized in that, After determining, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device, the method further includes: Based on the identified controlled device, a communication connection is established with the controlled device to execute the user interaction event.

22. The method according to any one of claims 16-21, characterized in that, The camera includes an event-based visual sensor, the EVS camera. The process of acquiring the captured image via a camera includes: The EVS camera generates an EVS event, which is generated based on the light source configured by the controlled device and / or the characteristic image displayed by the controlled device. The EVS event is used to generate the captured image.

23. The method according to claim 22, characterized in that, The light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

24. The method according to any one of claims 16-21, characterized in that, The camera includes an active pixel sensor (APS) camera.

25. The method according to claim 24, characterized in that, The process of determining the electronic device that needs to perform a user interaction event as the controlled device based on features in the captured image includes: By comparing the feature points in the display screen of the controlled device with the captured image, the first position of the image corresponding to the display screen of the controlled device in the captured image is obtained; Based on the first position, the positional relationship between the image of the area where the controlled device is located and the captured image is obtained.

26. The method according to any one of claims 16-25, characterized in that, The camera includes EVS camera functionality and APS camera functionality. The process of acquiring the captured image via the camera includes: The captured image is obtained by combining the EVS camera function and the APS camera function of the camera.

27. The method according to any one of claims 16-26, characterized in that, The method further includes: The display control interface is used to receive user operations to control the controlled device.

28. The method according to any one of claims 16-27, characterized in that, The method further includes: The system acquires a first captured image and a second captured image in sequence. The first captured image includes an image captured towards the area where the first controlled device is located, and the second captured image includes an image captured towards the area where the second controlled device is located. Control the linkage between the first controlled device and the second controlled device.

29. A remote control method, characterized in that, The method is applied to a parsing device, and the method includes: The control device receives images captured by a camera, including images captured from the area where the controlled device is located. The electronic device that needs to perform the user interaction event is identified as the controlled device based on the features in the captured image. Based on the fact that the controlled device is the parsing device, in response to a user interaction event, the operation indicated by the user interaction event is executed; wherein, the user interaction event includes an interaction event in which the control device points to the controlled device; or, Based on the fact that the controlled device is a controlled device other than the parsing device, information for instructing the controlled device is sent to the control device.

30. The method according to claim 29, characterized in that, The operation of executing the user interaction event indication includes: in response to the pointing position of the control device, displaying a first element at a display position on the controlled device corresponding to the pointing position; And / or, The user interaction event further includes: a selection operation on a control component on the control device; the operation of executing the user interaction event instruction includes: executing the operation corresponding to the selection operation according to the selection operation.

31. The method according to claim 29 or 30, characterized in that, The camera includes a vision sensor, and the captured images include identifying data of the controlled device captured using the vision sensor and / or data generated from capturing the display images of the controlled device.

32. The method according to claim 31, characterized in that, The identifying data includes the outline data of the controlled device, and / or data for locating the position of the controlled device in the captured image.

33. The method according to claim 31 or 32, characterized in that, After determining, based on features in the captured image, that the electronic device requiring a user interaction event is the controlled device, the method further includes: Based on the positional relationship between the identification data and the captured image, or based on the data generated from capturing the display image of the controlled device, the display position of the pointing position of the control device on the controlled device is obtained.

34. The method according to any one of claims 29-33, characterized in that, The information sent to the control device to instruct the controlled device is used by the control device to establish a communication connection with the controlled device based on the determined controlled device, so as to execute the user interaction event.

35. The method according to any one of claims 29-34, characterized in that, The camera includes an event-based visual sensor, the EVS camera. The receiving control device sends the captured image obtained by the control device through the camera, including: The control device receives an EVS event generated by the EVS camera, which is sent by the control device. The EVS event is generated based on the light source configured by the controlled device and / or the feature image displayed by the controlled device. The EVS event is used to generate the captured image.

36. The method according to claim 35, characterized in that, The light source flashes at a first frequency, and / or the feature image flashes at a second frequency.

37. The method according to claim 35 or 36, characterized in that, The method further includes: Depending on whether the controlled device is the parsing device, the feature image is displayed at a preset position on the display screen, or the feature image is displayed in full screen.

38. The method according to any one of claims 29-34, characterized in that, The camera includes an active pixel sensor (APS) camera.

39. The method according to claim 38, characterized in that, The process of determining the electronic device that needs to perform a user interaction event as the controlled device based on features in the captured image includes: By comparing the feature points in the display screen of the controlled device with the captured image, the first position of the image corresponding to the display screen of the controlled device in the captured image is obtained; Based on the first position, the positional relationship between the image of the area where the controlled device is located and the captured image is obtained.

40. The method according to any one of claims 29-39, characterized in that, The camera includes EVS camera function and APS camera function, which are used together to acquire the captured image.

41. An electronic device, characterized in that, include: The electronic device includes a processor, a memory, and a camera, wherein the memory and the camera are coupled to the processor, the memory is used to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, causes the electronic device to perform the method as described in any one of claims 16-28.

42. The method according to claim 41, characterized in that, The camera includes an event-based visual sensor (EVS) camera; or, the camera includes an active pixel sensor (APS) camera; or, the camera includes both EVS camera functionality and APS camera functionality.

43. An electronic device, characterized in that, include: The electronic device includes a processor, a memory, and a display screen, wherein the memory and the display screen are coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, and the electronic device performing the method as described in any one of claims 29-40 when the processor reads the computer instructions from the memory.

44. An electronic device, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, which, when the processor reads the computer instructions from the memory, cause the electronic device to perform the method as described in any one of claims 29-40.

45. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 16-28; or causes the electronic device to perform the method as described in any one of claims 29-40.

46. ​​A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 16-28; or, causes the computer to perform the method as described in any one of claims 29-40.

Images