Device control method, device, user device and computer readable storage medium
By displaying real scene images on the user's device and superimposing the control sub-interface of the connected device, users can quickly identify and control devices in the scene, solving the problem of complex device connection operations and improving the user experience.
Patent Information
- Application Number
- CN202210084628.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-01-25
AI Technical Summary
In multiple connected smart device environments, when users need to temporarily connect the device, it is cumbersome to query the device name, especially in unfamiliar scenarios.
The image acquisition instructions display the real scene image and superimpose the control subinterface of the connected device on the image. The user can directly control the target device through triggering operations, simplifying the connection process.
Users can quickly identify and control the devices that can be connected in the scenario, reducing the complexity and operation steps of device connection, and improving the user experience without knowing the device name in advance.
Smart Images

Figure CN114422644B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a device control method, apparatus, user equipment and computer-readable storage medium. Background Art
[0002] Currently, when users use Bluetooth speakers, smart TVs and other devices, they can search for the names of connectable devices on the device search interface of their mobile phones and then connect to them.
[0003] However, as more and more smart devices are available for connection, users may need to temporarily connect some devices. For example, when a user comes to a relatively unfamiliar environment (such as a hotel), he or she needs to connect each device in the environment by consulting the instructions without knowing the names of the connectable devices in the environment, which makes the user's device connection operation more cumbersome. Summary of the invention
[0004] The embodiments of the present application provide a device control method, apparatus, user equipment, and computer-readable storage medium, which can reduce the complexity of device control.
[0005] In a first aspect, a device control method includes:
[0006] In response to the image acquisition instruction, a first interface is displayed; the first interface includes a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image;
[0007] A trigger operation is obtained based on the control sub-interface, and a target connectable device is controlled according to the trigger operation.
[0008] In a second aspect, a device control apparatus includes:
[0009] A display module, configured to display a first interface in response to an image acquisition instruction; the first interface includes a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image;
[0010] The control module is used to obtain a trigger operation based on the control sub-interface and control the target connectable device according to the trigger operation.
[0011] In a third aspect, a user device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned device control method.
[0012] In a fourth aspect, a computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned device control method when executed by a processor.
[0013] In a fifth aspect, a computer program product includes a computer program, which implements the steps of the above-mentioned device control method when executed by a processor.
[0014] In the above-mentioned device control method, apparatus, user device and computer-readable storage medium, the user device responds to the image acquisition instruction and displays a first interface including a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image; then, a trigger operation is obtained based on the control sub-interface, and the target connectable device is controlled according to the trigger operation. Since the first interface displayed by the user device includes the display scene image and the connectable device superimposed on the display scene image, the user can quickly perceive which connectable devices are in the current scene through the first interface after acquiring the real scene image; further, since the first interface includes the control sub-interface of the connectable device, the user can complete the control of the target connectable device by executing a trigger operation on the control sub-interface of the target connectable device, thereby simplifying the control operation of the connectable device. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 This is an application environment diagram of a device control method in one embodiment of the present application.
[0017] Figure 2 This is a flow chart of a device control method in one embodiment of the present application.
[0018] Figure 3 This is a schematic diagram of the first interface in one embodiment of the present application.
[0019] Figure 4 This is a schematic diagram of the first interface in one embodiment of the present application.
[0020] Figure 5 This is a schematic diagram of a control sub-interface in one embodiment of the present application.
[0021] Figure 6 This is a schematic diagram of the second operation sub-interface in one embodiment of the present application.
[0022] Figure 7 This is a schematic diagram of a control sub-interface in one embodiment of the present application.
[0023] Figure 8This is a flow chart of a device control method in one embodiment of the present application.
[0024] Fig. 9 This is a flow chart of a device control method in one embodiment of the present application.
[0025] Fig.10 This is a flow chart of a device control method in one embodiment of the present application.
[0026] Fig.11 This is a schematic diagram of a device control method in one embodiment of the present application.
[0027] Fig.12 This is a schematic diagram of a device control method in one embodiment of the present application.
[0028] Fig.13 This is a flow chart of a device control method in one embodiment of the present application.
[0029] Fig.14 This is a schematic diagram of a device control method in one embodiment of the present application.
[0030] Fig.15 This is a schematic diagram of a device control method in one embodiment of the present application.
[0031] Fig.16 This is a schematic diagram of a device control method in one embodiment of the present application.
[0032] Fig.17 This is a flow chart of a device control method in one embodiment of the present application.
[0033] Fig.18 This is a structural block diagram of a device control device in one embodiment of the present application.
[0034] Fig.19 This is a structural block diagram of a device control device in one embodiment of the present application.
[0035] Fig. 20 This is a schematic diagram of the structure of a user device in one embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0037] The device control method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. The user device 100 can communicate and connect with the connectable device 200 in the current scene through the network. The user device 100 can be various personal computers, laptops, smart phones, tablet computers and portable wearable devices; the portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The connectable device 200 can be an IoT device such as a smart speaker, a smart TV, a smart air conditioner, and a smart car device.
[0038] In one embodiment, Figure 2 As shown, a device control method is provided, which is applied to Figure 1 The user equipment 100 in FIG. 1 is taken as an example for description, including:
[0039] S102. In response to an image acquisition instruction, display a first interface; the first interface includes a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image.
[0040] The image acquisition instruction may be an instruction triggered by a user on a camera operation control of a user device. The camera operation control may be a physical control on a user device or a virtual control displayed on a display screen of the user device, which is not limited here. The camera operation control may be a control set in an application of the user device or a control set in an operation window of the user device, which is not limited here. For example, the camera operation control may be a control set in an application of the user device for connecting to an external device.
[0041] After the user sends the above-mentioned image acquisition instruction, the user device can capture the real scene image within the camera's field of view, and superimpose the control sub-interface of the connectable device in the real scene on the display scene image, generate the first interface and then display it to the user.
[0042] The above-mentioned connectable device can be an IoT device set in the current scene, such as a smart speaker, smart TV, smart air conditioner, smart desk lamp, smart curtain, etc.; the above-mentioned IoT device can also be an on-board device in a smart car, a car system, etc., and the type of IoT device is not limited here. The above-mentioned connectable device can establish a communication connection with the user device, and the above-mentioned communication connection can be a Bluetooth connection, a cellular network connection, or a WiFi connection, which is not limited here.
[0043] The control sub-interface can be used to control the connectable device. The control sub-interface can include controls for controlling the connectable device, and can also include a remaining power indicator of the connectable device, a network connection status indicator, the current working status of the connectable device, etc.; the display method of the control sub-interface is not limited here. It should be noted that different types of connectable devices may correspond to different control sub-interfaces.
[0044] The user device can receive broadcast messages sent by each connectable device, and determine which connectable devices are in the current scene based on the broadcast messages; further, the user device can determine the corresponding control sub-interface based on the content carried in the broadcast message, and then overlay the control sub-interface of the connectable device on the real scene image captured by the camera to generate and display the first interface. The user device can overlay the control sub-interfaces of all connectable devices in the current scene on the real scene image, or it can display the control sub-interfaces of some connectable devices, which is not limited here. In one implementation, the control sub-interface displayed on the above-mentioned first interface can be the control sub-interface corresponding to the connectable device within the field of view of the camera.
[0045] The above-mentioned real scene image can be a static image collected and stored by the user device, or a preview image displayed in the image acquisition window of the user device, which is not limited here. For example, after the user enters the current scene, the image acquisition instruction can be triggered to display the first interface on the user device; the preview image displayed on the first interface can be a preview image within the field of view of the camera, and the preview image is superimposed with the connectable devices in the current field of view, which can be device A and device B; when the user moves the user device and adjusts the field of view of the camera to another position, the connectable devices in the adjusted field of view, which can be device B and device C, can be displayed on the first interface.
[0046] The control sub-interfaces of each connectable device can be displayed in a list-like manner overlaid on the real scene image, such as Figure 3 Optionally, the display position of the control sub-interface in the real scene image may correspond to the position of the connectable device in the real scene. Figure 4 As shown, the connectable devices included in the real scene may be device A and device B, and the control sub-interface of device A may be located at the position of device A in the real scene image; the control sub-interface of device B may be located at the position of device B in the real scene image.
[0047] S104: Acquire a trigger operation based on the control sub-interface, and control the target connectable device according to the trigger operation.
[0048] The user device can obtain the trigger operation performed by the user on the control sub-interface based on the control sub-interface. The user device can determine the connectable device corresponding to the control sub-interface operated by the user as the target device, and control the target connectable device according to the trigger operation. When the user device controls the connectable device, the control type may include connection control, disconnection, device operation, system update, sleep, etc., which are not limited here.
[0049] The user equipment may control the target connectable device through signaling interaction with the target connectable device; or the target connectable device may be controlled through other devices such as a server. The above control methods are not limited here.
[0050] In the above-mentioned device control method, the user device responds to the image acquisition instruction and displays a first interface including a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image; then, a trigger operation is obtained based on the control sub-interface, and the target connectable device is controlled according to the trigger operation. Since the first interface displayed by the user device includes the display scene image and the connectable device superimposed on the display scene image, the user can quickly perceive which connectable devices are in the current scene through the first interface after acquiring the real scene image; further, since the first interface includes a control sub-interface of the connectable device, the user can complete the control of the target connectable device by executing a trigger operation on the control sub-interface of the target connectable device, thereby simplifying the control operation of the connectable device.
[0051] In one embodiment, Figure 5 As shown, this embodiment relates to a control sub-interface displayed on a first interface, and the control sub-interface may include device information of a connectable device and a connection control for connecting the connectable device.
[0052] The device information may be a diagram representing the device type of the connectable device, or may be text describing the connectable device information, which is not limited here. Optionally, the device information may include the device name of the connectable device and / or the device type of the connectable device.
[0053] The user can trigger the connection control by clicking, double-clicking, dragging, etc. If the user device detects that the connection control is triggered, a connection request is generated; and the connection request is sent to the target connectable device. The connection request is used to request to establish a communication connection with the target connectable device. The connection request may include the name of the user device, the name of the target connectable device, the IP address of the target connectable device, etc., which are not limited here.
[0054] After the above-mentioned connection control is triggered and the user device establishes a communication connection with the target connectable device, the control sub-interface of the target connectable device may present a connected state. For example, the connection control on the control sub-interface may be switched to an untriggerable state; or, the control sub-interface may be switched to a prompt interface of the connected state; or, the background color of the control sub-interface may be switched to a target color to indicate that the connectable device has established a connection with the user device through the target color; or, a disconnection control may be displayed on the control sub-interface to obtain the user's disconnection instruction through the control; the presentation method of the above-mentioned connected state is not limited here.
[0055] In one implementation, after the user device establishes a communication connection with the target connectable device, the user device can switch from the first interface to the second interface. The second interface includes a real scene image and a second operation sub-interface of the target connectable device superimposed on the real scene image, such as Figure 6 The second operation sub-interface includes operation items that match the device type of the target connectable device.
[0056] Among them, the above-mentioned operation items can be used to operate and control the target connectable device. The above-mentioned operation control can control the switch of the connectable device, such as the switch of the smart desk lamp; it can also control the working mode of the connectable device, and the type of the above-mentioned operation items is not limited here. The corresponding operation items for different types of connectable devices may be different. For example, the operation items of a smart TV can be used to select an input source, select a TV program, control volume, control screen brightness, etc.; the control items of a smart desk lamp can be used to select the on and off state of the desk lamp, and adjust the brightness of the desk lamp.
[0057] In the above-mentioned device control method, the control sub-interface of the connectable device includes a connection control, so that the user can establish a communication connection with the connectable device based on the connection control, without having to search and query the name of the connectable device before establishing the connection, which simplifies the operation steps of device connection; especially for users entering unfamiliar scenes, through the control sub-interface on the first interface, the device can be connected without knowing the device name in advance, thereby improving the user's operation experience. Furthermore, after the user device establishes a communication connection with the target connectable device, the first interface is switched to the second interface, so that the user can operate and control the connected connectable device on the second interface, which can not only show the user that the device is connected by switching the interface, but also further obtain the user's operation on the connected device on the second interface, further simplifying the user's operation control steps.
[0058] In one embodiment, based on the above embodiment, this embodiment relates to a display method of a control sub-interface. Figure 7 As shown, the control sub-interface may also include an operation control corresponding to the connectable device. If the user device detects that the operation control is triggered, the first operation sub-interface of the target connectable device may be displayed. Among them, the first operation sub-interface may include an operation item that matches the device type of the target connectable device. The specific definition of the operation item can refer to the above embodiment, which is not repeated here. For the same connectable device, the operation items displayed on the corresponding first operation sub-interface and the second operation sub-interface may be the same. The first operation sub-interface and the second operation sub-interface may be different in that the second operation sub-interface is superimposed on the real scene image after the user device establishes a communication connection with the target connectable device, so that the user can select the corresponding operation item on the second operation sub-interface, at which time the user may need to further operate the target connectable device, or may not need to further operate it; and the first operation sub-interface is the interface displayed after the user triggers the operation control of the connectable device and determines that the user needs to operate and control the connectable device. The first operation sub-interface may be superimposed on the real scene image, or it may be directly switched from the first interface where the real scene image is located to the first operation sub-interface.
[0059] Since the control sub-interface includes connection controls and operation controls, when the user device establishes a communication connection with the target connectable device, or when the user device does not establish a communication connection with the target connectable device, the user's trigger operation can be obtained through the operation control to operate and control the target connectable device. For example, the target connectable device can be a smart desk lamp. In one case, the user device has established a Bluetooth connection with the smart desk lamp. When the user triggers the operation control, the user device can control the switch of the smart desk lamp; in another case, when the user device does not establish a Bluetooth connection with the smart desk lamp, when the user triggers the operation control, the user device can also control the switch of the smart desk lamp through other devices such as a server.
[0060] In the above device control method, the control sub-interface on the first interface includes connection controls and operation controls, which provide users with a variety of control options, making the user's control of the connectable devices more flexible.
[0061] Figure 8 FIG. 1 is a flow chart of a device control method in an embodiment. This embodiment involves a user device that can obtain a processing method after a user's triggering operation through an operation item on the first operation sub-interface or the second operation sub-interface. Based on the above embodiment, Figure 8 As shown, the above method also includes:
[0062] S202: If it is detected that the operation item is triggered, generate an operation control signal corresponding to the triggered operation item.
[0063] The above operation control signal may correspond to the operation item one by one. For example, if the switch operation item is triggered, a switch control signal is generated; if the volume operation item is triggered, a volume control signal is generated.
[0064] S204: Perform operation control on the target connectable device according to the operation control signal.
[0065] The user device performs operation control on the target connectable device according to the operation control signal. In one implementation, when the user device has established a communication connection with the target connectable device, the user device can send the operation control signal to the target connectable device through the communication connection between the connectable device. The above operation control method can reduce data interaction between multiple devices and simplify the control process.
[0066] In another implementation, regardless of whether the user device and the target connectable device have established a communication connection, an operation control signal is sent to the server to perform operation control on the target connectable device through the server. For example, the user device can be connected to the server, and the target connectable device can also be connected to the server; after the user device generates the operation control signal, the operation control signal can be sent to the server, which is then transmitted to the target connectable device through the server to perform operation control on the target connectable device. The above operation control method can realize operation control on the target connectable device when the user device and the target connectable device have not established a communication connection, making operation control on the target connectable device more convenient.
[0067] The above operation control signal can also be sent to the target connectable device through multiple paths. The user device can send the operation control signal to the server and the target connectable device at the same time, and the server can transmit the received operation control signal to the target connectable device.
[0068] In the above device control method, the user device can generate an operation control signal according to the user's trigger operation, and then transmit the operation control signal to the target connectable device through different paths. The connectable devices in different connection states can be operated and controlled, making the operation control more flexible.
[0069] Fig. 9 FIG. 1 is a flow chart of a device control method in an embodiment. This embodiment relates to a method for generating a first interface by a user device. Based on the above embodiment, Fig. 9 As shown, before S102, the above-mentioned step further includes:
[0070] S302. Acquire location information of each connectable device in the current scene; the location information includes the distance between the connectable device and the user device, and the orientation information of the connectable device relative to the user device.
[0071] The user equipment may calculate the location information of each connectable device through measurement, or may directly receive the location information of each connectable device, which is not limited here.
[0072] The location information may include the distance between the connectable device and the user device, and the orientation information of the connectable device relative to the user device. The orientation information may be the angle between the line connecting the connectable device and the user device and the focal plane normal vector of the user device.
[0073] In one implementation, the user device may obtain the coordinates of the user device itself, the coordinates of the connectable device, and the focal plane normal vector of the user device to determine the location information of the connectable device. The coordinates may be coordinates in a coordinate system with the user device as a reference, coordinates in a coordinate system with the current scene as a reference, or earth coordinates, etc., which are not limited here.
[0074] In another implementation, if the user device has a high-precision angle measurement capability, such as an ultra-wideband (UltraWide Band, UWB) angle measurement capability, and the angle measurement sensitive direction of the user device is consistent with the focal plane normal vector direction of the camera, and when the connectable device is equipped with a UWB beacon, the user device can measure the relative position of the user device and the connectable device without knowing its own coordinates, and obtain the location information of the connectable device.
[0075] S304: Determine, according to each piece of location information and the field of view of the user equipment, candidate connectable devices within the field of view.
[0076] On the basis of determining the location information of each connectable device, the user equipment may determine which connectable devices are candidate connectable devices located within the field of view according to the current field of view.
[0077] S306: Superimpose and display the control sub-interface of the candidate connectable device on the real scene image within the field of view according to the position information to generate a first interface.
[0078] Based on the above steps, the user device can correspond the actual spatial coordinates of the candidate connectable device to the image coordinates on the real scene image, and superimpose the control sub-interface of the candidate connectable device on the real scene image according to the above image coordinates to generate a first interface.
[0079] In the above device control method, the user device overlays the control sub-interface on the real scene image according to the location information, so that the user can more intuitively see which positions are equipped with connectable devices, and then control the connectable devices that need to be controlled.
[0080] Fig.10 FIG. 1 is a flow chart of a device control method in an embodiment. This embodiment relates to a method for a user device to obtain location information of a connectable device. Based on the above embodiment, Fig.10 As shown, the above S302 includes:
[0081] S402: Acquire multiple positioning coordinates sent by multiple positioning beacons set in the current scene.
[0082] Multiple UWB beacons can be set in the current scenario, and the user device obtains the positioning coordinates of the above multiple UWB beacons. The user device can obtain the positioning coordinates of each UWB beacon in the current scenario by accessing the server, or it can receive the broadcast message sent by each UWB beacon and obtain the above positioning coordinates from the broadcast message. The method for obtaining the positioning coordinates is not limited here. The above broadcast message can be a low-power Bluetooth (Bluetooh Low Energy, referred to as BLE) broadcast or a WiFi broadcast, which is not limited here.
[0083] S404: Determine a first coordinate of the user equipment according to the multiple positioning coordinates.
[0084] The user equipment can measure the distance between the user equipment and each UWB beacon, and then calculate the first coordinate of the user equipment itself. Fig.11 For example, the user equipment can determine the coordinates of the user equipment on the two-dimensional plane through the positioning coordinates of three UWB beacons and the distances R1, R2, and R3 between the user equipment and the above UWB beacons. For three-dimensional spatial coordinates, the user equipment needs to obtain the positioning coordinates of at least four UWB beacons and the distances between the user equipment and the four UWB beacons.
[0085] S406. Acquire the second coordinates of the connectable device; the second coordinates are determined according to the broadcast information sent by the connectable device, or according to the positioning information sent by the server.
[0086] The above-mentioned connectable devices may include connectable devices with fixed positions and connectable devices with variable positions. For connectable devices with fixed positions, their coordinates are also fixed; the user device can obtain the second coordinates of the connectable device by receiving the broadcast message sent by the above-mentioned connectable device, or by accessing the server, receiving the positioning information sent by the server, and obtaining the second coordinates of the connectable device from the above-mentioned positioning information. For connectable devices with variable positions, the connectable device can use the same method as the user device to solve its own second coordinates through multiple UWB beacons, and can send the above-mentioned second coordinates to the user device through broadcast information, or can send the above-mentioned second coordinates to the server, so that the user device can obtain the above-mentioned second coordinates from the server.
[0087] S408: Obtain a focal plane normal vector of a camera of the user device.
[0088] After obtaining its own coordinates, the user device also needs to obtain the focal plane normal vector of the camera. In one implementation, the user device can display prompt information, which is used to prompt the user to move the user device along a preset path; then obtain multiple moving coordinates of the user device during the movement process, and determine the camera orientation of the user device based on the multiple moving coordinates; finally, obtain the focal plane normal vector based on the camera orientation and the pitch angle of the camera.
[0089] The preset path may be centered on the user, with the camera of the user device facing the side of the user, and the user device may be rotated, such as Fig.12 In the above figure, the user device moves from the initial position to the final position, and the user device can calculate the movement trajectory, thereby determining the inner and outer sides of the user device according to the trajectory, that is, the side facing the user and the side away from the user; and then determining the side away from the user as the camera direction.
[0090] After the user device determines the camera orientation, it can obtain the camera pitch angle according to a sensor set on the user device, such as a rotation vector sensor. Based on the camera orientation and pitch angle, the user device can determine the focal plane normal vector of the camera.
[0091] S410: Determine location information of a connectable device according to the first coordinate, the second coordinate, and the focal plane normal vector.
[0092] Based on the above steps, the user equipment can determine the location information of the connectable device according to the first coordinate, the second coordinate and the focal plane normal vector.
[0093] In the above device control method, when a positioning beacon is set in the current scene, the user device can obtain its own first coordinate, the second coordinate of the connectable device and the focal plane normal vector of the camera, so as to obtain more accurate location information of the connectable device and improve the accuracy of superimposing the control sub-interface on the real scene image.
[0094] Fig.13 FIG. 1 is a flow chart of a device control method in an embodiment. This embodiment relates to a method for a user device to obtain location information of a connectable device. Based on the above embodiment, Fig.13 As shown, the above S302 includes:
[0095] S502, collecting real scene images, and identifying positioning reference lines in the real scene images; the positioning reference lines include vertical reference lines.
[0096] The user device may capture the real scene image after obtaining the image capture instruction. Further, the user device may use AR scene understanding to identify the positioning reference line in the real scene image. Taking the indoor scene as an example, the positioning reference line may be the boundary line between adjacent walls, such as the connection line between the ceiling, floor and wall. The positioning reference line may include a vertical reference line, and the vertical reference line may be a wall connection line perpendicular to the ground.
[0097] S504: Determine the size of the positioning reference line according to preset model parameters of the current scene.
[0098] For most indoor scenes, the indoor space can be an approximate cuboid, and the model parameters of the indoor scene can be preset in the server. The above model parameters may include various dimensions of the indoor space, such as wall height, ground length, etc. The user device can obtain the model parameters of the current scene from the server.
[0099] The user equipment may match the model parameters with the above-identified positioning reference line to obtain the size of the positioning reference line.
[0100] S506: Determine a third coordinate of the user device and a focal plane normal vector of the camera of the user device according to preset camera parameters, the size of the reference line, and the pitch angle of the camera.
[0101] Based on the above steps, the user equipment may determine the third coordinate of the user equipment according to preset camera parameters, the size of the reference line, and the pitch angle of the camera.
[0102] The user device can determine the height of the user device and the distance between the user device and each vertical reference line according to the pitch angle of the camera and the camera parameters. Fig.14 As shown, O in the figure can identify the camera of the user device, AB in the figure can be the vertical reference line in the current scene, and A′B′C′ in the figure can represent the imaging of the camera. Given the focal length f and the length of the vertical reference line AB, the size of A′B′ is calculated in the imaged image, and the distance OC from the camera to the vertical reference line AB can be: (f×AB) / A′B′. The height of the user device can be obtained by (B′C′×AB) / A′B′.
[0103] Furthermore, the user equipment may determine the coordinates of the user equipment on the horizontal plane according to the distance between the user equipment and at least two vertical reference lines. Fig.15As shown, the user device can simultaneously capture two vertical reference lines, and calculate the coordinates of the user device in the horizontal plane according to the distances R1 and R2 between the user device and the two vertical reference lines. On the basis of obtaining the horizontal plane coordinates of the user device and the height of the user device, the user device obtains the third coordinate of the user device according to the horizontal plane coordinates and the height of the user device.
[0104] The user device can determine the focal plane normal vector of the camera of the user device according to the preset camera parameters, the size of the reference line, and the pitch angle of the camera. Fig.16 As shown, the user device can calculate the deflection of the focal plane normal vector relative to the vertical reference line in the horizontal plane based on the distance between the vertical reference line AB and the imaging center. After the vertical reference line AB deviates from the axis of the camera lens, its image A′B′ deviates from the imaging center by a certain distance. The deflection angle of the camera axis relative to the vertical reference line AB can be arctan(PC / OC); wherein point P is the intersection between the image of the vertical reference line and the imaging center. If the object distance is much greater than the phase distance, OC is the focal length f of the camera, and the deflection angle can be arctan(PC / f).
[0105] S508: Determine location information of the connectable device according to the third coordinate, the fourth coordinate of the connectable device, and the focal plane normal vector.
[0106] The user device can also obtain the fourth coordinates of the connectable device. The above-mentioned connectable devices may include connectable devices with fixed positions and connectable devices with variable positions. For connectable devices with fixed positions, their coordinates are also fixed; the user device can obtain the fourth coordinates of the connectable device by receiving a broadcast message sent by the above-mentioned connectable device, or by accessing a server, receiving positioning information sent by the server, and obtaining the fourth coordinates of the connectable device from the above-mentioned positioning information. For connectable devices with variable positions, UWB beacons may be provided on the above-mentioned connectable devices; the user device can measure the distance between the connectable device and the connectable device at multiple spatial positions, and then determine the fourth coordinates of the connectable device based on the multiple distance values measured.
[0107] Based on the above steps, the user equipment may determine the location information of the connectable device according to the third coordinate, the fourth coordinate of the connectable device and the focal plane normal vector.
[0108] With the above device control method, the user device can obtain the location information of the connectable device in a scenario where there is no UWB beacon, thereby improving the applicability of the device control method.
[0109] In one embodiment, a device control method is provided, such as Fig.17 As shown, the above method includes:
[0110] S602: Obtain location information of each connectable device in the current scene.
[0111] S604: Determine, according to each piece of location information and the field of view of the user equipment, a candidate connectable device within the field of view.
[0112] S606: Display the control sub-interface of the candidate connectable device on the real scene image within the field of view in a superimposed manner according to the position information to generate the first interface.
[0113] S608: In response to the image acquisition instruction, display a first interface.
[0114] S610: If it is detected that the connection control on the first interface is triggered, a connection request is generated.
[0115] S612: Send the connection request to the target connectable device.
[0116] S614: If it is detected that the operation control is triggered, display a first operation sub-interface of the target connectable device.
[0117] S616: If it is detected that an operation item on the first operation sub-interface is triggered, generate an operation control signal corresponding to the triggered operation item.
[0118] S618: Perform operation control on the target connectable device according to the operation control signal.
[0119] The implementation principle and technical effects of the above-mentioned device control method refer to the above-mentioned embodiments and will not be described in detail here.
[0120] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0121] Based on the same inventive concept, the embodiment of the present application also provides a device control device for implementing the device control method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more device control device embodiments provided below can refer to the limitations on the device control method above, and will not be repeated here.
[0122] In one embodiment, Fig.18 As shown, a device control device is provided, comprising:
[0123] The display module 10 is used to display a first interface in response to the image acquisition instruction; the first interface includes a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image;
[0124] The control module 20 is used to obtain a trigger operation based on the control sub-interface and control the target connectable device according to the trigger operation.
[0125] In one embodiment, based on the above embodiment, the control sub-interface includes device information of the connectable device and a connection control for connecting the connectable device.
[0126] In one embodiment, based on the above embodiment, the above device information includes a device name of the connectable device and / or a device type of the connectable device.
[0127] In one embodiment, based on the above embodiment, the control module 20 is specifically used for: generating a connection request if it is detected that the connection control is triggered; the connection request is used to request to establish a communication connection with the target connectable device; and sending the connection request to the target connectable device.
[0128] In one embodiment, based on the above embodiment, the control sub-interface also includes operation controls corresponding to the connectable device; the above-mentioned control module 20 is specifically used for: if it is detected that the operation control is triggered, then the first operation sub-interface of the target connectable device is displayed; the first operation sub-interface includes operation items that match the device type of the target connectable device.
[0129] In one embodiment, based on the above embodiment, the above control module 20 is also used to: switch from the first interface to the second interface after establishing a communication connection with the target connectable device; the second interface includes a real scene image and a second operation sub-interface of the target connectable device superimposed on the real scene image; the second operation sub-interface includes operation items that match the device type of the target connectable device.
[0130] In one embodiment, based on the above embodiment, the control module 20 is further used to: if it is detected that the operation item is triggered, generate an operation control signal corresponding to the triggered operation item; and perform operation control on the target connectable device according to the operation control signal.
[0131] In one embodiment, based on the above embodiment, the control module 20 is specifically used to send an operation control signal to a target connectable device through a communication connection with the connectable device.
[0132] In one embodiment, based on the above embodiment, the control module 20 is specifically used to: send an operation control signal to the server, so as to perform operation control on the target connectable device through the server.
[0133] In one embodiment, based on the above embodiment, the display position of the control sub-interface in the real scene image corresponds to the position of the connectable device in the real scene.
[0134] In one embodiment, based on the above embodiment, Fig.19 As shown, the above-mentioned device also includes a generating module 30, which is used to: obtain the location information of each connectable device in the current scene; the location information includes the distance between the connectable device and the user device, and the orientation information of the connectable device relative to the user device; determine the candidate connectable devices located within the field of view according to each location information and the field of view of the user device; and superimpose the control sub-interface of the candidate connectable device on the real scene image within the field of view according to the location information to generate a first interface.
[0135] In one embodiment, based on the above embodiment, the generation module 30 is specifically used to: obtain multiple positioning coordinates sent by multiple positioning beacons set in the current scene; determine the first coordinates of the user device based on the multiple positioning coordinates; obtain the second coordinates of the connectable device; the second coordinates are determined based on the broadcast information sent by the connectable device, or, based on the positioning information sent by the server; obtain the focal plane normal vector of the camera of the user device; determine the location information of the connectable device based on the first coordinates, the second coordinates and the focal plane normal vector.
[0136] In one embodiment, based on the above embodiment, the generation module 30 is specifically used to: display prompt information; the prompt information is used to prompt the user to move the user device along a preset path; obtain multiple moving coordinates of the user device during the movement, and determine the camera direction of the user device based on the multiple moving coordinates; obtain the focal plane normal vector based on the camera direction and the pitch angle of the camera.
[0137] In one embodiment, based on the above embodiment, the generation module 30 is specifically used to: collect real scene images and identify positioning reference lines in the real scene images; the positioning reference lines include vertical reference lines; according to preset model parameters of the current scene, the size of the positioning reference line is determined; according to preset camera parameters, the size of the reference line and the pitch angle of the camera, the third coordinate of the user device and the focal plane normal vector of the camera of the user device are determined; according to the third coordinate, the fourth coordinate of the connectable device and the focal plane normal vector, the location information of the connectable device is determined.
[0138] In one embodiment, based on the above embodiment, the generation module 30 is specifically used to: determine the height of the user device and the distance between the user device and the vertical reference line according to the pitch angle of the camera and the camera parameters; determine the coordinates of the user device in the horizontal plane according to the distance between the user device and at least two vertical reference lines, and obtain the third coordinates of the user device according to the coordinates of the horizontal plane and the height of the user device; calculate the focal plane normal vector of the camera according to the distance of the vertical reference line from the imaging center on the imaging plane of the camera.
[0139] The implementation principle and technical effects of the above-mentioned device control device refer to the above-mentioned method embodiment and will not be described in detail here.
[0140] Each module in the above-mentioned device control device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0141] In one embodiment, a user equipment is provided, and its internal structure diagram can be as follows: Fig. 20As shown. The user device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the user device is used to provide computing and control capabilities. The memory of the user device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the user device is used to exchange information between the processor and the external device. The communication interface of the user device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a device control method is implemented. The display unit of the user device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0142] Those skilled in the art will understand that Fig. 20 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0143] The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the device control method.
[0144] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the device control method.
[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A device control method, characterized in that: include: Get the location information of each connectable device in the current scene; The location information includes the distance between the connectable device and the user equipment, and the orientation information of the connectable device relative to the user equipment; Determine, according to each of the location information and the field of view of the user equipment, a candidate connectable device within the field of view; Displaying the control sub-interface of the candidate connectable device on the real scene image within the field of view in a superimposed manner according to the position information to generate a first interface; In response to the image acquisition instruction, the first interface is displayed; the first interface includes a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image; wherein the display position of the control sub-interface in the real scene image corresponds to the position of the connectable device in the real scene; A trigger operation is acquired based on the control sub-interface, and a target connectable device is controlled according to the trigger operation.
2. The method according to claim 1, characterized in that The control sub-interface includes device information of the connectable device and a connection control for connecting the connectable device.
3. The method according to claim 2, characterized in that The device information includes a device name of the connectable device and / or a device type of the connectable device.
4. The method according to claim 2, characterized in that: The acquiring a trigger operation based on the control sub-interface and controlling a target connectable device according to the trigger operation includes: If it is detected that the connection control is triggered, a connection request is generated; the connection request is used to request to establish a communication connection with the target connectable device; The connection request is sent to the target connectable device.
5. The method according to claim 2, characterized in that: The control sub-interface also includes an operation control corresponding to the connectable device; the acquiring a trigger operation based on the control sub-interface and controlling the target connectable device according to the trigger operation include: If it is detected that the operation control is triggered, a first operation sub-interface of the target connectable device is displayed; the first operation sub-interface includes operation items that match the device type of the target connectable device.
6. The method according to claim 4, characterized in that The method further comprises: After establishing a communication connection with the target connectable device, switch from the first interface to the second interface; the second interface includes the real scene image and a second operation sub-interface of the target connectable device superimposed on the real scene image; the second operation sub-interface includes operation items that match the device type of the target connectable device.
7. The method according to claim 5 or 6, characterized in that: The method further comprises: If it is detected that the operation item is triggered, an operation control signal corresponding to the triggered operation item is generated; The target connectable device is operationally controlled according to the operational control signal.
8. The method according to claim 7, characterized in that The performing operation control on the target connectable device according to the operation control signal includes: The operation control signal is sent to the target connectable device through the communication connection with the connectable device.
9. The method according to claim 7, characterized in that: The performing operation control on the target connectable device according to the operation control signal includes: The operation control signal is sent to a server so as to perform operation control on the target connectable device through the server.
10. The method according to claim 1, characterized in that The control sub-interface also includes at least one of a remaining power indicator, a network connection status indicator, and a current working status of the connectable device.
11. The method according to claim 1, characterized in that: The control type of the target connectable device includes at least one of connection control, disconnection, device operation, system update, and sleep.
12. The method according to claim 1, characterized in that The obtaining of location information of each connectable device in the current scene includes: Obtain multiple positioning coordinates sent by multiple positioning beacons set in the current scene; Determine a first coordinate of the user equipment according to the multiple positioning coordinates; Acquire a second coordinate of the connectable device; the second coordinate is determined according to broadcast information sent by the connectable device, or determined according to positioning information sent by the server; Obtaining a focal plane normal vector of a camera of the user device; The location information of the connectable device is determined according to the first coordinate, the second coordinate, and the focal plane normal vector.
13. The method according to claim 12, characterized in that The obtaining of the focal plane normal vector of the camera of the user equipment includes: Display prompt information; the prompt information is used to prompt the user to move the user device according to a preset path; Acquire a plurality of movement coordinates of the user equipment during movement, and determine a camera orientation of the user equipment according to the plurality of movement coordinates; The focal plane normal vector is obtained according to the camera orientation and the pitch angle of the camera.
14. The method according to claim 1, characterized in that The obtaining of location information of each connectable device in the current scene includes: Acquire the real scene image and identify the positioning reference line in the real scene image; the positioning reference line includes a vertical reference line; Determining the size of the positioning reference line according to preset model parameters of the current scene; Determine a third coordinate of the user device and a focal plane normal vector of the camera of the user device according to preset camera parameters, the size of the reference line, and the pitch angle of the camera; The position information of the connectable device is determined according to the third coordinate, the fourth coordinate of the connectable device and the focal plane normal vector.
15. The method according to claim 14, characterized in that The determining, according to preset camera parameters, the size of the reference line, and the pitch angle of the camera, the third coordinate of the user equipment and the focal plane normal vector of the camera of the user equipment comprises: Determine the height of the user equipment and the distance between the user equipment and the vertical reference line according to the pitch angle of the camera and the camera parameters; Determine the coordinates of the user equipment in a horizontal plane according to the distance between the user equipment and at least two vertical reference lines, and obtain a third coordinate of the user equipment according to the coordinates of the horizontal plane and the height of the user equipment; The focal plane normal vector of the camera is calculated according to the distance of the vertical reference line from the imaging center on the imaging plane of the camera.
16. A device control device, characterized in that: Applied to user equipment, including: A generation module, configured to obtain location information of each connectable device in the current scene; the location information includes the distance between the connectable device and the user device, and the orientation information of the connectable device relative to the user device; determine candidate connectable devices within the field of view according to the location information and the field of view of the user device; and superimpose and display the control sub-interface of the candidate connectable device on the real scene image within the field of view according to the location information to generate a first interface; A display module, configured to display the first interface in response to an image acquisition instruction; the first interface includes a real scene image and a control sub-interface of a connectable device in the real scene superimposed on the real scene image; wherein a display position of the control sub-interface in the real scene image corresponds to a position of the connectable device in the real scene; The control module is used to obtain a trigger operation based on the control sub-interface and control a target connectable device according to the trigger operation.
17. A user equipment, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the computer program is executed by the processor, the processor is caused to execute the steps of the device control method according to any one of claims 1 to 15.
18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.
19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.
