Multi-terminal Bluetooth interaction method and device

Abstract

The present application discloses a multi-terminal Bluetooth interaction method and device, and relates to the field of Bluetooth communication technology. Among them, the multi-terminal Bluetooth interaction method proposed in the present application obtains Bluetooth information of nearby Bluetooth devices by scanning nearby Bluetooth devices, and then obtains a device layout image, wherein the device layout image contains images of multiple Bluetooth devices, obtains the category label and location information of each of the Bluetooth devices according to the device layout image, generates an interaction object at the corresponding position of the device layout image according to the category label, location information and Bluetooth information of each of the Bluetooth devices, and finally interacts with the Bluetooth device in response to the triggering of the interaction object. In this way, when there are multiple devices around, this solution does not need to search for devices in a long list of Bluetooth devices, and can interact with multiple Bluetooth devices intuitively.

Description

Technical Field

[0001] The present application relates to the field of Bluetooth communication technology, and in particular to a multi-terminal Bluetooth interaction method and device. Background Art

[0002] As mobile Internet is developing rapidly, Bluetooth technology has maintained rapid growth in the IoT market, and has become a standard feature of various mobile smart terminals. The increasing number of Bluetooth devices has also brought huge challenges to human-computer interaction, such as Bluetooth device identification.

[0003] Existing solutions to the Bluetooth device identification problem all have certain limitations. In actual usage scenarios, users usually assign a readable text name to the available Bluetooth devices and use the device name to identify the wireless device. Under the current interaction logic, device identification often carries a certain degree of user subjectivity, that is, the device name may be a long string of characters or an identifier (UUID), etc. Users need to search against the identifier to connect and interact. Although this text-based identification strategy has a certain degree of versatility, it becomes very inefficient when the device name is tampered with or the user cannot understand it.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a multi-terminal Bluetooth interaction method and device, aiming to improve the intuitiveness of Bluetooth device interaction.

[0006] To achieve the above objectives, the present application proposes a multi-terminal Bluetooth interaction method, the method comprising:

[0007] Scan nearby Bluetooth devices and obtain Bluetooth information of nearby Bluetooth devices;

[0008] Acquire a device layout image, wherein the device layout image includes images of multiple Bluetooth devices;

[0009] Performing preliminary recognition on the device arrangement image by using a target detection algorithm to obtain the bounding box coordinates of each of the Bluetooth devices;

[0010] Perform secondary recognition on the device layout image through a semantic segmentation model to obtain the precise boundary contour and category label of each Bluetooth device, and obtain the two-dimensional coordinates of each Bluetooth device in the device layout image;

[0011] Obtain the distance between each Bluetooth device and the camera in the device layout image, and convert the two-dimensional coordinates of each Bluetooth device in the device layout image into the three-dimensional coordinates (x cam ,y cam , z cam ), wherein the distance between each of the Bluetooth devices and the camera is used as the z-axis coordinate of the Bluetooth device in the camera coordinate system;

[0012] According to the three-dimensional coordinates (x cam ,y cam , z cam ) generating an interactive object at a position corresponding to the image arranged on the device;

[0013] The method interacts with the Bluetooth device in response to triggering of the interactive object.

[0014] In one embodiment, the distance between each Bluetooth device and the camera in the device layout image is obtained, and the two-dimensional coordinates of each Bluetooth device in the device layout image are converted into the three-dimensional coordinates (x cam ,y cam , z cam ) steps include:

[0015] Obtain the camera's internal parameters and the pixel coordinates (u, v) in the Bluetooth device image coordinate system, the internal parameters including the focal length (f x , f y ) and the optical center (c x , c y );

[0016] The three-dimensional coordinates (x, v) of the Bluetooth device in the camera coordinate system are obtained according to the intrinsic parameters of the camera and the pixel coordinates (u, v) in the Bluetooth device image coordinate system. cam ,y cam , z cam );

[0017] The three-dimensional coordinates (x cam ,y cam , z cam ) is obtained according to the following formula:

[0018] z cam =d(u,v)

[0019] x cam =(u−c x )×z cam / f x

[0020] y cam =(v−c y )×z cam / f y

[0021] Among them, z cam is the depth of field of the Bluetooth device or ,

[0022] The distance between the Bluetooth device and the camera is estimated by the received signal strength RSSI.

[0023] In one embodiment, the distance between each Bluetooth device and the camera in the device layout image is obtained, and the two-dimensional coordinates of each Bluetooth device in the device layout image are converted into the three-dimensional coordinates (x cam ,y cam , z cam ) steps:

[0024] The depth of field of each Bluetooth device in the device arrangement image is acquired by a depth of field calculation algorithm, and the depth of field of each Bluetooth device is used as the distance between each Bluetooth device and the camera.

[0025] In one embodiment, the depth of field calculation algorithm includes a stereo vision algorithm, a monocular depth estimation algorithm and a depth of field estimation algorithm.

[0026] In one embodiment, the distance between each Bluetooth device and the camera in the device layout image is obtained, and the two-dimensional coordinates of each Bluetooth device in the device layout image are converted into the three-dimensional coordinates (x cam ,y cam , z cam ) steps:

[0027] By obtaining the RSSI value of the Bluetooth device, the distance between each Bluetooth device and the camera is obtained through the RSSI value ;

[0028] Wherein, the distance between each of the Bluetooth devices and the camera Obtained according to the following formula:

[0029] ;

[0030] in,

[0031] It is the estimated distance between the Bluetooth device and the camera estimated by the received signal strength RSSI;

[0032] is the reference RSSI value;

[0033] RSSI is the current received signal strength;

[0034] n is the environmental attenuation factor.

[0035] In one embodiment, the Bluetooth information includes a device category, and the three-dimensional coordinates (x cam ,y cam , z cam ) The step of generating an interactive object at a position corresponding to the image arranged on the device comprises:

[0036] Match the corresponding device class to the class label.

[0037] In one embodiment, the method comprises:

[0038] Get the Bluetooth information of hidden Bluetooth devices in the device layout image;

[0039] The position of the hidden Bluetooth device is determined according to the Bluetooth information of the hidden Bluetooth device, and an interactive object is generated at a position corresponding to the device layout image.

[0040] In one embodiment, the Bluetooth information includes device type, angle of arrival, and received signal strength RSSI, the step of determining the position of the hidden Bluetooth device according to the Bluetooth information of the hidden Bluetooth device, and generating an interactive object at a position corresponding to the device layout image comprises:

[0041] According to the arrival angle Identify the Bluetooth device located in front of the camera;

[0042] determining a hidden Bluetooth device in the device arrangement image according to the identified Bluetooth device and the Bluetooth device located in front of the camera;

[0043] According to the arrival angle of the hidden Bluetooth device The two-dimensional coordinates (x device ,y device );

[0044] According to the two-dimensional coordinates (x device ,y device ) and device category generate interactive objects of hidden Bluetooth devices at corresponding positions in the device layout image.

[0045] In one embodiment, the two-dimensional coordinates (x , y ) of the hidden Bluetooth device in the camera coordinate system are determined according to the arrival angle and the received signal strength RSSI of the hidden Bluetooth device. device ,y device ) steps:

[0046] Assume that the hidden Bluetooth device in the device layout image is located at the same horizontal plane as the camera;

[0047] The distance between the device and the receiver is obtained through the RSSI value. The formula is:

[0048]

[0049] The two-dimensional coordinates (x device ,y device ) is obtained according to the following formula:

[0050]

[0051]

[0052] The present application also proposes a Bluetooth interaction device, which adopts the multi-terminal Bluetooth interaction method as described in any of the above items.

[0053] The multi-terminal Bluetooth interaction method proposed in this application obtains Bluetooth information of nearby Bluetooth devices by scanning nearby Bluetooth devices, and then obtains a device layout image, wherein the device layout image includes images of multiple Bluetooth devices, obtains the category label and location information of each Bluetooth device according to the device layout image, generates an interaction object at the corresponding position of the device layout image according to the category label, location information and Bluetooth information of each Bluetooth device, and finally interacts with the Bluetooth device in response to the triggering of the interaction object. In this way, when there are multiple devices around, this solution does not need to search for devices in a long list of Bluetooth devices, and can intuitively interact with multiple Bluetooth devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0056] Figure 1A flowchart of an embodiment of a multi-terminal Bluetooth interaction method of the present application is provided;

[0057] Figure 2 A flowchart diagram of another embodiment of the multi-terminal Bluetooth interaction method of the present application;

[0058] Figure 3 A flowchart diagram of another embodiment of the multi-terminal Bluetooth interaction method of the present application is provided;

[0059] Figure 4 A flowchart diagram of another embodiment of the multi-terminal Bluetooth interaction method of the present application is provided;

[0060] Figure 5 A flowchart diagram is provided for another embodiment of the multi-terminal Bluetooth interaction method of the present application.

[0061] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0064] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0065] As mobile Internet is developing rapidly, Bluetooth technology has maintained rapid growth in the IoT market, and has become a standard feature of various mobile smart terminals. The increasing number of Bluetooth devices has also brought huge challenges to human-computer interaction, such as Bluetooth device identification.

[0066] Existing solutions to the Bluetooth device identification problem all have certain limitations. In actual usage scenarios, users usually assign a readable text name to the available Bluetooth devices and use the device name to identify the wireless device. Under the current interaction logic, device identification often carries a certain degree of user subjectivity, that is, the device name may be a long string of characters or an identifier (UUID), etc. Users need to search against the identifier to connect and interact. Although this text-based identification strategy has a certain degree of versatility, it becomes very inefficient when the device name is tampered with or the user cannot understand it.

[0067] To solve the above problems, the present invention proposes a multi-terminal Bluetooth interaction method and apparatus, aiming to improve the intuitiveness of Bluetooth device interaction.

[0068] Please refer to Figure 1 , Figure 1 The flowchart provided in the embodiment of the multi-terminal Bluetooth interaction method of the present application is as follows. The multi-terminal Bluetooth interaction method proposed in the present application includes steps S100 to S700, wherein:

[0069] Step S100, scanning nearby Bluetooth devices to obtain Bluetooth information of nearby Bluetooth devices;

[0070] Step S200, obtaining a device layout image, wherein the device layout image includes images of multiple Bluetooth devices;

[0071] Step S300, performing preliminary recognition on the device arrangement image by using a target detection algorithm to obtain the bounding box coordinates of each Bluetooth device;

[0072] Step S400, performing secondary recognition on the device layout image through a semantic segmentation model, obtaining the precise boundary contour and category label of each Bluetooth device, and obtaining the two-dimensional coordinates of each Bluetooth device in the device layout image;

[0073] Step S500, obtaining the distance between each Bluetooth device and the camera in the device layout image, and converting the two-dimensional coordinates of each Bluetooth device in the device layout image into the three-dimensional coordinates (x , m ) of the Bluetooth device in the camera coordinate system according to the distance between each Bluetooth device and the camera. cam ,y cam , z cam );

[0074] The distance between each Bluetooth device and the camera is used as the z-axis coordinate of the Bluetooth device in the camera coordinate system;

[0075] Step S600: according to the three-dimensional coordinates (x cam,y cam , z cam ) generating an interactive object at a position corresponding to the image arranged on the device;

[0076] Step S700: interacting with the Bluetooth device in response to the triggering of the interactive object.

[0077] With the rapid development of the Internet of Things and Bluetooth technology, there are often a large number of Bluetooth devices around in home and office scenes. These devices include smart speakers, Bluetooth headsets, Bluetooth mice, keyboards, printers, health monitoring devices, smart locks, etc., with various types and functions. This phenomenon is especially obvious in public places such as office areas and shopping malls. For example, on a large office floor or shopping mall, there may be multiple Bluetooth devices such as smart lights, air conditioners, thermostats, etc.

[0078] In such an environment, due to the large number of identical or similar Bluetooth devices, it becomes extremely difficult to distinguish devices only by text descriptions. Users may encounter confusion and inconvenience when selecting devices, especially when multiple devices have similar names. For example, when users need to operate a specific smart lighting system, they may spend more time and energy due to the inability to accurately identify the device, resulting in a poor interactive experience.

[0079] For example, in an office with multiple smart lights of the same model and appearance hanging on the wall, users control the switch or brightness of each light through Bluetooth connection. However, when users try to control these light connections through smart terminals, it is often difficult to accurately identify the target device based on simple text descriptions such as "the first light", "the light on the right", etc. This ambiguous indication method not only increases the complexity of user operation, but may also cause unnecessary trouble and confusion due to misoperation.

[0080] In this embodiment, firstly, the Bluetooth module built into the smart terminal can be used to scan nearby Bluetooth devices to obtain Bluetooth information of these devices, such as device names, MAC addresses, etc.

[0081] Then, a real-time image of the current environment is obtained through a camera module built into the smart terminal, for example, a real-time image of the current environment is captured by a camera of AR glasses or a camera of a smart phone to obtain a device layout image. The device layout image includes images of multiple Bluetooth devices.

[0082] Next, the device layout image is analyzed to identify the location of each Bluetooth device in the device layout image, and corresponding category labels are assigned according to Bluetooth information or image features, such as lights, speakers, etc. At the same time, the location information of each device is also obtained according to the Bluetooth information and the image information of the device layout image.

[0083] Next, according to the category label, location information and Bluetooth information of each Bluetooth device, the system generates corresponding virtual interactive objects at the corresponding positions of the device layout image. These interactive objects can be in the form of icons, buttons, dialog boxes, etc., which can intuitively represent the category and current status of the device.

[0084] Finally, the user interacts with the Bluetooth device through the triggering of the interactive object. For example, by clicking on the icon of a device, the smart terminal establishes a Bluetooth connection with the device. The smart terminal can be AR glasses or a smart phone, and the user can interact with the interactive object through the application interface on the AR glasses or the smart phone. Exemplarily, click on the interactive object corresponding to the smart bulb to establish a Bluetooth connection with the smart bulb, and pop up the interactive interface. The user can adjust the brightness or color of the light in the interactive interface. The system receives the user's interactive operation, converts it into a corresponding Bluetooth control command and sends it to the target device, thereby realizing the function of controlling the device.

[0085] In this embodiment, users can view and control Bluetooth devices at home through an intuitive AR interface. The system automatically identifies the location of the device and generates interactive objects, so that users do not need to manually find and select devices, which improves the convenience of user experience and operational efficiency. It can be applied to multiple scenarios such as smart home and smart office, greatly enhancing the interactive experience between users and devices.

[0086] In this embodiment, the device layout image obtained by the camera is first preprocessed, such as adjusting the scale, cropping, enhancement, normalization, etc., to meet the input requirements of the model. The preprocessed device layout image is preliminarily analyzed using pre-trained target detection algorithms such as YOLO, Faster R-CNN, etc. The YOLO target detection algorithm generates preliminary bounding box coordinates by detecting multiple possible objects (Bluetooth devices) in the image. Each box surrounds a detected Bluetooth device. The bounding box coordinates include the coordinates of the four corner points of each device, defining the approximate position of the device in the image. Each bounding box also carries a confidence score, indicating the reliability of the detection result. Next, the image area containing each device is extracted from the preliminarily identified bounding box. These areas contain detailed information about the device, but may contain some irrelevant background information. Pre-trained semantic segmentation models (such as DeepLab, U-Net, etc.) can be used to accurately identify each extracted image area. The semantic segmentation model classifies each pixel in the image, thereby accurately dividing the outline and category of the device. The model outputs the category label for each pixel and generates an accurate boundary outline of the device. These category labels can be predefined device types, such as "light bulb", "speaker", "door lock", etc. The category label and precise location information of each device are extracted from the semantic segmentation results. The precise location information includes the pixel-level boundary outline of the device, which is more accurate than the bounding box of the preliminary detection.

[0087] It can be understood that in the present invention, the AOA and AOD information of the scanned Bluetooth device can be used to

[0088] The coordinates of the Bluetooth devices in the device layout image are obtained by using the RSSI, and the Bluetooth devices that can be connected by Bluetooth are screened out from the electronic devices identified in the image.

[0089] In this embodiment, it can be understood that the two-dimensional coordinates (x cam ,y cam ) is step S300, performing preliminary recognition on the device arrangement image through a target detection algorithm, and obtaining the coordinates of the center of the bounding box of each Bluetooth device in the step of obtaining the coordinates of the bounding box of each Bluetooth device.

[0090] In this embodiment, the distance between each Bluetooth device and the camera in the device layout image is obtained, and the distance between each Bluetooth device and the camera is used as the z-axis coordinate of the Bluetooth device in the camera coordinate system, thereby converting the two-dimensional coordinate of each Bluetooth device in the device layout image into the three-dimensional coordinate (x-axis coordinate) of the Bluetooth device in the camera coordinate system. cam ,y cam , z cam ), so that the three-dimensional coordinates (x cam ,y cam , z cam ) Generate an interactive object at a position corresponding to the device layout image. The user can interact through the interactive object superimposed on the position corresponding to the device layout image, thereby realizing the interaction of multiple Bluetooth devices without the need to drop down a list or open a special APP for interaction.

[0091] In one possible implementation, reference Figure 2 , Figure 2 A flowchart of another embodiment of the multi-terminal Bluetooth interaction method of the present application. In step S500, the distance between each Bluetooth device and the camera in the device layout image is obtained, and the two-dimensional coordinates of each Bluetooth device in the device layout image are converted into the three-dimensional coordinates (x cam ,y cam , z cam） The steps include:

[0092] Step S510, obtaining the internal parameters of the camera and the pixel coordinates (u, v) in the Bluetooth device image coordinate system, wherein the internal parameters include the focal length (f x , f y ) and the optical center (cx , c y );

[0093] Step S520, obtaining the three-dimensional coordinates (x, v) of the Bluetooth device in the camera coordinate system according to the internal parameters of the camera and the pixel coordinates (u, v) in the Bluetooth device image coordinate system. cam ,y cam , z cam );

[0094] The three-dimensional coordinates (x cam ,y cam , z cam ) is obtained according to the following formula:

[0095] z cam =d(u,v)

[0096] x cam =(u−c x )×z cam / f x

[0097] y cam =(v−c y )×z cam / f y

[0098] Among them, z cam is the depth of field of the Bluetooth device or ,

[0099] The distance between the Bluetooth device and the camera is estimated by the received signal strength RSSI.

[0100] The camera coordinate system is a concept in computer vision that defines the position and orientation of an object relative to the camera so that points in the image can be mapped to positions in three-dimensional space. The origin of the camera coordinate system is located at the optical center of the camera, usually the center of the camera lens. It should be noted that in this application, the origin of the camera coordinate system is located at the optical center of the main camera of the smart terminal. The z-axis of the camera coordinate system points to the front of the camera, along the optical axis of the lens. This axis represents the distance of the object from the camera. The larger the z value, the farther the object is from the camera. The x-axis is parallel to the image plane and horizontally to the right. The y-axis is also parallel to the image plane and vertically downward.

[0101] In this embodiment, the intrinsic parameters of the camera are the intrinsic parameters of the main camera, and the focal length (fx, fy) is the focal length of the main camera in the x and y directions, usually in pixels. x , c y) is the optical center (principal point) of the image plane, and in this embodiment, it is the pixel coordinate of the center point of the camera sensor.

[0102] The pixel coordinates (u, v) in the image coordinate system represent the position of the Bluetooth device on the device layout image.

[0103] z cam The depth of field of the Bluetooth device or the distance between the Bluetooth device and the camera obtained according to the received signal strength indicator value , which can be directly obtained from the device layout image, corresponds to the z-axis position of the Bluetooth device in the camera coordinate system.

[0104] In a feasible implementation manner, the step S300 obtains the distance between each of the Bluetooth devices in the device layout image and the camera, and converts the two-dimensional coordinates of each of the Bluetooth devices in the device layout image into the three-dimensional coordinates (x , m ) of the Bluetooth device in the camera coordinate system according to the distance between each of the Bluetooth devices and the camera. cam ,y cam , z cam ), the depth of field of each of the Bluetooth devices in the device arrangement image is obtained by a depth of field calculation algorithm, and the depth of field of each of the Bluetooth devices is used as the distance between each of the Bluetooth devices and the camera.

[0105] Since most existing smartphones are equipped with camera modules with multiple cameras, in this embodiment, a smartphone equipped with a camera module with multiple cameras can be used to take pictures to obtain pictures with depth of field information, that is, to obtain the device layout image, and then obtain the depth of field of the Bluetooth device based on the device layout image.

[0106] Specifically, two or more cameras may be used to capture images of the same scene, and the depth of field of the Bluetooth device may be obtained by calculating the parallax between the images. This method is similar to the working principle of the human eye, and a depth map may be created.

[0107] It is understandable that for smart devices with only one camera, it is also possible to use a single camera to capture images and obtain depth information through machine learning algorithms and neural network models, such as depth learning. These models are trained with a large amount of data and can infer the depth information of the image. For example, Google Pixel's "single camera depth of field" technology can be used. Using depth information, augmented reality applications can more accurately integrate virtual objects with real scenes.

[0108] In this embodiment, the depth of field calculation algorithm can select a stereo vision algorithm, a monocular depth estimation algorithm or a depth of field estimation algorithm according to the performance of the smart terminal camera module to obtain the depth of field of each Bluetooth device in the device layout image to obtain the distance between each Bluetooth device and the camera.

[0109] The stereo vision algorithm is a classic method for depth calculation based on binocular cameras. It obtains left and right images of the same scene through two cameras and uses parallax to infer depth.

[0110] Specifically, two images of the same scene are first acquired through two cameras at different locations. Then, computer vision algorithms (such as SIFT, SURF, or ORB) are used to identify feature points in the images and match them in the two images. The parallax is then calculated based on the displacement of the matched feature points in the two images. Finally, the depth of each pixel is calculated using the parallax value and the baseline distance between the cameras.

[0111] The monocular depth estimation algorithm is an image depth estimation method based on deep learning. It only uses a single 2D image and predicts the depth of field in the image through a neural network model.

[0112] The depth estimation algorithm estimates the distance of an object from the camera based on the blur level of the image (such as Laplace transform or Gaussian blur). The depth of each pixel is calculated using a pre-calibrated relationship model between blur and depth of field.

[0113] In a feasible implementation manner, the step S300 obtains the distance between each of the Bluetooth devices in the device layout image and the camera, and converts the two-dimensional coordinates of each of the Bluetooth devices in the device layout image into the three-dimensional coordinates (x , m ) of the Bluetooth device in the camera coordinate system according to the distance between each of the Bluetooth devices and the camera. cam ,y cam , z cam ), by acquiring the RSSI value of the Bluetooth device, the distance between each Bluetooth device and the camera is obtained according to the RSSI value;

[0114] The distance between each Bluetooth device and the camera is obtained according to the following formula:

[0115] .

[0116] In this formula:

[0117] It is the estimated distance between the Bluetooth device and the camera estimated by the received signal strength RSSI;

[0118] It is the reference RSSI value, usually measured at a distance of 1 meter;

[0119] RSSI is the current received signal strength (dBm);

[0120] n is the environmental attenuation factor, usually between 2 and 4, depending on the complexity of the environment.

[0121] In this embodiment, the distance between each Bluetooth device and the camera can be obtained by obtaining the RSSI value of the Bluetooth device. , and The value of z cam , and then determine the three-dimensional coordinates (x cam ,y cam , z cam ).

[0122] It is understandable that in some embodiments of the present invention, it is also possible to combine and the depth of field of the Bluetooth device to perform weighted averaging to obtain a more accurate positioning result. It can help estimate the horizontal distance between the device and the object, while the depth of field information can help determine the vertical or front-to-back position of the object in the scene. By weighted averaging these two types of information, the position of the object can be determined more accurately and ensure that the virtual object can be superimposed correctly.

[0123] Specifically, The weights of depth of field DoF are and ;

[0124]

[0125] At shorter distances, D RSSI Generally more reliable. If the signal strength difference between two Bluetooth devices is significant, a higher weight can be given. If the depth of field data is obtained through a computer vision algorithm, the weight can be reduced when it is greatly affected by ambient light or object material. and Can be obtained through experiments.

[0126] In one possible implementation, reference Figure 3 , Figure 3A flowchart of another embodiment of the multi-terminal Bluetooth interaction method of the present application. The Bluetooth information includes the device category, and the step S500 obtains the distance between each Bluetooth device and the camera in the device layout image, and converts the two-dimensional coordinates of each Bluetooth device in the device layout image into the three-dimensional coordinates (x cam ,y cam , z cam ) steps include:

[0127] S530: Match the corresponding device category with the category label, and generate an interactive object at a corresponding position of the device layout image.

[0128] In this embodiment, illustratively, when the application scenario involves daily device use, the device category covers a variety of devices such as headphones, speakers, printers, keyboards, lighting equipment, door locks, etc. In the Bluetooth technical specification, the category of Bluetooth devices is identified by a 24-bit (3-byte) Class of Device (CoD) field, which describes in detail the major category, minor category, and services provided by the device. The major category (Major Device Class) refers to the major category to which the device belongs, such as computers, mobile phones, audio devices, etc. The minor category (Minor Device Class) further refines the specific type of the device. For example, in computer-type devices, it can be further subdivided into desktops, laptops, etc. The service category (Service Class) indicates the type of service supported by the device, such as audio transmission, network access, etc.

[0129] This 24-bit field is divided into different parts, each part represents different information: The first 6 bits are the service category, which is used to indicate the type of service supported by the device, such as audio, network, etc. The next 5 bits are the major device category, which is used to indicate which large category the device belongs to, such as audio equipment, computer equipment, etc. The next 6 bits are the minor device category, which is used to further refine the device type. The last 7 bits are currently reserved or used for expansion.

[0130] In this embodiment, when a Bluetooth device starts and begins broadcasting, for example, when it can be discovered by other devices, it sends its CoD information in the broadcast packet. When the smart terminal receives this broadcast packet, it reads the CoD field in it to obtain the category of the device. Then, the obtained device category is matched with the device corresponding to the category label obtained by the semantic segmentation model, and the three-dimensional coordinates (x cam ,y cam , z cam）An interactive object is generated at a corresponding position of the device layout image. In this way, the interactive object can be superimposed at the corresponding position of each device in the device layout image. When there are multiple devices around, there is no need to search for the device in a long list of Bluetooth devices. Just click the corresponding interactive object to intuitively interact with multiple Bluetooth devices.

[0131] In one possible implementation, reference Figure 4 , Figure 4 A flowchart diagram of another embodiment of the multi-terminal Bluetooth interaction method of the present application is provided. The method comprises:

[0132] S800, obtaining Bluetooth information of hidden Bluetooth devices in the device layout image;

[0133] S900: Determine the position of the hidden Bluetooth device according to the Bluetooth information of the hidden Bluetooth device, and generate an interactive object at a position corresponding to the device layout image.

[0134] Since the device layout image is an image captured in real time by a device camera or a static image taken in advance, the device layout image shows the scene in the actual environment, but there may be invisible Bluetooth devices therein, for example, the device may be blocked by furniture.

[0135] In this embodiment, the hidden Bluetooth device is a Bluetooth device that is invisible or blocked in the image. These devices broadcast their existence and location through Bluetooth signals. Although they cannot be seen by the naked eye, their information can be captured through Bluetooth scanning. The location of the hidden Bluetooth device can be determined based on the Bluetooth information of the hidden Bluetooth device, and an interactive object is generated at the corresponding position of the device layout image.

[0136] Users can select Bluetooth devices that are invisible in the image but actually exist to interact with based on their memory of the environment and their understanding of the device's location. This interaction method can further enhance the user experience.

[0137] In this embodiment, the method of generating an interactive object by acquiring the information of the hidden Bluetooth device to obtain the position of the hidden Bluetooth device and mapping the position to the device layout image can enable the user to interact with the hidden Bluetooth device more intuitively in an actual scene.

[0138] In one possible implementation, reference Figure 5 , Figure 5 A flowchart of another embodiment of the multi-terminal Bluetooth interaction method of the present application is provided. The Bluetooth information includes device category, angle of arrival and received signal strength RSSI, the step S900, the step of generating an interactive object at a position corresponding to the device layout image according to the position information of the hidden Bluetooth device, comprises:

[0139] S910, according to the arrival angle Identify the Bluetooth device located in front of the camera;

[0140] S920, determining a hidden Bluetooth device in the device arrangement image according to the identified Bluetooth device and the Bluetooth device located in front of the camera;

[0141] S930, according to the arrival angle of the hidden Bluetooth device The two-dimensional coordinates (x device ,y device );

[0142] S940, according to the two-dimensional coordinates (x device ,y device ) and device category generate interactive objects of hidden Bluetooth devices at corresponding positions in the device layout image.

[0143] In this embodiment, the angle of arrival (AoA) refers to the angle at which the Bluetooth signal reaches the receiver. By measuring the angle of arrival of the signal, the direction of the Bluetooth device can be determined. If the Bluetooth device supports the AoA function, the angle of arrival of the Bluetooth device signal can be calculated to help determine the position of the Bluetooth device relative to the smart terminal, such as in front of or to the side of the smart terminal, thereby determining the Bluetooth device in front of the camera in the scanned Bluetooth device list.

[0144] In this embodiment, the step S930 determines the two-dimensional coordinates (x , y ) of the hidden Bluetooth device in the camera coordinate system according to the arrival angle and the received signal strength RSSI of the hidden Bluetooth device. device ,y device ) steps: Assume that the hidden Bluetooth device in the device layout image is located at the same horizontal plane as the camera;

[0145] The distance between the device and the receiver is obtained through the RSSI value according to the following formula:

[0146]

[0147] The two-dimensional coordinates (x device ,y device ) is obtained according to the following formula:

[0148]

[0149]

[0150] In this application, the two-dimensional coordinates (x device ,y device ), according to the two-dimensional coordinates (x device ,y device ) and device category generate interactive objects of hidden Bluetooth devices at corresponding positions in the device layout image.

[0151] Users can select Bluetooth devices that are not visible in the image but actually exist to interact with based on their memory of the environment and their understanding of the device's location.

[0152] The multi-terminal Bluetooth interaction method proposed in this application obtains Bluetooth information of nearby Bluetooth devices by scanning nearby Bluetooth devices, and then obtains a device layout image, wherein the device layout image includes images of multiple Bluetooth devices, obtains the category label and location information of each Bluetooth device according to the device layout image, generates an interaction object at the corresponding position of the device layout image according to the category label, location information and Bluetooth information of each Bluetooth device, and finally interacts with the Bluetooth device in response to the triggering of the interaction object. In this way, when there are multiple devices around, this solution does not need to search for devices in a long list of Bluetooth devices, and can intuitively interact with multiple Bluetooth devices.

[0153] The present application also provides a Bluetooth interaction device, which adopts the multi-terminal Bluetooth interaction method as described in any of the above items.

[0154] The Bluetooth interaction device provided by the present application adopts the multi-terminal Bluetooth interaction method in the above embodiment, which can solve the technical problem of inconvenient interaction of multiple Bluetooth devices in the prior art. Compared with the prior art, the beneficial effects of the Bluetooth interaction device provided by the present application are the same as the beneficial effects of the multi-terminal Bluetooth interaction method provided by the above embodiment, and the other technical features in the vehicle positioning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0155] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0156] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0158] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-terminal Bluetooth interaction method, characterized in that: The method includes: Scan nearby Bluetooth devices and obtain Bluetooth information of nearby Bluetooth devices; Acquire a device layout image, wherein the device layout image includes images of multiple Bluetooth devices; Performing preliminary recognition on the device arrangement image by using a target detection algorithm to obtain the bounding box coordinates of each of the Bluetooth devices; Perform secondary recognition on the device layout image through a semantic segmentation model to obtain the precise boundary contour and category label of each Bluetooth device, and obtain the two-dimensional coordinates of each Bluetooth device in the device layout image; Obtain the distance between each Bluetooth device and the camera in the device layout image, and convert the two-dimensional coordinates of each Bluetooth device in the device layout image into the three-dimensional coordinates (x cam ,y cam , z cam ); According to the three-dimensional coordinates (x cam ,y cam , z cam ) generating an interactive object at a position corresponding to the image arranged on the device; interacting with the Bluetooth device in response to triggering of the interactive object; The obtaining of the distance between each of the Bluetooth devices in the device layout image and the camera, and converting the two-dimensional coordinates of each of the Bluetooth devices in the device layout image into the three-dimensional coordinates (x cam ,y cam , z cam The steps include: Obtain the camera's internal parameters and the pixel coordinates (u, v) in the Bluetooth device image coordinate system, the internal parameters including the focal length (f x , f y ) and the optical center (c x , c y ); The three-dimensional coordinates (x, v) of the Bluetooth device in the camera coordinate system are obtained according to the intrinsic parameters of the camera and the pixel coordinates (u, v) in the Bluetooth device image coordinate system. cam ,y cam , z cam ); The three-dimensional coordinates (x cam ,y cam , z cam ) is obtained according to the following formula: x cam =(u−c x )×z cam / f x y cam =(v−c y )×z cam / f y in, is the depth of field of the Bluetooth device, The distance between the Bluetooth device and the camera is estimated by the received signal strength RSSI. and They are and the weight of the depth of field DoF.

2. The method according to claim 1, characterized in that The obtaining of the distance between each of the Bluetooth devices in the device layout image and the camera, and converting the two-dimensional coordinates of each of the Bluetooth devices in the device layout image into the three-dimensional coordinates (x cam ,y cam , z cam ) steps: The depth of field of each Bluetooth device in the device arrangement image is acquired by a depth of field calculation algorithm, and the depth of field of each Bluetooth device is used as the distance between each Bluetooth device and the camera.

3. The method according to claim 2, characterized in that The depth of field calculation algorithm includes a stereo vision algorithm, a monocular depth estimation algorithm and a depth of field estimation algorithm.

4. The method according to claim 3, characterized in that The obtaining of the distance between each of the Bluetooth devices in the device layout image and the camera, and converting the two-dimensional coordinates of each of the Bluetooth devices in the device layout image into the three-dimensional coordinates (x cam ,y cam , z cam ) steps: By obtaining the RSSI value of the Bluetooth device, the distance between each Bluetooth device and the camera is obtained according to the RSSI value ; Wherein, the distance between each of the Bluetooth devices and the camera Obtained according to the following formula: ； in, The distance between the Bluetooth device and the camera is estimated by the received signal strength RSSI; is the reference RSSI value; RSSI is the current received signal strength; n is the environmental attenuation factor.

5. The method according to claim 1, characterized in that The Bluetooth information includes a device category, and the three-dimensional coordinates (x cam ,y cam , z cam ) The step of generating an interactive object at a position corresponding to the image arranged on the device comprises: Match the corresponding device class to the class label.

6. The method according to claim 1, characterized in that The method includes: Get the Bluetooth information of hidden Bluetooth devices in the device layout image; The position of the hidden Bluetooth device is determined according to the Bluetooth information of the hidden Bluetooth device, and an interactive object is generated at a position corresponding to the device layout image.

7. The method according to claim 6, characterized in that The Bluetooth information includes device category, arrival angle and received signal strength RSSI, the step of determining the position of the hidden Bluetooth device according to the Bluetooth information of the hidden Bluetooth device, and generating an interactive object at a position corresponding to the device layout image comprises: According to the arrival angle Identify the Bluetooth device located in front of the camera; determining a hidden Bluetooth device in the device arrangement image according to the identified Bluetooth device and the Bluetooth device located in front of the camera; According to the arrival angle of the hidden Bluetooth device The two-dimensional coordinates (x device ,y device ); According to the two-dimensional coordinates (x device ,y device ) and device category generate interactive objects of hidden Bluetooth devices at corresponding positions in the device layout image.

8. The method according to claim 7, characterized in that The method determines the two-dimensional coordinates (x ) of the hidden Bluetooth device in the camera coordinate system according to the arrival angle and the received signal strength RSSI of the hidden Bluetooth device. device ,y device ) steps: Assume that the hidden Bluetooth device in the device layout image is located at the same horizontal plane as the camera; The distance between the device and the receiver is obtained through the RSSI value. The formula is: in, The distance between the Bluetooth device and the camera is estimated by the received signal strength RSSI; is the reference RSSI value; RSSI is the current received signal strength; n is the environmental attenuation factor; The two-dimensional coordinates (x device ,y device ) is obtained according to the following formula: 。

Images