Projection method, apparatus, electronic device, and readable storage medium

Abstract

The application discloses a projection method and device, electronic equipment and readable storage medium, and belongs to the communication technical field. The method is applied to a first device, the first device can show a virtual display area, the virtual display area includes a preset area, the first device is in communication connection with a second device, and the method comprises the following steps: receiving a first input of the second device in the preset area; and in response to the first input, projecting display content of the second device to the virtual display area.

Description

Technical Field

[0001] This application belongs to the field of communication technology, and specifically relates to a projection method, apparatus, electronic device and readable storage medium. Background Technology

[0002] With the rapid development of smart terminal devices, such as smart glasses, new forms and technologies of smart glasses, such as Augmented Reality (AR) and Mixed Reality (MR), have emerged. AR smart glasses present scenes and people that are partly real and partly virtual, bringing virtual information into the real world. MR smart glasses combine Virtual Reality (VR) and AR technologies.

[0003] AR smart glasses and MR devices can enable interaction between users' smart devices and virtual screens to a certain extent. However, users must first enter a preset interface, then manually select and pair the devices to establish interconnection between the smart device and the smart glasses. Only then can the data to be interacted with be projected onto the interconnected smart glasses. The projection process is cumbersome and complex. For users, the augmented reality effect is not realistic enough. How to simplify the projection process and make "augmented reality" more realistic is a technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a projection method, apparatus, electronic device, and readable storage medium that can solve the problem that the augmented reality effect of existing smart devices is not realistic enough.

[0005] In a first aspect, embodiments of this application provide a projection method applied to a first device, the first device being able to display a virtual display area, the virtual display area including a preset area, the first device being communicatively connected to a second device, the projection method comprising: receiving a first input from the second device in the preset area; and responding to the first input by projecting the display content of the second device onto the virtual display area.

[0006] Secondly, embodiments of this application provide a projection device applied to a first device, the first device being able to display a virtual display area, the virtual display area including a preset area, the first device being communicatively connected to a second device, the projection device including: a receiving module for receiving a first input from the second device in the preset area; and a projection module for responding to the first input by projecting the display content of the second device onto the virtual display area.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, the projection method is applied to a first device, which can display a virtual display area, including a preset area. The first device is communicatively connected to a second device. By receiving a first input from the second device in the preset area, and responding to the first input, the display content of the second device is projected onto the virtual display area. Thus, by using an interactive operation method where the second device touches the virtual display area of ​​the first device, the display content of the second device can be directly projected onto the second device, greatly simplifying the projection process, improving the interaction efficiency between devices, and making the "augmented reality" of the first device more realistic. This allows the user of the first device to experience a more realistic experience in the virtual projection screen, improving the user experience of using the first device. Attached Figure Description

[0012] Figure 1 This is a schematic flowchart of the projection method according to an embodiment of this application.

[0013] Figure 2 This is a schematic diagram illustrating an application scenario of the projection method according to an embodiment of this application.

[0014] Figure 3 This is a user operation diagram of the second device according to an embodiment of this application.

[0015] Figure 4 This is a schematic diagram of the projection system according to an embodiment of this application.

[0016] Figure 5 This is a schematic diagram of the projection device according to an embodiment of this application.

[0017] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0018] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that implements an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The projection methods, devices, systems, and electronic devices provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0022] In one embodiment of this application, the projection method is applied to a first device, which can display a virtual display area, the virtual display area including a preset area, the first device being communicatively connected to a second device, and the projection method including: receiving a first input from the second device in the preset area; and in response to the first input, projecting the display content of the second device onto the virtual display area.

[0023] Now for reference Figure 1 , Figure 1 This is a schematic flowchart of a projection method according to an embodiment of the present application, in which the projection method is applied to a first device.

[0024] The first device can be smart glasses, such as AR smart glasses or MR smart glasses. Through corresponding augmented reality technology, a virtual display area is projected to be displayed to the user wearing the smart glasses. For example, the virtual display area displays a virtual computer screen or other virtual screens. The user can operate the virtual display area through the smart glasses or by operating the terminal device associated with the smart glasses.

[0025] The preset area of ​​the virtual display area is used to trigger the first device to perform a projection operation. The first device's projection operation is triggered by input from the second device within the preset area. The second device can be a mobile electronic device, such as a terminal device.

[0026] like Figure 1 As shown, the projection method of this application embodiment includes the following steps 102 to 104.

[0027] Step 102: Receive the first input from the second device in the preset area.

[0028] The first input is the user holding the second device, who operates on a preset area of ​​the virtual display area shown by the first device through the second device, such as the second device touching the preset area of ​​the first device.

[0029] Step 104: In response to the first input, project the display content of the second device onto the virtual display area.

[0030] In other words, if the second device touches a preset area of ​​the first device, it means that the user holding the second device wants to transfer the display content of the second device to the first device and project it onto the virtual display area of ​​the first device. In response to this touch operation, the display content of the second device is projected onto the virtual display area of ​​the first device.

[0031] If the second device has input in the virtual display area of ​​the first device, but the input is not in the preset area, the above projection operation will not be performed.

[0032] Optionally, before the first device receives the first input from the second device in the preset area, it further includes: capturing video frames of the virtual display area when the second device makes the target input using the camera; identifying the actual position of the second device when making the target input in the virtual display area based on the video frames; determining whether the actual position falls within the preset area; and determining that the target input is the first input in the preset area if the actual position falls within the preset area.

[0033] In the above embodiments, the camera of the first device monitors whether there is a touch operation by the second device on a preset area of ​​the first device.

[0034] The camera of the first device monitors the virtual display area projected by the first device in order to monitor the user's operations in the virtual display area and determine whether user interaction is required in response to the operations.

[0035] For example, for a first device like smart glasses, cameras can be set on one or both sides of the smart glasses to capture video frames of the virtual display area. Based on the images of the video frames, it can identify whether the second device touches the virtual display area of ​​the first device, and whether the area where the touch occurs is a preset area.

[0036] Optionally, identifying the actual position of the second device when it performs target input in the virtual display area based on video frames includes: extracting features of the second device, features of the virtual display area, and position change features of the second device relative to the virtual display area from the video frames using a Simultaneous Localization and Mapping (SLAM) algorithm; and identifying whether the second device operates at the actual position in the virtual display area by inputting the features and position change features of the second device into a neural network recognition model. The neural network recognition model is trained based on sample data including the features and position change features of the second device and labels indicating whether the second device operates at the target position in the virtual display area.

[0037] Simultaneous Localization and Mapping (SLAM) algorithms extract typical features of objects, such as the feature dataset of the second device and the feature dataset of the virtual display area. The SLAM algorithm also extracts the position change features of the second device relative to the virtual display area. These features are input into a pre-trained neural network recognition model for recognition, thereby determining whether there is an action of the second device touching the virtual display area, and the actual position of the virtual display area touched by the second device.

[0038] If a second device is detected to be touching the virtual display area, it is further determined whether the actual position corresponding to the touch operation falls within a preset area of ​​the virtual display area.

[0039] Optionally, determining whether the actual position falls within a preset area includes: determining a first orientation of the actual position relative to the first device, the orientation including direction and three-dimensional coordinates; comparing the first orientation with a preset second orientation, the second orientation being the orientation of the preset position in the preset area relative to the first device; and determining that the actual position falls within the preset area if the angle difference between the direction of the first orientation and the direction of the second orientation meets a first angle threshold and the coordinate difference between the three-dimensional coordinates of the first orientation and the three-dimensional coordinates of the second orientation meets a first coordinate threshold.

[0040] In this embodiment, the preset position can be a reference point within a preset area, such as the center point of the preset area. The second orientation is the orientation of the reference point relative to the first device, that is, the three-dimensional coordinates of the reference point and its direction angle relative to the first device. The preset area is considered to be within a preset range surrounding the reference point.

[0041] After determining the actual position of the second device in the virtual display area, the coordinate difference and angle difference between the first and second positions can be determined by comparing the actual position relative to the first device with the reference point position relative to the first device. If the coordinate difference and angle difference do not exceed the maximum difference between the reference point and other positions within the preset area, i.e., the preset coordinate threshold and preset angle threshold, it can be determined that the actual position falls within the preset area, that is, it is determined that the second device has made the first input to the preset area of ​​the first device; otherwise, it is determined that the second device has not made the first input to the preset area of ​​the first device.

[0042] The preset position on the preset area is known in advance relative to the second position of the first device, while the actual position is determined by the first device when inputting in the virtual display area.

[0043] Optionally, determining the first orientation of the actual position relative to the first device includes: determining the first orientation of the actual position relative to the first device based on the parallax between the lateral coordinates of the actual position imaged on the corresponding views of the two cameras of the first device and the distance from the actual position to the imaging plane.

[0044] Using a binocular ranging algorithm, the position of the second device touching the virtual display area is determined relative to the orientation of the first device. In this embodiment, a camera is set on each of the left and right sides of the first device. The touch position will be imaged on the corresponding views of the two cameras respectively. The parallax between the horizontal coordinates of the touch position in the two image views is calculated, and combined with the distance from the touch position to the image plane, the orientation of the touch position relative to the first device can be calculated.

[0045] The preset area is a pre-defined area used by the second device to trigger data interaction operations and project displayed content. Therefore, the orientation of the preset position within the preset area relative to the first device is known. Of course, the orientation of the preset position relative to the first device can also be determined using the aforementioned binocular ranging algorithm.

[0046] like Figure 2 As shown, when a user wearing smart glasses 20 (i.e., the first device) holds a terminal device 10 (i.e., the second device) and performs a "touch" operation towards a preset area B in the virtual display area A projected by smart glasses 20, the camera 22 on smart glasses 20 can detect this action of the terminal device 10 touching the preset area B in the virtual display area A. In other words, smart glasses 20 can determine that the user requests the display content of the terminal device 10 to be projected onto the virtual display area A of smart glasses 20.

[0047] After identifying that the second device has touched a preset area within the virtual display area of ​​the first device through the above steps, in step 104, in response to the first input, the display content of the second device can be projected onto the virtual display area of ​​the first device. Before projection, the first device needs to identify the second device that made the first input and communicate with the second device to receive the display content that the first device needs to project.

[0048] There may be multiple second devices around the first device. The first device needs to identify the actual second device that touches the preset area of ​​the first device from among the surrounding second devices.

[0049] Optionally, before projecting the display content of the second device onto the virtual display area in response to the first input, the method further includes: determining a third position of each device relative to the first device within a preset distance of the first device; comparing the direction of each third position with the direction of the second position, and comparing the three-dimensional coordinates of each third position with the three-dimensional coordinates of the second position; and identifying the second device from among the devices if the angle difference between the direction of the third position of a device and the direction of the second position satisfies a second angle threshold and the coordinate difference between the three-dimensional coordinates of the third position and the three-dimensional coordinates of the second position satisfies a second coordinate threshold.

[0050] Identifying a second device that performs a first input within a preset area of ​​the first device can be achieved by considering the orientation of the second device relative to the first device.

[0051] Optionally, determining the third position of each device within a preset distance of the first device relative to the first device includes: determining the third position of each device within a preset distance of the first device relative to the first device using ultra-wideband (UWB) positioning technology.

[0052] Ultra Wide Band (UWB) can achieve centimeter-level accuracy positioning indoors. UWB technology can identify the location and orientation of multiple devices relative to the first device in real time.

[0053] The preset distance of the first device can be the range covered by the UWB base station, and both the first device and each other have UWB communication enabled, thus enabling interconnection through UWB communication. When the first device detects that a device has touched the preset area, it is triggered to use UWB positioning technology to find the surrounding devices and calculate the position of each device relative to the first device, including three-dimensional coordinates and direction.

[0054] Then, the third position of each device relative to the first device is compared with the preset second position, which is the position of the preset position on the preset area of ​​the first device relative to the first device. That is, the direction of each third position is compared with the direction of the second position, and the three-dimensional coordinates of each third position are compared with the three-dimensional coordinates of the second position. This calculates the angle difference and the three-dimensional coordinate difference between the second position and the corresponding third position of each device. If both the angle difference and the three-dimensional coordinate difference meet the corresponding preset thresholds, it indicates that the third position of a certain device is very close to the second position. It can then be determined that the device is the second device that touches the preset area in the virtual display area, thus identifying the second device that performs the first input in the preset area.

[0055] UWB technology enables accurate identification of the second device, ensuring that the display content data of the second device can be accurately transmitted to the first device, rather than to other unspecified devices.

[0056] Optionally, in step 104, in response to the first input, projecting the display content of the second device onto the virtual display area includes: in response to the first input, sending a notification to the second device to trigger the second device to send data corresponding to the display content; receiving the data and projecting it onto the virtual display area.

[0057] After the second device is identified, the first device can then interact with the second device.

[0058] The first device sends a notification to the corresponding identified second device. Upon receiving the notification, the second device knows that it can now project display content onto the first device, and thus can send the display content data to be projected to the first device. The interaction between the first and second devices can transmit data via UWB communication or WIFI communication.

[0059] like Figure 4 As shown, both the second device 10 and the first device 20 are equipped with UWB communication interfaces and WIFI communication interfaces. When the corresponding communication interfaces are enabled, the two devices can communicate with each other through UWB or WIFI in the corresponding modes to achieve data interaction.

[0060] After receiving the display content data transmitted by the second device, the first device can project the display content data onto the virtual display area or share it with the first device, depending on the data interaction mode selected by the second device. Alternatively, the second device can choose not to select a data interaction mode and, for example, perform a tap operation, directly project the display content data transmitted by the second device onto the virtual display area of ​​the first device.

[0061] When multiple data interaction modes exist, the second device can select the mode to be executed, including data projection mode and data sharing mode. If the second device selects data projection mode, the first device will project the data onto the virtual display area after receiving the corresponding display content data. If the second device selects data sharing mode, the first device will save the data after receiving the corresponding display content data.

[0062] Users can select the corresponding data interaction mode through the user interface (UI) of the second device. For example... Figure 3 As shown, users can determine the data interaction mode or function triggered by touching a preset area by clicking the "Share" or "Cast" control on the UI. For example... Figure 3 After the UI displays image 12, if the user selects the "Share" control, the data interaction mode is set to data sharing mode, thus transmitting image 12 to the first device for saving; if the user selects the "Cast" control, the data interaction mode is set to data voting mode, thus projecting the display content of image 12 onto the virtual display area of ​​the first device.

[0063] Thus, by touching the preset area B of the virtual display area of ​​the first device with the second device 10, the image 12 displayed on the second device 10 can be transmitted to the first device 20, thereby projecting the image 12 onto the virtual display area of ​​the first device 20.

[0064] In this embodiment, the projection method is applied to a first device, which can display a virtual display area, including a preset area. The first device is communicatively connected to a second device. By receiving a first input from the second device in the preset area, the first device projects the display content of the second device onto the virtual display area in response to the first input. Thus, by using the interactive operation method of the real second device touching the virtual display area of ​​the first device, the display content of the second device can be directly projected onto the second device, which greatly simplifies the projection process, improves the interaction efficiency between devices, and makes the "augmented reality" of the first device more realistic. This allows the user of the first device to experience a more realistic experience in the virtual projection screen, improving the user experience of using the first device.

[0065] The projection method provided in this application can be executed by a projection device. This application uses a projection device executing the projection method as an example to illustrate the projection device provided in this application.

[0066] like Figure 5As shown, this application embodiment provides a projection device 800, applied to a first device. The first device can display a virtual display area, which includes a preset area. The first device is communicatively connected to a second device. The projection device includes: a receiving module 820, used to receive a first input from the second device in the preset area; and a projection module 840, used to project the display content of the second device onto the virtual display area in response to the first input.

[0067] Optionally, the first device is equipped with a camera, and the projection device 800 further includes: a determining module, used to capture video frames of the virtual display area when the second device makes a target input through the camera before receiving the first input from the second device in the preset area; identify the actual position of the second device when making the target input in the virtual display area based on the video frames; determine whether the actual position falls within the preset area; and determine that the target input is the first input of the preset area if the actual position falls within the preset area.

[0068] Optionally, the determining module is specifically used to: determine the first orientation of the actual position relative to the first device, the orientation including direction and three-dimensional coordinates; compare the first orientation with a preset second orientation, the second orientation being the orientation of a preset position on a preset area relative to the first device; and determine that the actual position falls within the preset area if the angle difference between the direction of the first orientation and the direction of the second orientation meets a first angle threshold and the coordinate difference between the three-dimensional coordinates of the first orientation and the three-dimensional coordinates of the second orientation meets a first coordinate threshold.

[0069] Optionally, the determining module is specifically used to: extract features of the second device, features of the virtual display area, and position change features of the second device relative to the virtual display area from the video frame using a Simultaneous Localization and Mapping (SLAM) algorithm; and identify whether the second device operates the actual position in the virtual display area by inputting the features and position change features of the second device into a neural network recognition model. The neural network recognition model is trained based on sample data including the features and position change features of the second device and the label of whether the second device operates the target position in the virtual display area.

[0070] Optionally, the determining module is specifically used to: determine the first orientation of the actual position relative to the first device based on the parallax between the lateral coordinates of the actual position imaged on the corresponding views of the two cameras of the first device and the distance from the actual position to the imaging plane.

[0071] Optionally, the projection device 800 further includes: an identification module, configured to, before projecting the display content of the second device onto the virtual display area in response to the first input, determine a third position of each device within a preset distance of the first device relative to the first device; compare the direction of each third position with the direction of the second position, and compare the three-dimensional coordinates of each third position with the three-dimensional coordinates of the second position; and identify the second device from among the devices if the angle difference between the direction of the third position of a device and the direction of the second position satisfies a second angle threshold, and the coordinate difference between the three-dimensional coordinates of the third position and the three-dimensional coordinates of the second position satisfies a second coordinate threshold.

[0072] Optionally, the identification module is specifically used to: determine the third position of each device relative to the first device within a preset distance using ultra-wideband (UWB) positioning technology.

[0073] Optionally, the projection module 840 is specifically used to: in response to the first input, send a notification to the second device to trigger the second device to send data corresponding to the display content; receive the data and project it onto the virtual display area.

[0074] In this embodiment, the projection device is applied to a first device, which can display a virtual display area, including a preset area. The first device is communicatively connected to a second device. By receiving a first input from the second device in the preset area, and responding to the first input, the display content of the second device is projected onto the virtual display area. Thus, by using the interactive operation method of the real second device touching the virtual display area of ​​the first device, the display content of the second device can be directly projected onto the second device, greatly simplifying the projection process, improving the interaction efficiency between devices, and making the "augmented reality" of the first device more realistic. This allows the user of the first device to experience a more realistic experience in the virtual projection screen, improving the user experience of using the first device.

[0075] The projection device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and can also be a personal computer (PC), etc. This application embodiment does not specifically limit the specific device.

[0076] The projection device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0077] The projection device provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0078] Optionally, such as Figure 4 As shown, this application embodiment also provides a projection system 100, including a first device 20 and a second device 10 in any of the above embodiments.

[0079] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 900, including a processor 940 and a memory 920. The memory 920 stores a program or instructions that can run on the processor 940. When the program or instructions are executed by the processor 940, they implement the various steps of the above projection method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0080] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0081] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0082] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0083] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0084] The electronic device 1000 can display a virtual display area, which includes a preset area, and the electronic device 1000 is communicatively connected to the second device.

[0085] User input unit 1007 is used to receive the first input from the second device in a preset area.

[0086] The processor 1010 is configured to project the display content of the second device onto the virtual display area in response to the first input.

[0087] Optionally, the electronic device 1000 is equipped with a camera, and the processor 1010 is further configured to: capture video frames of the virtual display area when the second device makes a target input, before receiving the first input from the second device in the preset area; identify the actual position of the second device when making the target input in the virtual display area based on the video frames; determine whether the actual position falls within the preset area; and determine that the target input is the first input of the preset area if the actual position falls within the preset area.

[0088] Optionally, the processor 1010 is specifically configured to: determine a first orientation of the actual position relative to the first device, the orientation including direction and three-dimensional coordinates; compare the first orientation with a preset second orientation, the second orientation being the orientation of a preset position on a preset area relative to the first device; and determine that the actual position falls within the preset area if the angle difference between the direction of the first orientation and the direction of the second orientation meets a first angle threshold and the coordinate difference between the three-dimensional coordinates of the first orientation and the three-dimensional coordinates of the second orientation meets a first coordinate threshold.

[0089] Optionally, the processor 1010 is specifically configured to: extract features of the second device, features of the virtual display area, and position change features of the second device relative to the virtual display area from the video frame using a Simultaneous Localization and Mapping (SLAM) algorithm; and identify whether the second device operates the actual position in the virtual display area by inputting the features and position change features of the second device into a neural network recognition model, wherein the neural network recognition model is trained based on sample data including the features and position change features of the second device and the label of whether the second device operates the target position in the virtual display area.

[0090] Optionally, the processor 1010 is specifically configured to: determine a first orientation of the actual position relative to the first device based on the parallax between the lateral coordinates of the actual position imaged on the corresponding views of the two cameras of the first device and the distance from the actual position to the imaging plane.

[0091] Optionally, the processor 1010 is further configured to, before projecting the display content of the second device onto the virtual display area in response to the first input, determine a third position of each device within a preset distance of the first device relative to the first device; compare the direction of each third position with the direction of the second position, and compare the three-dimensional coordinates of each third position with the three-dimensional coordinates of the second position; and identify the second device from among the devices if the angle difference between the direction of the third position of a device and the direction of the second position satisfies a second angle threshold and the coordinate difference between the three-dimensional coordinates of the third position and the three-dimensional coordinates of the second position satisfies a second coordinate threshold.

[0092] Optionally, the processor 1010 is specifically used to: determine the third position of each device relative to the first device within a preset distance using ultra-wideband (UWB) positioning technology.

[0093] Optionally, the processor 1010 is specifically configured to: in response to the first input, send a notification to the second device to trigger the second device to send data corresponding to the display content; receive the data and project it onto the virtual display area.

[0094] In this embodiment, the electronic device can display a virtual display area, which includes a preset area. The first device and the second device are communicatively connected. The first device receives a first input from the second device in the preset area. In response to the first input, the display content of the second device is projected onto the virtual display area. Thus, by using the interactive operation method of the real second device touching the virtual display area of ​​the first device, the display content of the second device can be directly projected onto the second device, which greatly simplifies the projection process, improves the interaction efficiency between devices, and makes the "augmented reality" of the first device more realistic. This allows the user of the first device to experience a more realistic experience in the virtual projection screen, improving the user experience of using the first device.

[0095] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0096] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0097] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0098] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above projection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0099] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0100] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above projection method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0101] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0102] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the projection method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0103] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0105] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A projection method, characterized in that, The projection method is applied to a first device, which can display a virtual display area, the virtual display area including a preset area, and the first device is communicatively connected to a second device. The projection method includes: Receive the first input from the second device in the preset area; In response to the first input, the display content of the second device is projected onto the virtual display area; Before projecting the display content of the second device onto the virtual display area in response to the first input, the method further includes: Determine the third position of each device within a preset distance of the first device relative to the first device; The directions of each third position are compared with the directions of the preset second position, and the three-dimensional coordinates of each third position are compared with the three-dimensional coordinates of the second position, wherein the second position is the orientation of the preset position on the preset area relative to the first device. If the angle difference between the direction of the third position of a device and the direction of the second position meets the second angle threshold, and the coordinate difference between the three-dimensional coordinates of the third position and the three-dimensional coordinates of the second position meets the second coordinate threshold, the second device is identified from the devices.

2. The method according to claim 1, characterized in that, The first device is equipped with a camera, and before receiving the first input from the second device in the preset area, the method further includes: The camera captures video frames of the virtual display area when the second device performs target input; The actual position of the second device when it performs the target input in the virtual display area is identified based on the video frame; Determine whether the actual location falls within the preset area; If the actual location falls within the preset area, the target input is determined to be the first input of the preset area.

3. The method according to claim 2, characterized in that, Determining whether the actual location falls within the preset area includes: Determine the first orientation of the actual position relative to the first device, where orientation includes direction and three-dimensional coordinates; Compare the first location with the second location; If the angle difference between the direction of the first orientation and the direction of the second orientation meets a first angle threshold, and the coordinate difference between the three-dimensional coordinates of the first orientation and the three-dimensional coordinates of the second orientation meets a first coordinate threshold, then the actual position is determined to fall within the preset area.

4. The method according to claim 2, characterized in that, The step of identifying the actual position of the second device when making the target input in the virtual display area based on the video frame includes: The Simultaneous Localization and Mapping (SLAM) algorithm is used to extract features of the second device, features of the virtual display area, and positional change features of the second device relative to the virtual display area from the video frame. By inputting the features of the second device and the position change features into a neural network recognition model, it can be determined whether the second device operates the actual position in the virtual display area. The neural network recognition model is trained based on sample data including the features of the second device and the position change features, and the label of whether the second device operates the target position in the virtual display area.

5. The method according to claim 3, characterized in that, Determining the first orientation of the actual position relative to the first device includes: Based on the parallax between the lateral coordinates of the actual position as imaged on the corresponding views of the two cameras of the first device and the distance from the actual position to the imaging plane, the first orientation of the actual position relative to the first device is determined.

6. The method according to claim 1, characterized in that, Determining the third position of each device within a preset distance of the first device relative to the first device includes: Using ultra-wideband (UWB) positioning technology, the relative positions of each device within a preset distance of the first device to the first device are determined.

7. A projection device, characterized in that, Applied to a first device, the first device can display a virtual display area, the virtual display area including a preset area, the first device is communicatively connected to a second device, and the projection device includes: A receiving module is configured to receive a first input from the second device in the preset area; A projection module is configured to project the display content of the second device onto the virtual display area in response to the first input; The identification module is used to determine, before projecting the display content of the second device onto the virtual display area in response to the first input, the third position of each device within a preset distance of the first device relative to the first device. The directions of each third position are compared with the directions of the preset second position, and the three-dimensional coordinates of each third position are compared with the three-dimensional coordinates of the second position, wherein the second position is the orientation of the preset position on the preset area relative to the first device. If the angle difference between the direction of the third position of a device and the direction of the second position meets the second angle threshold, and the coordinate difference between the three-dimensional coordinates of the third position and the three-dimensional coordinates of the second position meets the second coordinate threshold, the second device is identified from the devices.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the projection method as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the projection method as described in any one of claims 1-6.

Images