An interactive projection input and output device

Abstract

The present application provides an interactive projection input and output device, including: a projection module for projecting screen content from an external device; a control capture module for determining the first coordinate of a user control point corresponding to the user control behavior within a control capture range; an instruction generation module for generating a position information instruction recognizable by an operating system of an external device according to the first coordinate of the user control point and the regional position information of the projection area in the first coordinate system; a wireless transmission module for receiving the screen projection information sent by the external device via a wireless connection, and sending the position information instruction to the external device via a wireless connection to control the external device. The interactive projection input and output device provided in the embodiment of the present application realizes the function of virtual wireless control of external devices, thereby enabling collaborative control of external devices without installing APP or software on the external devices.

Description

Technical Field

[0001] The present application relates to the technical field of interactive projection systems, and in particular to an interactive projection input and output device. Background Art

[0002] The interactive projection system technology combines hybrid virtual reality technology with motion capture technology. It is the result of further development of virtual reality technology. Its core technology is the intelligent recognition and control system of infrared image acquisition and infrared laser positioning. It captures real-time video data through infrared cameras and transmits the data to the image processing unit. Through specific algorithms, dynamic background modeling is performed in the video data stream, the target is separated from the background, the target's location information is determined, and the location information is converted into a computer control signal to achieve control of the computer and other intelligent multimedia systems.

[0003] For example, by using embedded laser projection technology, the touch projection device can project a virtual keyboard. By connecting the virtual projection device to the PC and installing software on the PC, the PC can be operated through the virtual keyboard, thereby replacing the function of the traditional keyboard. However, in the relevant technology, the virtual keyboard or virtual mouse cannot transmit absolute coordinates, that is, the function of the virtual touch screen cannot be realized, and the traditional touch projection device is equipped with a CPU and an operating system, so it only controls the touch projection device itself. When controlling other external smart devices, in order to ensure that the touch projection device can generate the correct control signal, the external smart devices need to install a special APP or software before use. Therefore, the traditional touch projection device cannot achieve universal, cross-system and device collaborative control. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides an interactive projection input and output device.

[0005] The interactive projection input and output device comprises:

[0006] A projection module, used for projecting screen content from an external device so that the screen content is displayed in a projection area corresponding to the external device in the projection screen;

[0007] a manipulation capture module, configured to form a manipulation capture range corresponding to the projection area, capture a user manipulation behavior occurring within the manipulation capture range; and determine, based on a first coordinate system established within the manipulation capture range, a first coordinate of a user manipulation point corresponding to the user manipulation behavior within the manipulation capture range;

[0008] an instruction generating module, configured to obtain a relative position relationship between the user control point and the projection area according to the first coordinates of the user control point and the area position information of the projection area in the first coordinate system, and generate a position information instruction recognizable by the operating system of the external device according to the relative position relationship;

[0009] A wireless transmission module is used to establish an uplink wireless connection and a downlink wireless connection with the external device, receive the projection information sent by the external device through the downlink wireless connection, and the projection information includes the projection content; and send the position information instruction to the external device through the uplink wireless connection, so that the operating system of the external device calls the underlying driver to execute the position information instruction, so as to implement control operations in response to the user's control behavior, thereby controlling the external device.

[0010] Optionally, the projection module includes: a projection function component and a projection controller; wherein,

[0011] The projection function component is used to project the screen content;

[0012] The projection controller is used to receive the projection adjustment instruction issued when the instruction generation module recognizes the gesture operation corresponding to the projection adjustment function, adjust the position, angle and / or size of the projection area, and perform spectral correction that is beneficial to protecting eyesight based on the optical electronic information contained in the projection content.

[0013] Optionally, the manipulation capture module includes a manipulation detector, and a manipulation capture range corresponding to the projection area is formed by setting the manipulation detector, and the manipulation detector captures the user manipulation behavior occurring within the manipulation capture range according to the detected environmental physical signal; and, according to a first coordinate system established within the manipulation capture range, determines the first coordinate of the user manipulation point corresponding to the user manipulation behavior within the manipulation capture range; and identifies and captures the regional position information of the projection area in the first coordinate system; or

[0014] The manipulation capture module includes: a detection signal generator and a manipulation detector; wherein,

[0015] The detection signal generator is used to form a control capture range corresponding to the projection area by sending a detection signal;

[0016] The manipulation detector is used to capture user manipulation behavior occurring within the manipulation capture range by detecting detection signals; and, based on a first coordinate system established within the manipulation capture range, determine the first coordinates of the user manipulation point corresponding to the user manipulation behavior within the manipulation capture range; and, identify and capture the area position information of the projection area in the first coordinate system.

[0017] Optionally, the instruction generation module includes a dedicated integrated circuit chip, which includes a hardware calculation circuit formed by burning, and the hardware calculation circuit is used to obtain the relative position relationship between the user control point and the projection area according to the first coordinates of the user control point and the area position information of the projection area in the first coordinate system, and generate a position information instruction recognizable by the operating system of the external device according to the relative position relationship.

[0018] Optionally, the projection information further includes at least one of the following: screen vertex coordinates, eccentric distance, screen resolution, horizontal and vertical screen orientation, bit rate, frame rate, and operating system type; wherein,

[0019] The horizontal and vertical screen orientation is used to instruct the instruction generation module to adjust the relative position relationship;

[0020] The screen vertex coordinates or the screen resolution are used to instruct the instruction generation module to: generate a second coordinate of a virtual control point to be displayed by the external device according to the relative position relationship and any one of the screen vertex coordinates and the screen resolution, and generate the position information instruction for implementing the control operation at the virtual control point according to the second coordinates, wherein the relative position relationship between the virtual control point and the external device to be displayed is the relative position relationship;

[0021] The code stream and the frame rate are used to coordinately calculate the screen resolution and the horizontal and vertical screen orientation;

[0022] The eccentric distance is used to correct the second coordinate of the virtual control point;

[0023] The operating system type is used to instruct the instruction generation module to generate a position information instruction adapted to the operating system type according to the relative position relationship.

[0024] Optionally, there are multiple external devices, and the projection module is specifically used to: project the screen contents of the multiple external devices to the projection areas corresponding to the respective external devices on the projection screen;

[0025] The instruction generation module is specifically used to: determine the specific projection area where the user control point is located according to the first coordinate of the user control point and the area position information of each projection area, and obtain the relative position relationship between the user control point and the specific projection area, and generate a position information instruction that can be recognized by the operating system of the specific external device corresponding to the specific projection area according to the relative position relationship;

[0026] The wireless transmission module is specifically used to: establish corresponding wireless connections with the multiple external devices respectively, receive screen projection information of the multiple external devices through each wireless connection, and the screen projection information includes the projection content; and receive the location information instruction of the specific external device transmitted by the instruction generation module, and send the location information instruction to the specific external device through the wireless connection corresponding to the specific external device, so that the operating system of the specific external device calls the underlying driver to execute the location information instruction, so as to realize the control operation in response to the user's control behavior, thereby controlling the external device.

[0027] Optionally, the instruction generation module is further used for: when a gesture operation corresponding to a file sharing function is identified, determining the sharing party external device and the location information of the shared file to be displayed on the sharing party external device through the starting point of the gesture operation, determining the shared party external device through the sliding direction or end point of the gesture operation, and generating a file sharing location information instruction, wherein the file sharing location information instruction is used to share the shared file from the sharing party external device to the shared party external device;

[0028] The wireless transmission module is also used to send the file sharing location information instruction to the external device of the sharing party, so that the external device of the sharing party: sends the shared file to the external device of the shared party through the wireless connection established with the shared party; or, sends the shared file to the wireless transmission module through the wireless connection established with the wireless transmission module, so that the wireless transmission module sends the received shared file to the external device of the shared party according to the wireless connection established with the external device of the shared party.

[0029] Optionally, the downlink wireless connection established between the wireless transmission module and the external device includes: a router, Bluetooth, NFC, UWB, zigbee, Lifi, Wifi, NB-IOT, eMTC, LoRa, tuner, GNSS, LNB, cellular 3G, cellular 4G, cellular 5G, cellular 6G or its own wireless hotspot;

[0030] The uplink wireless connection established between the wireless transmission module and the external device includes: Bluetooth, BLE, NFC, router, Wifi, NB-IOT, eMTC, LoRa, tuner, GNSS, LNB, Z-Wave, cellular 3G, cellular 4G, cellular 5G, cellular 6G or wireless USB.

[0031] Optionally, the interactive projection input and output device further includes:

[0032] A camera module, used for collecting image information;

[0033] The audio module is used to collect audio information and play the audio information received by the wireless transmission module from the external device.

[0034] Optionally, the projection module includes: a vision protection filter lens or a vision protection filter lens film, and the vision protection filter includes a vision protection filter lens or a vision protection filter film, which is used to reduce the density of high-energy blue light that is harmful to eyesight and / or reduce the stroboscopic effect.

[0035] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0036] In an embodiment of the present application, the screen image to be displayed of the external device, the projection area formed by the projection module, and the control range formed by the control capture module are interrelated, and the absolute coordinates of the user control point captured are used to generate a position information instruction recognizable by the operating system of the external device through the instruction generation module, and sent to the external device through the wireless transmission module, which is equivalent to forming a virtual wireless touch screen to remotely control the external device; since the instruction generation module generates the position information instruction according to the relative position relationship between the user control point and the projection area, and the external device can respond to the control operation corresponding to the position information instruction with the help of the underlying driver, it is possible to collaboratively control the external device without installing an APP or software on the external device;

[0037] Furthermore, the instruction generation module can also generate position information instructions according to the relative position relationship between the user control point and the projection area and the screen resolution of the external device, thereby being able to collaboratively control external devices of any screen resolution or any screen size;

[0038] Furthermore, the instruction generation module can also generate position information instructions based on the relative position relationship between the user control point and the projection area, the screen resolution of the external device, and the operating system type, thereby being able to collaboratively control external devices with any operating system, any screen resolution, or any screen size. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0040] Figure 1 It is a system architecture diagram shown in the present application according to an exemplary embodiment.

[0041] Figure 2 It is a block diagram of an interactive projection input and output device according to an exemplary embodiment of the present application.

[0042] Figure 3 It is a block diagram of another interactive projection input and output device according to an exemplary embodiment of the present application.

[0043] Figure 4 It is an application scenario diagram shown in this application according to an exemplary embodiment.

[0044] Figure 5 It is a block diagram of another interactive projection input and output device according to an exemplary embodiment of the present application.

[0045] Figure 6 It is another application scenario diagram shown in the present application according to an exemplary embodiment.

[0046] Figure 7 This is another application scenario diagram shown in the present application according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] Next, the embodiments of the present application are described in detail.

[0051] Figure 1 This application is a system architecture diagram according to an exemplary embodiment, such as Figure 1 As shown, the system architecture includes an interactive projection input and output device 10 and an external device 20, wherein a wireless connection is established between the interactive projection input and output device 10 and the external device 20, and the external device 20 can transmit data information including projection information to the interactive projection input and output device 10 through the wireless connection, and the interactive projection input and output device 10 can also transmit data information including location information instructions to the external device 20 through the wireless connection.

[0052] Figure 2 is a block diagram of an interactive projection input and output device 10 according to an exemplary embodiment of the present application, such as Figure 2 As shown, the interactive projection input and output device 10 includes the following modules:

[0053] The projection module 110 is used to project the screen content from the external device 20, so that the screen content is displayed in the projection area corresponding to the external device 20 in the projection picture. The screen content involved in this application refers to the screen content information in the screen information, which is the information corresponding to the pixel matrix in the video frame. The projection module involved in the embodiment of the present application includes a projection function component 1101 and a projection controller 1102, wherein the projection function component 1101 includes a projection lens, an LED light source, a DMD (Digital Micromirror Device), and a DMD driver chip for realizing basic projection functions. The projection picture involved in the embodiment of the present application can be a virtual picture of LED projection, laser projection, liquid projection, quantum (photon) projection, etc., which is composed of solid semiconductor light-emitting materials, optical systems and / or mechanical microsystems, and heat dissipation systems, on a physical plane, a physical surface, or even in the air (without a physical bottom surface). Taking a physical plane as an example, when the projection module projects the LED light source to the physical plane through a lens, it will automatically adjust the focal length to clearly image on the physical plane, and the imaged picture on the physical plane is the projection picture. The projection area involved in the embodiment of the present application refers to part or all of the screen area in the projection screen used to display the projected content. Usually, the projection area is determined by the projection controller 1102 assigning a micro-lens group adapted to the projected content to the projected content, and the projected content is projected onto the projection area. In one embodiment, the projection controller 1102 can automatically determine the position, size and size of the projection area according to the video size in the projection information, for example, allocating a projection area as large as possible and with the same video size on the projection screen for the projected content, or a projection area of ​​any video size, size and position, which is not limited in the embodiment of the present application.

[0054] The control capture module 120 refers to a device module of a mainboard that does not have an operating system and a CPU installed, and is used to form a control capture range corresponding to the projection area, capture user control behavior occurring within the control capture range; and determine, based on a first coordinate system established within the control capture range, a first coordinate of a user control point corresponding to the user control behavior within the control capture range, and send the first coordinate to the instruction generation module 130.

[0055] Figure 3 is a block diagram of another interactive projection input and output device 10 according to an exemplary embodiment of the present application, such as Figure 3As shown, the manipulation capture module 120 involved in the embodiment of the present application may only include a manipulation detector 1201. By setting the manipulation detector 1201, a manipulation capture range corresponding to the projection area is formed. The manipulation detector 1201 captures the user manipulation behavior occurring in the manipulation capture range according to the detected environmental physical signal; and, according to the first coordinate system established in the manipulation capture range, the first coordinate of the user manipulation point corresponding to the user manipulation behavior in the manipulation capture range is determined. For example, when the manipulation detector 1201 is a heat signal detector, a temperature detection range may be formed on the projection screen by the heat signal detector, and the temperature detection range corresponds to the projection area. In the first coordinate system of the temperature detection range, the first coordinate of the user manipulation point corresponding to the user manipulation behavior is identified and determined according to the temperature difference change in the temperature detection range; when the manipulation detector 1201 is an audio signal detector, an ultrasonic detection range may be formed on the projection screen by the audio signal detector, and the ultrasonic detection range corresponds to the projection area. In the first coordinate system of the ultrasonic detection range, the first coordinate of the user manipulation point corresponding to the user manipulation behavior is identified and determined according to the ultrasonic disturbance in the ultrasonic detection range.

[0056] Alternatively, the manipulation capture module involved in the embodiment of the present application may include a detection signal generator 1202 and a manipulation detector 1201, wherein the detection signal generator 1202 is used to form a manipulation capture range corresponding to the projection area by issuing a detection signal; the manipulation detector 1201 is used to capture the user manipulation behavior occurring within the manipulation capture range by detecting the detection signal; and, based on a first coordinate system established within the manipulation capture range, determine the first coordinate of the user manipulation point corresponding to the user manipulation behavior within the manipulation capture range. For example, when the detection signal generator 1202 is an optical signal generator and the manipulation detector 1201 is an optical signal detector, the optical signal A signal generator forms a light film on the projection screen, and the light film corresponds to the projection area. The optical signal detector captures the position of a light spot formed on the light film due to the user's manipulation behavior. According to the light spot position, the first coordinate of a user manipulation point corresponding to the user manipulation behavior is determined in the first coordinate system of the light film; or, a laser dot array can be formed on the projection screen by an optical signal generator, and the laser dot array corresponds to the projection area. The TOF time-of-flight device or structured light detector captures the spatial position of a manipulated object formed in the laser dot array due to the user's manipulation behavior. According to the spatial position, the first coordinate of the user manipulation point corresponding to the user manipulation behavior is determined in the first coordinate system of the laser dot array.

[0057] The control capture range involved in the embodiment itself corresponds to the projection area of ​​the external device 20, and the corresponding relationship may include: the control capture range at least covers the projection area. In this scenario, the user can directly perform control operations on the projection area, so the user can use the control habits of the traditional touch screen on the virtual touch screen formed by the control capture range and the projection area to obtain a control interaction experience similar to that of the traditional touch screen. The above-mentioned correspondence may also include: the control capture range and the projection area are separated from each other. For example, in a stereoscopic projection control scenario, the projection area is inside a three-dimensional space, and the projection control capture range may be irrelevant to the three-dimensional space and independently located in another three-dimensional space or on a two-dimensional plane. In this scenario, the user does not need to directly touch the elements in the projection area, but performs control operations on other spaces or planes corresponding to the projection control capture range, so that the user obtains a control interaction experience adapted to the stereoscopic projection control scenario.

[0058] The first coordinate system involved in the embodiment of the present application refers to the coordinate system of the manipulation capture module, specifically refers to the coordinate system defined by the manipulation detector 1201 when the detection result obtained by the detection is spatially represented. In the first coordinate system, the first coordinate of the user's manipulation point can be obtained by obtaining the position used to characterize the user's manipulation behavior. In one embodiment, the first coordinate system involved in the present application is an absolute coordinate system, and the first coordinate is the absolute coordinate under the absolute coordinate system. In the embodiment of the present application, after capturing a series of physical signals generated by the user's manipulation behavior, the coordinate information of the user's manipulation point can be located by image recognition technology. This is recorded in detail in the relevant technology and will not be repeated here.

[0059] The instruction generation module 130 is used to obtain the relative position relationship between the user control point and the projection area according to the first coordinates of the user control point and the regional position information of the projection area in the first coordinate system, and generate a position information instruction recognizable by the operating system of the external device 20 according to the relative position relationship, and send the position information instruction to the wireless transmission module 140.

[0060] Optionally, the instruction generation module 130 involved in the embodiment of the present application refers to an input and output hardware device module other than the motherboard on which the operating system and the CPU are installed, including a dedicated integrated circuit chip, such as an ASIC (Application Specific Integrated Circuit), a DSP chip, an FPGA (Field Programmable Gate Array), etc., wherein the dedicated integrated circuit chip includes a hardware calculation circuit formed by burning, and the hardware calculation circuit is used to obtain the relative position relationship between the user control point and the projection area according to the first coordinate of the user control point and the regional position information of the projection area in the first coordinate system, and generate a position information instruction recognizable by the operating system of the external device 20 according to the relative position relationship. In the embodiment of the present application, since the function of the instruction generation module 130 is realized by the hardware calculation circuit formed by burning on the dedicated integrated circuit chip, it has a high adaptability to repetitive tasks, and therefore has a higher execution efficiency than a general-purpose computer system with a traditional CPU plus an operating system of similar cost, and has lower energy consumption than a general-purpose computer system when the same calculation efficiency is achieved. Due to less heat dissipation, a fanless design can be adopted, saving the additional cost generated by solving the heat dissipation problem.

[0061] The regional position information of the projection area in the first coordinate system in the embodiment of the present application can be obtained in the following two ways: first, it is obtained by the projection controller 1102. After the projection controller 1102 allocates the corresponding projection area, the initial regional position information of the projection area is determined in its own coordinate system, and then the coordinate system is unified with the control capture module. For example, the device structure of the interactive projection input and output device 10 can be designed to keep the spatial relative relationship between the projection module coordinate system and the control capture module unchanged or not affect the structural type of the coordinate system 1, thereby unifying the coordinate system, or providing one or more additional check points for the projection module to be provided. Unify the respective coordinate systems with the control capture module, or unify the respective coordinate systems to the coordinate system of the real size through the distance measurement principle of the respective detectors and predetermine a common reference origin. The above methods can realize the unification of the respective coordinate systems of the projection module and the control capture module. At this time, the position information obtained in the two modules can be interoperable with each other and can indicate the correct position relationship. The projection controller 1102 transmits the initial area position information of the projection area in its own coordinate system to the instruction generation module 130, so as to form a correct relative position relationship with the first coordinate of the user control point obtained by the instruction generation module 130 from the control capture module;

[0062] Secondly, there is no need to unify the coordinate system. After capturing the first coordinate of the user control point corresponding to the user control behavior within the control capture range, the control detector 1201 in the control capture module further identifies and captures the regional position information of the projection area in the first coordinate system, and sends the first coordinate of the user control point and the regional position information to the instruction generation module 130. Since the detection results are all completed in the control capture module, the correct relative position relationship can naturally be obtained.

[0063] In an embodiment of the present application, the relative position relationship between the user control point and the projection area refers to the relative position of the user control point relative to the inside of the projection area. For example, when the projection area is a rectangle, the relative position can be identified as "lower left vertex: (a, b)", where a and b are real numbers in the range of 0 to 1, which are used to indicate that the lower left vertex of the projection area is the reference origin, the lower left vertex to the lower right vertex is the x-axis direction, and the lower left vertex to the upper left vertex is the y-axis direction. The distance from the user control point to the lower left vertex in the x-axis direction is a times the total length of the projection area in the x-axis direction, and the distance from the user control point to the lower left vertex in the y-axis direction is b times the total length of the projection area in the y-axis direction, that is, (a, b) can be regarded as the normalized coordinate of the first coordinate of the user control point in the projection area. Through this relative position relationship, the second coordinate of a point with the same position relationship can be generated in any other similar type of area, and the second coordinate is determined by the second coordinate system of the other similar type of area. For example, it is only necessary to determine the information of the screen coordinate system of the external device 20 as the second coordinate system, and the second coordinate of the virtual control point to be displayed by the external device 20 can be generated through the relative position relationship, wherein the relative position relationship between the virtual control point and the screen of the external device 20 is the relative position relationship. It should be pointed out that the external device 20 itself may not have a screen, but this does not affect the determination of the virtual control point, which is equivalent to the external device 20 using the virtual screen composed of the information to be displayed to determine the screen coordinate system, thereby further determining the second coordinate of the virtual control point. In this embodiment, the second coordinate system is specifically the screen coordinate system of the external device 20, and the second coordinate is specifically the absolute coordinate under the screen coordinate system.

[0064] In an embodiment of the present application, the instruction generation module 130 can determine the coordinate system information of the external device 20 by using any one of the screen vertex coordinates and the screen resolution obtained from the wireless transmission module 140. For example, when determined by the screen vertex coordinates, the screen vertex with coordinates (0,0) can be determined as the coordinate origin, and the direction from the coordinate origin toward the screen vertex with coordinates (m,0) can be determined as the x-axis, and the direction from the coordinate origin toward the screen vertex with coordinates (0,n) can be determined as the y-axis, where m and n are both positive integers, thereby determining the screen coordinate system of the external device 20; or, when determined by the screen resolution, the area attributes of the screen of the external device 20 are directly obtained according to the screen resolution m×n, and the corresponding screen vertices (0,0), (m,0), (0,n) and (m,n) are generated and the screen coordinate system of the external device 20 is established. According to the determined screen coordinate system, the total length of the screen of the external device 20 in the x-axis direction is multiplied by a in the relative position relationship (a, b) to obtain the horizontal coordinate of the second coordinate of the virtual control point to be displayed by the external device 20, and the total length of the screen of the external device 20 in the y-axis direction is multiplied by b in the relative position relationship (a, b) to obtain the vertical coordinate of the second coordinate of the virtual control point to be displayed by the external device 20, that is, the second coordinate of the virtual control point is (m×a, n×b), and then a position information instruction for implementing a corresponding control operation at the virtual control point is generated according to the second coordinate. In addition, the eccentric distance is used to correct the second coordinate of the virtual control point.

[0065] In the embodiment of the present application, the instruction generation module 130 can generate a location information instruction adapted to the operating system type by obtaining the operating system type from the wireless transmission module 140, thereby ensuring that the location information instruction generated according to the relative position relationship can be adapted to the operating system type of the external device 20, so that the operating system of the external device 20 calls the underlying driver to execute the location information instruction to achieve control operations in response to the user's manipulation behavior. The location information instructions involved in the embodiment of the present application include mouse, keyboard, touchpad and touch screen drive instructions, or coordinate information that can trigger corresponding drive instructions. It should be pointed out that the coordinate information involved in the present application does not necessarily refer to the coordinate information in the traditional coordinate system. Any information that can contain the absolute position or relative position of the target object should be regarded as the coordinate information involved in the present application. The present application does not impose any restrictions on the expression form of the coordinate information contained in the location information instruction. The underlying driver involved in the embodiment of the present application refers to the driver including the mouse, keyboard, touchpad and touch screen. Unlike the application program, the underlying driver is a special program that enables the computer and the device to communicate. It can be said to be equivalent to the hardware interface. The operating system can only control the operation of the hardware device through this interface. If the driver of a device is not installed correctly, it will not work properly. In the embodiment of the present application, the location information instruction is directly sent to the operating system of the external device 20 through a wireless connection, so that the external device 20 directly calls the underlying driver through the operating system to respond to the location information instruction without the need for a wireless connection through the application layer. Since the interactive projection input and output device 10 has no operating system, there is no application program, and the external device 20 does not need to install any APP or software. The control of the external device 20 is directly achieved through the underlying wireless connection. The control operations involved in the embodiments of the present application refer to operations that can be performed by a mouse, keyboard, touchpad and touch screen, including pressing keys, moving the cursor, dragging and sliding. Since the interactive projection input and output device 10 involved in the present application only sends the position information instruction to the external device 20, and the response to the position information instruction and further operation effects are completed on the external device 20 side, it is equivalent to the interactive projection input and output device 10 only transmitting the second coordinate information of the virtual control point carried in the generated position information instruction to the external device 20, which is then converted into a specific touch drive instruction by the operating system of the external device 20. Therefore, the interactive projection input and output device 10 involved in the present application can be regarded as a peripheral device that can replace the mouse, keyboard, touchpad and touch screen.

[0066] Figure 4 This is an application scenario diagram shown according to an exemplary embodiment of the present application, such as Figure 4As shown, taking the collaborative control of the external device 20 as an example, the process of converting the first coordinate into the second coordinate by the instruction generation module 130 in the embodiment of the present application is described in detail. The instruction generation module 130 obtains the first coordinate of the user control point S as (x5, x6), and the coordinate information of the four vertices of the projection area contained in the regional position information, which are N (x1, y1), M (x2, y2), Q (x3, y3) and P (x4, y4). The relative position relationship between the user control point and the projection area is calculated according to the above principle. Taking point N as the reference origin as an example, the relative position relationship can be expressed as "N (x1, y1): ((x5-x1) / (x2-x1), (y5-y1) / (y4-y1))". According to the relative position relationship, combined with any information in the acquired screen vertex coordinates and screen resolution, it can be Determine the coordinate system information of the external device 20, for example, the coordinates of the screen vertices are n(x'1, y'1), m(x'2, y'2), q(x'3, y'3) and p(x'4, y'4). Normally, the values ​​of y'1, y'2 and x'4 are 0. Therefore, the second coordinate system of the screen of the external device 20 can be determined, that is, point n is the reference origin, the direction from point n to point m is the x-axis, and the direction from point n to point p is the y-axis, thereby determining the second coordinate (x'5, y'5) of the virtual control point s as ((x5-x1)×(x'2-x'1) / (x2-x1), (y5-y1)×(y'4-y'1) / (y4-y1)). Finally, the instruction generation module 130 determines in real time based on the second coordinate and the historical second coordinate that the second coordinate is compatible with the control logic under the operating system of the external device 20, and generates a position information instruction under the operating system of the external device 20 based on the control logic and the corresponding instruction format, and the position information instruction directly includes the touch driving instruction that the operating system of the external device 20 can call the corresponding touch underlying driver to execute; in another embodiment, the instruction generation module 130 can directly send the second coordinate information as the position information instruction to the external device 20 after determining the second coordinate (x'5, y'5) of the virtual control point s, so that the operating system of the external device 20 automatically converts the second coordinate information contained in the received position information instruction into a specific touch driving instruction under its own operating system according to the corresponding Bluetooth protocol. Therefore, in this embodiment, the instruction generation module 130 no longer needs to complete the operation of converting the second coordinate into a touch driving instruction under the operating system of the external device 20.

[0067] The wireless transmission module 140 is used to establish an uplink wireless connection and a downlink wireless connection with the external device 20, receive the screen projection information sent by the external device 20 through the downlink wireless connection, and the screen projection information includes the screen projection content; and send the location information instruction to the external device 20 through the uplink wireless connection, so that the operating system of the external device 20 calls the underlying driver to execute the location information instruction, so as to implement the control operation in response to the user's manipulation behavior, thereby controlling the external device 20.

[0068] The wireless connection involved in the embodiments of the present application includes, but is not limited to: router, Bluetooth, NFC (Near Field Communication), UWB (Ultra Wide Band), Zigbee, Lifi (Light Fidelity), NB-IOT (Narrow Band Internet of Things), eMTC (enhanced Machine-Type Communication), LoRa (Long Range Radio), tuner (radio frequency coordination), GNSS (Global Navigation Satellite System), LNB (Low Noise Block), cellular 3G, cellular 4G, cellular 5G, cellular 6G, Wifi (Wireless LAN), Wi-Fi ... Fidelity, wireless network communication technology), 5G, 6G, wireless USB or its own wireless hotspot. For example, the communication protocol types used in the downlink wireless connection established between the wireless transmission module and the external device include: router, Bluetooth, NFC, UWB, zigbee, Lifi, NB-IOT, eMTC, LoRa, tuner, GNSS, LNB, cellular 3G, cellular 4G, cellular 5G, cellular 6G, Wifi, 5G, 6G or its own wireless hotspot; the communication protocol types used in the uplink wireless connection established between the wireless transmission module and the external device include: Bluetooth, BLE ((Bluetooth Low Energy, low-power Bluetooth), NFC, router, Wifi, NB-IOT, eMTC, LoRa, tuner, GNSS, LNB, Z-Wave, cellular 3G, cellular 4G, cellular 5G, cellular 6G, NFC, router, Wifi, 5G, 6G or wireless USB.In addition, in the embodiment of the present application, the wireless connection established between the wireless transmission module 140 and a certain external device 20 may be one or more. When the number of the wireless connections is multiple, the types of the multiple wireless connections may be the same or different. For example, the wireless transmission module 140 receives the projection information sent by the external device 20 through the uplink Wifi connection, and at the same time sends the location information instruction to the external device 20 through the same Wifi connection or another Wifi wireless connection through the wireless connection. At this time, the uplink wireless connection and the downlink wireless connection are the same connection, or the same type of connection; or, the wireless transmission module 140 receives the projection information sent by the external device 20 through the uplink Wifi connection, and sends the location information instruction to the external device 20 through the wireless connection through the Bluetooth connection. At this time, the uplink wireless connection and the downlink wireless connection are different connections and the connection types are also different, thereby realizing the transmission of absolute coordinates through the Bluetooth connection to control the external device 20. Since the first coordinate transmitted is an absolute coordinate rather than a relative coordinate, reverse control of the external touch screen can be achieved.

[0069] In addition to the above-mentioned projection content, the screen projection information involved in the embodiments of the present application may also include: one or more of the following: screen vertex coordinates, screen resolution, horizontal and vertical screen orientation, bitstream, frame rate, operating system type, video size, etc., and the present application does not limit this. Among them, the bitstream and the frame rate are used to cooperate in calculating the screen resolution and the horizontal and vertical screen orientation. For example, when the wireless transmission module 140 needs to transmit the screen resolution and the horizontal and vertical screen orientation to the instruction generation module 130, the bitstream and bitrate can be used to calculate the corresponding screen resolution and horizontal and vertical screen orientation information, and then transmit the information to the generation module 130. Alternatively, the bitstream and bitrate can be directly transmitted to the instruction generation module 130, so that the instruction generation module 130 can calculate the screen resolution and the horizontal and vertical screen orientation information according to the received bitstream and bitrate.

[0070] In the embodiment of the present application, the screen image of the external device 20, the projection area formed by the projection module, and the control range formed by the control capture module are interrelated. The first coordinate of the captured user control point is used to generate a position information instruction recognizable by the operating system of the external device 20 through the instruction generation module 130, and is sent to the external device 20 through the wireless transmission module 140, which is equivalent to forming a virtual wireless touch screen to remotely control the external device 20; since the instruction generation module 130 generates the position information instruction according to the relative position relationship between the user control point and the projection area, and the external device 20 can respond to the control operation corresponding to the brief drive instruction with the help of the underlying driver, it is possible to collaboratively control the external device 20 without installing an APP or software on the external device 20;

[0071] Furthermore, the instruction generation module 130 can also generate a position information instruction according to the relative position relationship between the user control point and the projection area and the screen resolution of the external device 20, thereby being able to collaboratively control the external device 20 of any screen resolution or any screen size;

[0072] Furthermore, the instruction generation module 130 can also generate position information instructions based on the relative position relationship between the user control point and the projection area, the screen resolution of the external device 20, and the operating system type, thereby being able to collaboratively control the external device 20 with any operating system, any screen resolution, or any screen size.

[0073] Optionally, the projection module 110 includes a vision protection filter, which includes a vision protection filter lens or a vision protection filter film. For example, in the optical path formed by the projection lens of the projection module 110, a nano blue light filter is added to reduce the density of high-energy blue light with a wavelength of 410 to 460 nanometers, while ensuring that the proportional relationship of blue light in each group of RGB colors is not affected, so as not to cause color distortion of the projection picture, so that the damage caused by high-energy blue light to the human body can be reduced on the basis of ensuring the clarity and authenticity of the picture; further, a low-pass filter of quartz or synthetic material can be added to the optical path formed by the projection lens to further reduce the impact of the stroboscopic effect on the human eye and improve the user's viewing experience. Since the blue light radiation received by children is about twice that of adults, in this embodiment, by adding a vision protection filter lens or a vision protection filter lens film, the interactive projection input and output device 10 can be better applied to children's education scenes, and the adverse effects on children's vision health when watching the projection picture can be minimized.

[0074] Optionally, the projection controller 1102 may also be used to receive a projection adjustment instruction issued when the instruction generation module 130 identifies a gesture operation corresponding to a projection adjustment function, and adjust the position, angle and / or size of the projection area according to the projection adjustment instruction. In an embodiment of the present application, the instruction generation module 130 is also used to identify the user's gesture operation based on the first coordinates of multiple groups of user control points, and when a gesture operation corresponding to the projection adjustment function is identified, a projection adjustment instruction is issued to the projection module, so that the projection controller 1102 in the projection module reallocates the corresponding projection area according to the projection adjustment instruction, thereby adjusting the position, angle and / or size of the projection area.

[0075] In the embodiment of the present application, the user can adjust the position, angle and / or size of the projection area through gesture operations corresponding to the projection adjustment function, thereby optimizing the user's operating experience.

[0076] Optionally, the projection controller 1102 can also be used to perform spectral correction that is beneficial to protecting eyesight based on the optical electronic information contained in the projected content, such as reducing the proportion of blue light in the spectrum, thereby minimizing the adverse effects on vision health without affecting picture clarity.

[0077] Optionally, the instruction generation module 130 can be used to instruct the instruction generation module 130 to adjust the relative position relationship according to the horizontal and vertical screen orientations. Since the horizontal and vertical screen orientations will change the shape of the projection area, the effect is equivalent to switching the screen resolution of the external device 20, for example, the screen resolution m×n is converted to n×m, while the external screen coordinate system does not change due to the horizontal and vertical screen orientations. At this time, the relative position relationship calculated based on the regional position information of the projection area and the first coordinates of the user control point will not be able to generate the correct second coordinates of the virtual control point in the screen coordinate system. Therefore, it is necessary to adjust the original relative position relationship according to the horizontal and vertical screen orientations so that the adjusted relative position relationship can generate the correct second coordinates of the virtual control point in the screen coordinate system.

[0078] In the embodiment of the present application, since the instruction generation module 130 takes into account the influence of the horizontal and vertical screen orientations on the calculation of the second coordinates of the virtual control point, when the external device 20 changes its horizontal and vertical screen orientations, the virtual wireless touch screen formed by the interactive projection input and output device 10 can still be used to correctly coordinate and control the external device 20.

[0079] Optionally, the wireless transmission module 140 is further used for:

[0080] Sending its own product identity information to the external device 20, so that the external device 20 searches for the corresponding underlying driver;

[0081] Receive a driver download request returned by the external device 20 when the underlying driver is not searched, and send the underlying driver corresponding to the driver download request to the external device 20, so that the external device 20 installs the control underlying driver and executes the location information instruction based on the underlying driver.

[0082] In an embodiment of the present application, when the external device 20 has not installed the underlying driver for executing the position information instruction, the underlying driver corresponding to the driver download request pre-stored in the interactive projection input and output device 10 can be sent to the external device 20, so that the external device 20 installs the underlying driver. In this way, even if the external device 20 is initially unable to recognize the position information instruction, it can still respond to the collaborative control of the interactive projection input and output device 10 by downloading and installing the corresponding underlying driver.

[0083] Figure 5 FIG. 1 is a block diagram of another interactive projection input and output device 10 according to an exemplary embodiment of the present application. Figure 5 As shown, in Figure 2 Based on the illustrated embodiment, the interactive projection input and output device 10 may further include:

[0084] The camera module 150 is used to collect image information, for example, to obtain image information in the environment through a camera;

[0085] The audio module 160 is used to collect audio information and play the audio information received by the wireless transmission module 140 from the external device 20, for example, collecting audio information in the environment through a microphone, and playing the audio information received by the wireless transmission module 140 from the external device 20 through a speaker.

[0086] In the embodiment of the present application, in addition to the camera module 150 and the audio module 160, more peripherals can be added to the interactive projection input and output device 10, such as a gravity sensor, a GPS positioning chip, etc., so that the interactive projection input and output device 10 involved in the present application can have stronger functionality. For example, when the external device 20 is a cloud device, not only can the screen content to be displayed output by the cloud device be displayed through screen projection, but more information can be interacted with the cloud device through the camera module 150 and the audio module 160 to realize richer scene functions such as video recording and recording.

[0087] Optionally, the interactive projection input and output device 10 provided in the present application can also be integrated into any other hardware device or product, for example, it can be integrated into glasses. In this case, the projection module 110 can project the projection image directly onto the retina of the human eye, or onto the lens of the glasses, or into the air. At the same time, the control capture module 120 can capture the user's gestures within the control capture range or track the "touch" operation behavior of the eyeball in the air, thereby generating a corresponding position information instruction in the instruction generation module 130, and transmitting the position information instruction to the external device 20 through the wireless transmission module 140, thereby realizing the control of the external device 20 such as a smart phone or a cloud host.

[0088] Optionally, when there are multiple external devices 20, the projection module is specifically used to project the projection contents of the multiple external devices 20 to the projection areas corresponding to the respective external devices 20 on the projection screen. In the embodiment of the present application, the projection controller 1102 can adaptively adjust each projection area according to the video size information of the multiple projection information, so as to achieve the optimal layout.

[0089] The instruction generation module 130 is specifically used to: determine the specific projection area where the user control point is located according to the first coordinate of the user control point and the regional position information of each projection area, and obtain the relative position relationship between the user control point and the specific projection area, and generate a position information instruction that can be recognized by the operating system of the specific external device 20 corresponding to the specific projection area according to the relative position relationship. In the embodiment of the present application, determining the specific projection area where the user control point is located is specifically achieved in the following manner: obtaining the four vertex coordinates of the corresponding projection area according to the regional position information of each projection area, and when the first coordinate of the user control point falls within the four vertex coordinates of a certain projection area, the projection area is determined as the specific projection area.

[0090] The wireless transmission module 140 is specifically used to: establish corresponding wireless connections with the multiple external devices 20 respectively, receive the projection information of the multiple external devices 20 through each wireless connection, and the projection information includes the projection content; and receive the location information instruction of the specific external device 20 transmitted by the instruction generation module 130, and send the location information instruction to the specific external device 20 through the wireless connection corresponding to the specific external device 20, so that the operating system of the specific external device 20 calls the underlying driver to execute the location information instruction, so as to realize the control operation in response to the user's control behavior, thereby controlling the external device 20.

[0091] In an embodiment of the present application, the wireless transmission module 140 assigns a mark to the external device 20 that successfully establishes a wireless connection, thereby obtaining a correspondence between the external device 20, the wireless connection and the mark, and sending the correspondence to the instruction generation module 130, so that the instruction generation module 130 determines the specific projection area and further determines the specific external device 20 corresponding to the specific projection area. According to the correspondence, when generating a position information instruction, the instruction generation module 130 adds a mark corresponding to the specific external device 20, and sends the marked position information instruction to the wireless transmission module 140. The wireless transmission module 140 removes the mark of the marked position information instruction according to the correspondence, and sends it to the corresponding specific external device 20 through the wireless connection corresponding to the mark.

[0092] like Figure 6 As shown, taking the collaborative control of smartphone A, smartphone B, smart tablet, and laptop computer in the application scenario diagram as an example, the embodiments of the present application are described in detail.

[0093] The wireless transmission module 140 will first establish corresponding wireless connections with the above-mentioned multiple external devices 20, receive the projection information of the multiple external devices 20 through each wireless connection, and send the parsed projection content and video size to the projection module. At the same time, a correspondence between the external device 20, the wireless connection and the mark will be established. For example, the correspondence of smartphone A is "0x0001-192.168.1.2-23", where "0x0001" is the internal device code of smartphone A, "192.168.1.2" is the network address of smartphone A, which is used to find the network connection established with smartphone A, and "23" is the mark corresponding to smartphone A. Similarly, a correspondence between smartphone B, laptop computer and smart tablet will be established, and these correspondences will be sent to the instruction generation module 130.

[0094] The projection controller 1102 can adaptively adjust each projection area according to the video size information of the multiple projection screen information obtained from the wireless transmission module 140, so as to achieve the following: Figure 6 The optimized layout of each projection area shown in the projection screen projects the screen contents of the above-mentioned multiple external devices 20 onto the projection area corresponding to each external device 20 on the projection screen, wherein smartphone A corresponds to projection area A, smartphone B corresponds to projection area B, the laptop corresponds to projection area C, and the smart tablet corresponds to projection area D.

[0095] The instruction generation module 130 obtains the regional vertex coordinates of each projection area based on the regional position information of each projection area. Assuming that the first coordinate of the user control point falls within the vertex coordinates of projection area A, it can be determined that the user is trying to collaboratively control the smart phone A corresponding to projection area A. Therefore, the instruction generation module 130 will obtain the relative position relationship between the user control point and the projection area A based on the first coordinate of the user control point and the vertex coordinates of projection area A. Assuming that the calculated relative relationship is "lower left vertex: (0.25, 0.8)", it means that with the lower left vertex as the reference origin, the length and width ratio of the user control point relative to the projection area in the x-axis direction is 0.25, and the length and width ratio in the y-axis direction is 0.8. The second coordinate of the virtual control point of smartphone A in its screen coordinate system can be generated based on the relative position relationship. Assuming that the screen resolution of smartphone A is 450×800, the second coordinate of the virtual control point can be calculated to be (112.5,640). The location information instruction recognizable by the Android operating system of smartphone A is generated based on the second coordinate, and is marked with the mark "23" corresponding to smartphone A according to the corresponding relationship sent by the wireless transmission module 140 and sent to the wireless transmission module 140.

[0096] Finally, the wireless transmission module 140 will receive the location information instruction marked with "23" from the instruction generation module 130, and find out that the external device 20 corresponding to the mark is smart phone A by searching the corresponding relationship. After removing the mark of the location information instruction, the instruction will be sent to smart phone A through the wireless connection established with smart phone A, so that the operating system of smart phone A calls the underlying driver to execute the location information instruction, so as to realize the control operation in response to the user's control behavior, thereby controlling the external device 20.

[0097] In the embodiment of the present application, when multiple external devices 20 are wirelessly connected to the interactive projection input and output device 10 at the same time, multiple virtual wireless touch screens formed in multiple projection areas on the projection screen can be used to simultaneously control the multiple external devices 20.

[0098] Optionally, based on the above-mentioned embodiment of the specification, the instruction generation module 130 is further used for: when a gesture operation corresponding to a file sharing function is recognized, determining the location information of the sharing party external device 20 and the shared file on the screen of the sharing party external device 20 through the starting point of the gesture operation, determining the shared party external device 20 through the sliding direction or end point of the gesture operation, and generating a file sharing location information instruction, wherein the file sharing location information instruction is used to share the shared file from the sharing party external device 20 to the shared party external device 20;

[0099] The wireless transmission module 140 is also used to send the file sharing location information instruction to the sharing party's external device 20, so that the sharing party's external device 20: sends the shared file to the shared party's external device 20 through a wireless connection established with the shared party; or, sends the shared file to the wireless transmission module 140 through a wireless connection established with the wireless transmission module 140, so that the wireless transmission module 140 sends the received shared file to the shared party's external device 20 according to the wireless connection established with the shared party's external device 20.

[0100] In the embodiment of the present application, the above-mentioned sharing location information instruction is a predefined keyboard and mouse macro, which can be created by the user or pre-stored in the system. The instruction generation module 130 can identify the gesture operation corresponding to the user's control behavior according to the first coordinates of multiple groups of user control points. When the gesture operation corresponding to file sharing is identified, the location information of the sharing party external device 20 and the shared file on the screen of the sharing party external device 20 is determined by the starting point of the gesture operation, the shared party external device 20 is determined by the sliding direction or end point of the gesture operation, and the shared party external device 20 is called. The keyboard and mouse macro suitable for the current file sharing scenario is sent to the wireless transmission module 140, and the wireless transmission module 140 is instructed to send the keyboard and mouse macro to the sharing party external device 20. Since the keyboard and mouse macro is essentially a string of location information instructions, after receiving the keyboard and mouse macro, the sharing party external device 20 will still call the underlying driver in the operating system to execute the keyboard and mouse macro, thereby realizing the control operation of sharing the shared file to the shared party external device 20 according to the operation process of the keyboard and mouse macro in the current scenario, and completing the control of the external device 20.

[0101] For example, when the wireless connection is a Bluetooth connection, the operation process corresponding to the keyboard and mouse macro is: the external device 20 of the sharing party long presses the screen position where the shared file is located, and then after waiting for a predetermined time, selects to send via Bluetooth in the pop-up selection bar, so as to send the shared file to the wireless transmission module 140 of the interactive projection input and output device 10, and the keyboard and mouse macro process ends. In addition, during the process of the external device 20 executing the keyboard and mouse macro, the wireless transmission module 140 will shield the location information instruction sent to the external device 20. When the wireless transmission module 140 receives the shared file sent by the external device 20 of the sharing party, it will continue to send the shared file to the external device 20 of the shared party determined according to the gesture operation, and the file sharing of the specified shared file is completed. In another embodiment, if the external device 20 of the sharing party has established a wireless connection with the shared party, the corresponding keyboard and mouse macro can be called and sent to the external device 20 of the sharing party, so that the external device 20 of the sharing party executes the control operation of sending the shared file to the external device 20 of the shared party through the wireless connection established with the shared party, and the file sharing of the specified shared file is completed.

[0102] like Figure 6As shown, taking the sharing of files from a smart tablet to a laptop in the application scenario diagram as an example, the embodiment of the present application is described in detail. In this application scenario, the projection content of the smart tablet is projected on the projection area D, and the projection content of the laptop is projected on the projection area C. When the instruction generation module 130 recognizes the gesture operation corresponding to the file sharing function, since the starting coordinates of the gesture operation fall in the projection area D, it can be determined that the external device 20 of the sharing party is the smart tablet corresponding to the projection area D. According to the aforementioned method, the second coordinates of the shared file in the screen coordinate system of the smart tablet are obtained according to the relative position relationship of the starting coordinates falling in the projection area D. Since the sliding direction of the gesture operation is toward the projection area C, or the end point falls on the projection area C, it can be determined that the external device 20 of the sharing party is the laptop corresponding to the projection area C.

[0103] The keyboard and mouse macro pre-defined by the user in this scenario is called, and the keyboard and mouse macro is composed of a group of position information instructions recognizable by the operating system of the smart tablet, wherein the second coordinate of the shared file is used as a variable to adjust the keyboard and mouse macro so that the keyboard and mouse macro can select the shared file corresponding to the starting point coordinates on the projection area D on the smart tablet, and then the adjusted keyboard and mouse macro is marked with the corresponding mark of the smart tablet, and finally sent to the wireless transmission module 140.

[0104] The wireless transmission module 140 sends the keyboard and mouse macro to the smart tablet through the wireless connection established with the smart tablet according to the mark on the keyboard and mouse macro, and shields the location information instruction sent to the smart tablet before receiving the shared file sent by the smart tablet. After receiving the keyboard and mouse macro, the operating system of the smart tablet calls the underlying driver to execute the keyboard and mouse macro. For example, after pressing the preset time at the second coordinate of the shared file in its own screen coordinate system, it selects to transmit through the wireless connection established with the interactive projection input and output device 10 in the pop-up selection bar, thereby sending the shared file to the wireless transmission module 140 of the interactive projection input and output device 10. After receiving the shared file sent by the smart tablet, the wireless transmission module 140 ends the shielding of the location information instruction, and continues to execute the subsequent steps of the file sharing function, sending the shared file to the laptop through the wireless connection established with the laptop, thereby completing the file sharing of the specified shared file.

[0105] In the embodiment of the present application, in a scenario where the interactive projection input and output device 10 establishes a wireless connection with multiple external devices 20, the designated shared files can be quickly shared, thereby improving the efficiency of user file sharing and increasing the user's interactivity in the interactive control scenario of multiple external devices 20.

[0106] The following is a detailed description of the functions of the interactive projection input and output device 10 involved in the embodiment of the present application by taking the process of the interactive projection input and output device 10 coordinating and controlling a smartphone A with an iOS operating system, a smart tablet with an Android operating system, a desktop computer with a Windows operating system, and a laptop with a Mac operating system. Figure 7 The above collaborative control scenario can be realized based on the mutual cooperation between a smart phone, a smart tablet, a desktop computer, a laptop computer and an interactive projection input and output device 10. The process includes the following steps:

[0107] The wireless transmission module 140 simultaneously establishes a WiFi hotspot connection with the smart phone, smart tablet, desktop computer and laptop computer, and receives the projection information sent by the corresponding external device 20 through the established WiFi hotspot connection, and transmits the projection information to the projection system; at the same time, a Bluetooth pairing request is sent to the nearby external device 20. After receiving the Bluetooth pairing connection request, the above-mentioned external device 20 respectively establishes a Bluetooth connection with the wireless transmission module 140, and realizes the encrypted transmission of information between the wireless transmission module 140 and the external device 20 according to the pre-made encryption algorithm, and transmits Encryption Information (encryption information), Product Identity Information (product identity information), Version Identity Information (version identity information), Identity Address Information (identity address information) and other information to each other, so that the smart phone, smart tablet, desktop computer and laptop computer can all be based on the Product Identity Information provided by the wireless transmission module 140. IdentityInformation searches whether its own operating system contains an adapted underlying driver. As a result, the smart phone, smart tablet and laptop computer all successfully searched for an adapted underlying driver, but the desktop computer did not detect an adapted driver request. Therefore, the desktop computer will send a driver download request to the wireless transmission module 140. When the wireless transmission module 140 receives the driver download request, it will send the corresponding underlying driver to the desktop computer, so that the desktop computer can install the underlying driver so that it can respond to the location information instruction later.

[0108] The projection controller 1102 in the projection module receives the projection information of the four external devices 20 transmitted by the wireless transmission module 140, parses the projection information of each external device 20, obtains the screen vertex coordinates, screen video size, screen resolution, horizontal and vertical screen orientation, projection content, code stream, frame rate and other information corresponding to the multiple external devices 20, allocates a projection area for the corresponding external device 20 according to the video size and projection content in the above information, and generates the vertex coordinates of the projection area after unifying the coordinate system with the control and capture module. Taking the smart tablet as an example, first, the projection orientation of the projection module and the capture orientation of the control and capture module are relatively fixed, so it can be ensured that the projection module and the control and capture module are in the same coordinate system. Since the screen video size of the smart tablet is 16:9, the DMD (Digital Micromirror) can be controlled Device, digital micromirror device) allocates a projection area 2 with a suitable size and position and a size of 16:9 for it, wherein, in order to obtain the projection area 2, the projection controller 1102 needs to allocate a corresponding micro-lens group for the projection area in the DMD matrix, so that the number of the micro-lens groups controls the size of the projection area 2, and the shape and position of the micro-lens group controls the size and position of the projection area 2. The projection content is projected into the projection screen through the projection lens under the control of the DMD, and the projection screen includes the projection area 2 allocated to the smart tablet. After the projection screen is formed, the projection controller 1102 generates the vertex coordinates of the projection area 2 according to the imaging focal length at this time and the position and number of the micro-lenses used to generate the projection area 2, and sends the vertex coordinates of the projection area 2 to the instruction generation module 130.

[0109] Similarly, projection areas are allocated to other external devices 20 and the corresponding projection contents are projected onto their respective projection areas. Finally, four non-overlapping projection areas 1, 2, 3 and 4 will be formed on the projection screen, and the vertex coordinates representing the area position information of each projection area are sent to the instruction generation module 130. At the same time, the screen vertex coordinates, screen resolution, and horizontal and vertical screen orientations of multiple external devices 20 are also sent to the instruction generation module 130. In another embodiment, since the projection module does not need to unify the coordinate system with the control capture module, there is no need to calculate and send the vertex coordinates of the projection area.

[0110] The control capture module includes a detection signal generator 1202 and a control detector 1201. In the embodiment of the present application, the detection signal generator 1202 is specifically an infrared laser transmitter, and the control detector 1201 is specifically an infrared camera, wherein the infrared laser transmitter can generate a laser virtual control film of about 0.5mm to 5mm close to the plane where the projection image is located; the infrared camera is placed above the control plane (usually on the same side as the projection lens) to capture infrared images. When the infrared image contains a light spot formed by the user manipulating the laser virtual control film, the user's control point is determined based on the position and shape of the light spot on the infrared image and image recognition related technologies. The coordinates of the user control point in the first coordinate system are obtained according to the shooting focal length, and the first coordinates are further obtained in the first coordinate system that is the same as the projection module, and the first coordinates are sent to the instruction generation module 130; in another embodiment of the specification, the control capture module does not need to unify the coordinate system with the projection module, but determines the image coordinate system as the first coordinate system, and after identifying the user control point on the infrared image, further identifies the projection area where the user control point is located, and determines the coordinates of the user control point and the four vertices of the projection area in the image coordinate system, and sends all these coordinates to the instruction generation module 130, where the coordinates of the four vertices are used to characterize the regional position information of the projection area.

[0111] The instruction generation module 130 obtains the relative position relationship between the user control point and the projection area according to the first coordinate of the user control point and the area position information of the projection area in the first coordinate system. Specifically, the relative position relationship is the length and width ratio of the user control point relative to the projection area in the x-axis direction and the y-axis direction. For example, assuming that the four vertex coordinates of the projection area 2 corresponding to the smart tablet are (1600, 1100), (2400, 1100), (1600, 1550), (2400, 1550), and the first coordinate of the user control point is (2000, 1235). First, since the first coordinate falls within the four vertices of the projection area 2, it can be determined that the smart phone 2 corresponding to the projection area 2 is currently being operated. Collaborative control, then it is necessary to find out the relative position relationship of the user control point relative to the projection area 2: taking the lower left vertex (1600, 1100) as the reference origin, calculate the proportion of the user control point in the x-axis direction relative to the projection area, which is the quotient of the distance of the control point in the x-axis direction relative to the reference origin and the total length of the projection area in the x-direction, that is, (2000-1600) / (2400-1600)=0.5, calculate the proportion of the user control point in the y-axis direction relative to the projection area, which is the quotient of the distance of the control point in the y-axis direction relative to the reference origin and the total length of the projection area in the y-direction, that is, (1235-1100) / (1550-1100)=0.3, so the above relative position relationship can be expressed as "lower left vertex: (0.5, 0.3)".

[0112] Furthermore, since the horizontal and vertical screen orientation of the smart tablet is horizontal, the above relative position relationship needs to be adjusted. When the lower left vertex is used as the reference origin, specifically, "lower left vertex: (a, b)" is adjusted to "lower left vertex: (b, 1-a)", so "lower left vertex: (0.5, 0.3)" is adjusted to "lower left vertex: (0.3, 0.5)". Further, according to the screen vertex coordinates or screen resolution of the smart tablet, the second coordinates of the virtual control point to be displayed by the smart tablet are generated. Assume that the screen vertices of the smart tablet are (0, 0) and (1080, 0) respectively. , (0,1920), (1080,1920), where (0,0) is the lower left vertex, the second coordinate of the virtual control point in the smart tablet screen coordinate system can be calculated through the adjusted relative position relationship, where the horizontal coordinate value is the total pixels of the screen x-axis multiplied by the x value in the relative position relationship, that is, (1080-0)×0.3=324, and the vertical coordinate value is the total pixels of the screen y-axis multiplied by the y value in the relative position relationship, that is, (1920-0)×0.5=960, and finally the second coordinate of the virtual control point in the smart tablet screen coordinate system is (324,960).

[0113] Since the operating system of the smart tablet is Android, it is possible to determine in real time that the second coordinate is adapted to the control logic under the Android operating system based on the second coordinate and the historical second coordinate, and generate a location information instruction under the Android operating system based on the control logic and the corresponding instruction format. Finally, the location information instruction is marked with a mark corresponding to the smart tablet and passed to the wireless transmission module 140. The mark is used to instruct the wireless transmission module 140 to send the location information instruction to the smart tablet.

[0114] The wireless transmission module 140 receives the location information instruction transmitted by the instruction generation module 130, and determines to send the location information instruction to the smart tablet based on the mark carried by the location information instruction, then removes the mark of the location information instruction based on the Bluetooth connection established with the smart phone in advance, and sends it to the smart tablet, so that after the smart tablet receives the location information instruction, the operating system calls the corresponding touch underlying driver to execute the location information instruction, so as to realize the control operation in response to the virtual control point, thereby controlling the external device 20.

[0115] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Those of ordinary skill in the art may understand and implement the present invention without creative work.

[0116] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0118] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0119] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An interactive projection input and output device, characterized in that: The device comprises: A projection module, used for projecting screen content from an external device so that the screen content is displayed in a projection area corresponding to the external device in the projection screen; a manipulation capture module, configured to form a manipulation capture range corresponding to the projection area, capture a user manipulation behavior occurring within the manipulation capture range; and determine, based on a first coordinate system established within the manipulation capture range, a first coordinate of a user manipulation point corresponding to the user manipulation behavior within the manipulation capture range; an instruction generation module, configured to obtain a relative position relationship between the user control point and the projection area according to the first coordinates of the user control point and the area position information of the projection area in the first coordinate system, and to generate a second coordinate of a virtual control point to be displayed by the external device according to the relative position relationship, and to generate a position information instruction recognizable by the operating system of the external device according to the second coordinate, wherein the position information instruction is used to implement a control operation at the virtual control point, wherein the relative position relationship between the virtual control point and the external device to be displayed is the relative position relationship between the user control point and the projection area; A wireless transmission module is used to establish an uplink wireless connection and a downlink wireless connection with the external device, receive the screen projection information sent by the external device through the downlink wireless connection, and the screen projection information includes the screen projection content; and send the location information instruction to the external device through the uplink wireless connection, so that the operating system of the external device calls the underlying driver to execute the location information instruction to control the external device.

2. The device according to claim 1, characterized in that The projection module includes: a projection function component and a projection controller; wherein, The projection function component is used to project the screen content; The projection controller is used to receive the projection adjustment instruction issued when the instruction generation module recognizes the gesture operation corresponding to the projection adjustment function, adjust the position, angle and / or size of the projection area, and perform spectral correction that is beneficial to protecting eyesight based on the optical electronic information contained in the projection content.

3. The device according to claim 1, characterized in that The manipulation capture module includes a manipulation detector, which is set to form a manipulation capture range corresponding to the projection area, and the manipulation detector captures the user manipulation behavior occurring within the manipulation capture range according to the detected environmental physical signal; and determines the first coordinate of the user manipulation point corresponding to the user manipulation behavior within the manipulation capture range according to the first coordinate system established within the manipulation capture range; and, identifying and capturing regional position information of the projection area in the first coordinate system; or The manipulation capture module includes: a detection signal generator and a manipulation detector; wherein, The detection signal generator is used to form a control capture range corresponding to the projection area by sending a detection signal; The manipulation detector is used to capture user manipulation behavior occurring within the manipulation capture range by detecting detection signals; and, based on a first coordinate system established within the manipulation capture range, determine the first coordinates of the user manipulation point corresponding to the user manipulation behavior within the manipulation capture range; and, identify and capture the area position information of the projection area in the first coordinate system.

4. The device according to claim 1, characterized in that The instruction generation module includes a dedicated integrated circuit chip, which includes a hardware calculation circuit formed by burning. The hardware calculation circuit is used to obtain the relative position relationship between the user control point and the projection area based on the first coordinates of the user control point and the area position information of the projection area in the first coordinate system, and generate a position information instruction recognizable by the operating system of the external device based on the relative position relationship.

5. The device according to claim 1, characterized in that The projection information also includes at least one of the following: screen vertex coordinates, eccentric distance, screen resolution, horizontal and vertical screen orientation, bit rate, frame rate, and operating system type; wherein, The horizontal and vertical screen orientation is used to instruct the instruction generation module to adjust the relative position relationship; The screen vertex coordinates or the screen resolution are used to instruct the instruction generation module to generate the position information instruction; The code stream and the frame rate are used to coordinately calculate the screen resolution and the horizontal and vertical screen orientation; The eccentric distance is used to correct the second coordinate of the virtual control point; The operating system type is used to instruct the instruction generation module to generate a position information instruction adapted to the operating system type according to the relative position relationship.

6. The device according to claim 1, characterized in that The number of the external devices is multiple, and the projection module is specifically used to: project the screen contents of the multiple external devices to the projection areas corresponding to the respective external devices on the projection screen; The instruction generation module is specifically used to: determine the specific projection area where the user control point is located according to the first coordinate of the user control point and the area position information of each projection area, and obtain the relative position relationship between the user control point and the specific projection area, and generate a position information instruction that can be recognized by the operating system of the specific external device corresponding to the specific projection area according to the relative position relationship; The wireless transmission module is specifically used to: establish corresponding wireless connections with the multiple external devices respectively, and receive screen projection information of the multiple external devices through each wireless connection, wherein the screen projection information includes the screen projection content; And, receiving the location information instruction of the specific external device from the instruction generation module, sending the location information instruction to the specific external device through the wireless connection corresponding to the specific external device, so that the operating system of the specific external device calls the underlying driver to execute the location information instruction, so as to realize the control operation in response to the user's control behavior, thereby controlling the external device.

7. The device according to claim 6, characterized in that The instruction generation module is further used for: when a gesture operation corresponding to a file sharing function is recognized, determining the location information of the sharing party external device and the shared file to be displayed on the sharing party external device through the starting point of the gesture operation, determining the shared party external device through the sliding direction or the end point of the gesture operation, and generating a file sharing location information instruction, wherein the file sharing location information instruction is used to share the shared file from the sharing party external device to the shared party external device; The wireless transmission module is also used to send the file sharing location information instruction to the external device of the sharing party, so that the external device of the sharing party: sends the shared file to the external device of the shared party through the wireless connection established with the shared party; or, sends the shared file to the wireless transmission module through the wireless connection established with the wireless transmission module, so that the wireless transmission module sends the received shared file to the external device of the shared party according to the wireless connection established with the external device of the shared party.

8. The device according to claim 1, characterized in that The downlink wireless connection established between the wireless transmission module and the external device includes: router, Bluetooth, NFC, UWB, zigbee, Lifi, Wifi, NB-IOT, eMTC, LoRa, tuner, GNSS, LNB, cellular 3G, cellular 4G, cellular 5G, cellular 6G or its own wireless hotspot; The uplink wireless connection established between the wireless transmission module and the external device includes: Bluetooth, BLE, NFC, router, Wifi, NB-IOT, eMTC, LoRa, tuner, GNSS, LNB, Z-Wave, cellular 3G, cellular 4G, cellular 5G, cellular 6G or wireless USB.

9. The device according to claim 1, characterized in that The interactive projection input and output device also includes: A camera module, used for collecting image information; The audio module is used to collect audio information and play the audio information received by the wireless transmission module from the external device.

10. The device according to claim 1, characterized in that The projection module comprises: a vision protection filter, wherein the vision protection filter comprises a vision protection filter lens or a vision protection filter film, and is used to reduce the density of high-energy blue light that is harmful to eyesight and / or reduce the stroboscopic effect.

Images