Screen projection reverse control method, screen projection system and screen projection device

Abstract

The embodiments of the present application provide a screen projection reverse control method, a screen projection system, and a screen projection device, and relate to the field of screen projection technology. The method is applied to a target screen projection device directly connected to a terminal device in at least one screen projection device included in the screen projection system, the screen projection system also includes a projection screen, and at least one screen projection device is communicatively connected to the projection screen. The method includes: receiving reverse control information sent by the projection screen; encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation; passing the operation message to an event node corresponding to the type of the control operation, and converting the operation message into a control signal about the terminal device through the event node; sending the control signal to the terminal device, so that the terminal device receives the control signal and performs a reverse control operation of the target type based on the position information. In this way, through at least one screen projection device, multi-level transmission of the reverse control signal is realized, and reverse control of the terminal device by the projection device is realized.

Description

Technical Field

[0001] The present application relates to the field of screen projection technology, and in particular to a screen projection reverse control method, a screen projection system and a screen projection device. Background Art

[0002] When the screen of the terminal device needs to be displayed, in order to achieve a better display effect, the screen of the terminal device can be projected to the projection screen for display. Usually, when you want to use the projection screen to project the screen of the terminal device, you need to first establish a communication connection between the terminal device and the projection screen. In this way, through this communication connection, you can project the screen of the terminal device to the projection screen for real-time display, and you can also achieve reverse control of the terminal device by operating the display screen of the projection screen.

[0003] For example, in a teaching scenario, the teacher can establish an HDMI (High Definition Multimedia Interface) connection or network connection between a PC (Personal Computer) and a projection screen, and then project the PC screen onto the projection screen via HDMI or the network. While explaining the course content, the teacher can control the display of the projection screen, and the projection screen will feed back the control information to the control device, so that the PC can be reversely controlled through the projection screen.

[0004] However, in the above-mentioned screen projection method, the screen projection and reverse control between the terminal device and the projection device are realized by the communication connection between the devices. However, in some scenarios, the terminal device and the projection screen do not communicate directly, but communicate through other devices as intermediate media. In this case, the terminal device cannot be reversely controlled by the above-mentioned reverse control method. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a screen projection reverse control method, a screen projection system and a screen projection device to achieve multi-level transmission of reverse control signals, thereby achieving reverse control of terminal devices. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a screen projection reverse control method, the method being applied to a target screen projection device directly connected to a terminal device in at least one screen projection device included in a screen projection system, the screen projection system further comprising a terminal device and a projection screen, the at least one projection device being communicatively connected to the projection screen, the method comprising:

[0007] Receiving reverse control information sent by the projection screen, wherein the reverse control information is generated when the projection screen detects a control operation in the projection mode, and the reverse control information includes a target type of the control operation and position information corresponding to the control operation;

[0008] According to the target type of the control operation, the reverse control information is encapsulated as an operation message supported by the kernel, wherein the operation message includes the location information corresponding to the control operation; the operation message is delivered to an event node corresponding to the type of the control operation, and the operation message is converted into a control signal about the terminal device through the event node, wherein the event node is a virtual node pre-created by the kernel for responding to the operation message; the control signal is sent to the terminal device, so that the terminal device receives the control signal and performs the reverse control operation of the target type based on the location information.

[0009] In a second aspect, an embodiment of the present application provides a screen projection system, the screen projection system comprising a terminal device, at least one screen projection device and a projection screen, the at least one screen projection device being communicatively connected to the projection screen, wherein:

[0010] The projection screen is used to generate counter-control information when a control operation is detected in the projection mode, and send the counter-control information to a target projection device directly connected to the terminal device among the at least one projection device, wherein the counter-control information includes a target type of the control operation and position information corresponding to the control operation;

[0011] The target screen projection device is used to implement any screen projection reverse control method described in the first aspect;

[0012] The terminal device is used to receive the control signal sent by the target screen projection device and perform a reverse control operation of the target type of the control operation based on the position information corresponding to the control operation in the control signal.

[0013] In a third aspect, an embodiment of the present application further provides a screen projection device, including:

[0014] Memory, used to store computer programs;

[0015] The processor is used to implement any screen projection reverse control method described in the first aspect when executing the program stored in the memory.

[0016] Beneficial effects of the embodiments of the present application:

[0017] In the solution provided in the embodiment of the present application, the reverse control information of the projection screen to the terminal device is transmitted to the terminal device by at least one projection device between the two, that is, the reverse control information of the projection screen to the terminal device is finally realized through the multi-level transmission of the projection screen, at least one projection device and the terminal device. Compared with the projection screen reverse control method that relies on the direct communication connection between the terminal device and the projection screen, the projection screen reverse control method provided in the embodiment of the present application does not require the terminal device to communicate directly with the projection screen, and is suitable for various projection screen reverse control scenarios. In addition, since the projection screen device can simulate itself as a control component through its own event node, convert the reverse control information into a control signal and send the control signal directly to the terminal device through a direct connection with the terminal device, the reverse control effect of transmitting the control signal received by the projection screen to the terminal device is simulated. Of course, the implementation of any product or method of the present application does not necessarily require all of the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0019] FIG1 (a) is a schematic diagram of a reverse control method of an existing screen projection method;

[0020] FIG1( b ) is another schematic diagram of a reverse control method of an existing screen projection method;

[0021] Figure 2 A schematic diagram of a flow chart of the screen projection reverse control method provided in an embodiment of the present application;

[0022] Figure 3 A structural schematic diagram of a screen projection system provided in an embodiment of the present application;

[0023] Figure 4 Based on Figure 2 A specific flow chart of step S202 in the illustrated embodiment;

[0024] FIG5 (a) is a schematic diagram of converting the absolute coordinates in the large screen into normalized coordinates;

[0025] FIG5( b ) is a schematic diagram of converting normalized coordinates into absolute coordinates on a PC screen;

[0026] Figure 6 A schematic diagram of the position conversion effect;

[0027] Figure 7A schematic diagram of the connection relationship between devices provided in the embodiment of the present application;

[0028] Figure 8 Based on Figure 2 A specific flow chart of step S201 in the illustrated embodiment;

[0029] FIG9 (a) is another structural schematic diagram of a screen projection system provided in an embodiment of the present application;

[0030] FIG9 (b) is another structural schematic diagram of the screen projection system provided in an embodiment of the present application;

[0031] FIG9 (c) is a schematic diagram of another structure of the screen projection system provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of the connection relationship between various devices in the screen projection system in a specific embodiment;

[0033] Fig.11 A schematic diagram of the interaction process between devices in the screen projection system in a specific embodiment;

[0034] Fig.12 A schematic diagram of the structure of the screen projection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0036] The following is an explanation of the professional terms involved in the embodiments of this application:

[0037] Screen projection: Display the screen of one device on another device via network / wired method.

[0038] Reverse control: feedback control of the display screen on the control device. The reverse control technology enables the display screen to not only display the screen signal received from the control device, but also to feed back the display status or user interaction information to the control device, thus achieving a more intelligent and interactive display effect. In the embodiment of the present application, the projection screen reversely controls the terminal device.

[0039] In the traditional screen projection method, the projection and reverse control between the terminal device and the projection screen are realized by the communication connection between the devices. That is to say, the terminal device and the projection screen can communicate directly, the terminal device can directly transmit the screen data to be projected to the projection screen, and the projection screen can directly reverse control the terminal device. As shown in Figure 1 (a), the terminal device is a PC and the projection screen is a large screen. The PC and the large screen are directly connected through an HDMI cable and a USBTouch (Universal Serial Bus Touch) cable. The PC sends the screen data to the large screen through the HDMI cable, and the large screen takes the screen data of the HDMI channel and displays the screen corresponding to the screen data. During the screen display process, the user can mark the key content displayed on the large screen by touching the large screen. In order to synchronize the screens of the large screen and the PC, when the large screen detects the user's touch operation, it can generate reverse control information about the PC based on the touch operation, and send the reverse control information to the PC through the USBTouch cable connection between it and the PC, so that the PC receives the reverse control information and marks the key content in the displayed screen based on the reverse control information.

[0040] Alternatively, as shown in FIG1( b), the terminal device is a PC, and the projection screen is a large screen. The PC and the large screen are connected via network communication. The PC is installed with projection software. When the screen displayed by the PC is to be projected to the large screen, the projection software can project the screen data of the PC to the large screen via the network. The large screen can receive and display the screen corresponding to the screen data sent by the projection software through the network connection. During the screen display process, the user can mark the key content displayed on the large screen by touching the key content on the large screen. In order to synchronize the display screens of the large screen and the PC, when the large screen detects the user's touch operation, it can generate counter-control information about the PC based on the touch operation, and send the counter-control information to the projection software in the PC through the network connection between the large screen and the PC, so that the projection software receives the counter-control information and marks the key content in the screen displayed on the PC based on the counter-control information.

[0041] However, when there is no direct communication between the terminal device and the projection screen, it is impossible to use the traditional projection method to achieve projection and reverse control between the terminal device and the projection screen. In order to solve the above problems, realize multi-level transmission of reverse control signals, and then realize reverse control of the terminal device, the present application provides a projection reverse control method, device, projection system, projection device, computer-readable storage medium and computer program product. The following first introduces a projection reverse control method provided in an embodiment of the present application. A projection reverse control method provided in an embodiment of the present application is applied to a target projection device directly connected to the terminal device in at least one projection device included in the projection system. The projection system also includes a terminal device and a projection screen, and at least one projection device is communicatively connected to the projection screen. As Figure 2 As shown, a screen projection reverse control method, the method comprising:

[0042] S201, receiving the reverse control information sent by the projection screen.

[0043] Among them, the reverse control information is generated when the projection screen detects a control operation in the projection mode, and the reverse control information includes the target type of the control operation and the position information corresponding to the control operation.

[0044] S202: Encapsulate the reverse control information into an operation message supported by the kernel according to the target type of the control operation.

[0045] The operation message includes location information corresponding to the control operation.

[0046] S203: delivering the operation message to an event node corresponding to the type of the control operation, and converting the operation message into a control signal for the terminal device through the event node.

[0047] The event node is a virtual node pre-created by the kernel for responding to the operation message.

[0048] S204: Send the control signal to the terminal device, so that the terminal device receives the control signal and performs a reverse control operation of the target type based on the position information.

[0049] It can be seen that in the solution provided by the embodiment of the present application, the reverse control information of the projection screen to the terminal device is transmitted to the terminal device by at least one projection device between the two, that is, the reverse control information of the projection screen to the terminal device is finally realized through the multi-level transmission of the reverse control information between the projection screen, at least one projection device and the terminal device. Compared with the projection screen reverse control method that relies on the direct communication connection between the terminal device and the projection screen, the projection screen reverse control method provided by the embodiment of the present application does not require direct communication between the terminal device and the projection screen, and is applicable to various projection screen reverse control scenarios. In addition, since the projection screen device can simulate itself as a control component through its own event node, convert the reverse control information into a control signal and send the control signal directly to the terminal device through a direct connection with the terminal device, it simulates the reverse control effect of transmitting the control signal received by the projection screen to the terminal device.

[0050] When there is no direct communication between the terminal device and the projection screen, in order to realize the reverse control of the projection screen over the terminal device, at least one projection device can be set between the terminal device and the projection screen, so that the terminal device, the projection screen and at least one projection device constitute a projection system. Among them, the target projection device in at least one projection device included in the projection system is directly connected to the terminal device, and multiple projection devices can be connected to the projection screen in communication.

[0051] The various projection devices in the projection system can be different types of devices. For example, the target projection device can be a projection device with an event node, while other projection devices may not have an event node. For another example, the target projection device can be a projection device with an event node, while the projection device that is communicatively connected to the projection screen can be a gateway device, etc.

[0052] Furthermore, the connection mode of multiple projection devices in the projection system can be a series connection, such as Figure 3 As shown, the screen projection system includes at least one screen projection device 320, namely, screen projection device 1, screen projection device 2 to screen projection device n, and also includes a terminal device 310 and a screen projection screen 330. Among them, multiple screen projection devices 320 are connected in series, a target screen projection device 3201 among the multiple screen projection devices 320 is directly connected to the terminal device 310, and the multiple screen projection devices 320 are connected in series and communicated with the screen projection screen 330. The series connection between multiple screen projection devices can be a physical series connection between each screen projection device, or there can be no physical connection between each screen projection device 320, only a communication series connection, and the device type and series connection method of each screen projection device are not limited here.

[0053] Of course, the connection method of multiple screen projection devices in the screen projection system can also be that the multiple screen projection devices except the target screen projection device directly connected to the terminal device are connected in parallel, and the parallel-connected screen projection devices communicate with the target screen projection device and the projection screen respectively. This is all reasonable, and no restriction is imposed on the connection method of each screen projection device. Various connection methods that can realize the screen projection reverse control method provided by this application are within the protection scope of this application.

[0054] In this way, the projection device that is connected to the projection screen can send image data to the projection screen through its communication connection with the projection screen, and receive the reverse control information about the terminal device fed back by the projection screen through its communication connection with the projection screen. The target projection device that is directly connected to the terminal device can directly receive the image data sent by the terminal device, and directly send the reverse control information fed back by the projection screen to the terminal device.

[0055] In the above step S201, the target projection device can receive the reverse control information sent by the projection screen. The reverse control information is generated when the projection screen detects a control operation in the projection mode, and the reverse control information includes the target type of the control operation and the position information corresponding to the control operation.

[0056] In projection mode, the projection screen can receive the picture data of the current picture displayed by the terminal device transmitted by the projection device, and display the picture corresponding to the picture data.

[0057] When the user wants to operate the screen displayed on the projection screen, the user can send corresponding control operations to the projection screen. For example, when the user wants to mark the key fields in the screen displayed on the projection screen, the user can slide the mouse to the position of the key fields in the projection screen, slide the mouse to select the key fields, and right-click the mouse to select "Mark" to complete the marking operation of the key fields.

[0058] The projection screen can detect the control operation of the user, and in order to synchronize the user's control operation on the projection screen to the terminal device to maintain the consistency of the terminal device and the screen displayed by the projection screen, the projection screen can generate counter-control information based on the detected control operation, and send the counter-control information to the target projection device through the communication connection between it and the projection device. Among them, the counter-control information may include the target type of the control operation detected by the projection screen, and the location information corresponding to the control operation.

[0059] The target types of control operations may include: mouse events and touch screen events. A mouse event is an event in which a user performs a control operation on the projection screen through a mouse connected to the projection screen; a touch screen event is an event in which a user performs a control operation on the projection screen by touching the projection screen. Of course, the above-mentioned mouse events and touch screen events are merely examples of target types of control operations, not limitations. The projection screen can also support other types of control operations, and accordingly, the target types of control operations can also include other events supported by the projection screen.

[0060] The position information corresponding to the control operation may include the starting position and ending position of the control operation; it may also include every position involved in the control operation, and the position information corresponding to the control operation included in the above-mentioned reverse control information is the position information of the projection screen corresponding to the control operation.

[0061] In addition, the reverse control information may also include a timestamp, a command type corresponding to the control operation, the number of touch points, etc. The timestamp included in the reverse control information may be a timestamp when the control operation is detected, a timestamp when the reverse control information is generated, or a timestamp when the control operation is detected and a timestamp when the reverse control information is generated. The command types include press, drag, right key, left key, lift, etc.

[0062] For example, a user uses a mouse to perform a control operation on the projected screen by clicking the right button 3 times. The reverse control information generated by the projected screen may include the mouse event, the position of the mouse in the projected screen when the mouse is clicked, the timestamp of detecting the control operation, the command type of the control operation being pressing the right button, and the number of touch points being 3.

[0063] The target projection device can receive the reverse control information sent by the projection screen.

[0064] As an implementation mode, the screen projection system includes multiple screen projection devices. The projection screen can send reverse control information to the projection device connected to it for communication. The projection device connected to the projection screen for communication can transmit the reverse control information to the target projection device through the connection relationship with other projection devices. The target projection device receives the reverse control information sent by the projection screen and forwarded by other projection devices.

[0065] As an implementation method, the screen projection system includes only one screen projection device, that is, the target screen projection device is directly connected to the terminal device, and the target screen projection device is in communication connection with the projection screen. The projection screen can send reverse control information to the target screen projection device, and the target screen projection device receives the reverse control information directly sent by the projection screen.

[0066] Furthermore, in the above step S202, the target screen projection device can encapsulate the reverse control information into an operation message supported by the kernel according to the target type of the control operation, wherein the operation message includes the position information corresponding to the control operation.

[0067] When the target projection device receives the reverse control information sent by the projection screen, it can parse the reverse control information to obtain the target type of the control operation of the reverse control information. Then, the target projection device can encapsulate the reverse control information into an operation message supported by the kernel of the target projection device according to the target type of the control operation. Among them, the operation message includes the position information corresponding to the control operation, and the position information corresponding to the control operation included in the operation message can be the position information in the projection screen corresponding to the control operation; it can also be the position information in the screen of the terminal device corresponding to the control operation.

[0068] As an implementation method, the target projection device can parse and obtain the position information corresponding to the control operation, and convert the obtained position information of the control operation in the projection screen into the position information in the screen of the terminal device corresponding to the control operation. Afterwards, according to the target type of the control operation, the position information in the screen of the terminal device corresponding to the control operation is encapsulated as an operation message, so that the position information corresponding to the control operation included in the operation message is the position information in the screen of the terminal device corresponding to the control operation.

[0069] As an implementation method, the target projection device can encapsulate the reverse control information into an operation message according to the target type of the control operation, so that the position information corresponding to the control operation included in the operation message is the position information in the projection screen corresponding to the control operation. In addition, in order to facilitate the conversion of the position information in the projection screen corresponding to the control operation into the position information in the screen of the terminal device corresponding to the control operation, the target projection device can encapsulate the pre-stored screen resolution of the projection screen in the operation message.

[0070] Furthermore, in the above step S203, the target projection device can pass the operation message to the event node corresponding to the type of control operation, and convert the operation message into a control signal about the terminal device through the event node. Among them, the event node is a virtual node pre-created by the kernel for responding to the operation message.

[0071] The kernel of the target projection device pre-creates various event nodes for responding to operation messages, such as virtual mouse nodes, virtual touch nodes, etc. Among them, the event node is a virtual node pre-created by the kernel for responding to operation messages, and the virtual node is a virtual processing unit in the electronic device for performing processing operations. The electronic device can create a virtual node of the type indicated by the received event node creation instruction, so that when an operation message is received, the virtual node of the type corresponding to the operation message can be used to perform the processing operation indicated by the operation message.

[0072] For the sake of clarity, the establishment method of event nodes will be described in detail below.

[0073] Among them, the event nodes are all virtual nodes, and the event nodes can convert the operation message into the operation signal corresponding to the event node type. For example, if the event node is a virtual touch screen node, the virtual touch screen node can convert the operation message into a touch screen event control signal; if the event node is a virtual mouse node, the virtual mouse node can convert the operation message into a mouse event control signal.

[0074] Since the target type of the control operation received by the projection screen is a mouse event or a touch screen event, in order to achieve the reverse control effect of transmitting the control operation to the terminal device, so that the user can generate the control operation issued by himself, which affects both the projection screen and the terminal device, the target projection device can convert the operation information into a control signal corresponding to the target type of the control operation through its own event node corresponding to the target type of the control operation.

[0075] Based on this, the target projection device can pass the encapsulated operation message to the event node corresponding to the target type of the control operation, so as to convert the operation message into a control signal about the terminal device corresponding to the target type of the control operation through the event node. Among them, the control signal may include the location information corresponding to the control operation.

[0076] For example, if the target type of the control operation is a mouse event, the target screen projection device can pass the encapsulated operation message to the virtual mouse node, so that the virtual mouse node converts the operation message into a mouse event control signal about the terminal device.

[0077] Furthermore, in the above step S204, the target screen projection device can send a control signal to the terminal device, so that the terminal device receives the control signal and performs a reverse control operation of the target type based on the position information.

[0078] After the reverse control information is converted into a control signal about the terminal device through the event node, the target projection device can send the control signal about the terminal device to the terminal device. Then the terminal device can receive the control signal and perform the reverse control operation according to the position information corresponding to the control operation in the control signal and the event type corresponding to the control signal.

[0079] Among them, the position information corresponding to the control operation in the control signal can be the position information of the projection screen corresponding to the control operation, or it can be the position information on the screen of the terminal device corresponding to the control operation.

[0080] As an implementation mode, the position information corresponding to the control operation in the control signal is the position information on the screen of the terminal device corresponding to the control operation. When the terminal device receives the control signal, it can parse the control signal, obtain the position information on the screen of the terminal device corresponding to the control operation, and perform the reverse control operation indicated by the control signal at the position information.

[0081] For example, when the control information is a mouse event control signal and the mouse event is clicking the right button of the mouse at position A on the screen of the terminal device, the terminal device can move the mouse marker to position A on the screen of the terminal device after receiving the mouse event control signal and display the screen after clicking the right button of the mouse.

[0082] As an implementation method, when the position information corresponding to the control operation in the control signal is the position information of the projection screen corresponding to the control operation, the control signal also includes the screen resolution of the projection screen. When the terminal device receives the control signal, it can parse the control signal to obtain the position information and screen resolution of the projection screen corresponding to the control operation of the projection screen, and convert the position information of the projection screen corresponding to the control operation into the position information in the screen of the terminal device corresponding to the control operation according to the screen resolution of the projection screen and its own screen resolution. Afterwards, according to the obtained position information in the screen, the reverse control operation indicated by the control signal is executed.

[0083] In this way, when the projection screen detects a control operation, it can generate a counter-control instruction about the terminal device, and forward the counter-control instruction to the terminal device through at least one projection device in the projection system, thereby realizing multi-level transmission of counter-control information between the projection screen, at least one projection device and the terminal device, and further realizing counter-control of the terminal device.

[0084] In this embodiment, the reverse control information of the projection screen to the terminal device is transmitted to the terminal device by at least one projection device between the two, that is, the reverse control information of the projection screen to the terminal device is finally realized through the multi-level transmission of the reverse control information between the projection screen, at least one projection device and the terminal device. Compared with the projection screen reverse control method that relies on the direct communication connection between the terminal device and the projection screen, the projection screen reverse control method provided in the embodiment of the present application does not require direct communication between the terminal device and the projection screen, and is applicable to various projection screen reverse control scenarios. In addition, since the projection screen device can simulate itself as a control component through its own event node, convert the reverse control information into a control signal and send the control signal directly to the terminal device through a direct connection with the terminal device, it simulates the reverse control effect of transmitting the control signal received by the projection screen to the terminal device.

[0085] As an implementation method of the present application, Figure 4 As shown, the above step S202, i.e., the step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation, may include:

[0086] S401: Parse the reverse control information to obtain the target type of the control operation and the position information corresponding to the control operation.

[0087] S402: According to the position information conversion method corresponding to the target type, the position information of the projection screen corresponding to the control operation is converted into the position information on the screen of the terminal device corresponding to the control operation.

[0088] S403: Encapsulate the location information on the screen of the terminal device and the target type into an operation message supported by the kernel according to the format of the operation message supported by the kernel.

[0089] In order to facilitate the reverse control of the terminal device using operation messages, the target projection device can convert the position information of the projection screen corresponding to the control operation into the position information of the terminal device corresponding to the control operation when encapsulating the reverse control message into an operation message.

[0090] Based on this, after the target projection device obtains the reverse control information sent by the projection screen, it can first parse the reverse control information to obtain the target type of the control operation and the location information corresponding to the control operation.

[0091] Of course, the reverse control information also includes information such as timestamp, command type corresponding to the control operation, number of touch points, etc. Therefore, the target projection device can parse the reverse control information and obtain information such as timestamp, command type corresponding to the control operation, number of touch points, etc.

[0092] Next, the target screen projection device can convert the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type of the control operation. After obtaining the position information on the screen of the terminal device corresponding to the control operation, the position information on the screen of the terminal device and the target type of the control operation are encapsulated as an operation message supported by the kernel according to the format of the operation message supported by the kernel.

[0093] For example, if the target type of the control operation is a mouse event, and the position information corresponding to the control operation is the absolute coordinates of the mouse on the projection screen. The target projection device can convert the absolute coordinates of the mouse on the projection screen into the absolute coordinates of the mouse on the screen of the terminal device according to the position information conversion method of the mouse event, and encapsulate the absolute coordinates of the mouse on the screen of the terminal device and the mouse event into an operation message supported by the kernel according to the format of the operation message supported by the kernel.

[0094] Of course, the reverse control information can also include timestamp, command type corresponding to the control operation, number of touch points, etc. Thus, the target projection device can encapsulate the position information on the screen of the terminal device, the target type of the control operation, timestamp, command type corresponding to the control operation, number of touch points, etc. as the operation message of the kernel message.

[0095] In this embodiment, the target projection device can parse the reverse control information to obtain the target type of the control operation and the position information corresponding to the control operation. Then, according to the position information conversion method corresponding to the target type, the position information of the projection screen corresponding to the control operation is converted into the position information in the screen of the terminal device corresponding to the control operation. And according to the format of the operation message supported by the kernel, the position information in the screen of the terminal device and the target type are encapsulated as the operation message supported by the kernel. In this way, the target projection device can convert the screen position information corresponding to the control operation into the position information in the screen of the terminal device corresponding to the control operation, so as to avoid the position information finally fed back to the terminal device still being the position information in the screen, resulting in the implementation position of the control operation in the screen of the terminal device being inconsistent with the implementation position of the control operation in the projection screen, thereby making the reverse control effect of the terminal device poor and affecting the user experience. In addition, the target projection device encapsulates the reverse control information into the operation message supported by the kernel, so as to facilitate the event node in the kernel to read the operation message and convert the operation message, thereby improving the success rate of message conversion.

[0096] As an implementation method of the embodiment of the present application, the target type of the control operation may include a mouse event and a touch event, and accordingly, the event node may include a virtual mouse node and a virtual touch node.

[0097] The above step S203, i.e., the step of transmitting the operation message to the event node corresponding to the type of the control operation, and converting the operation message into a control signal about the terminal device through the event node, may include:

[0098] If the target type of the control operation is a mouse event, the operation message is transmitted to a virtual mouse node, and the operation message is converted into a mouse event control signal about the terminal device through the virtual mouse node; or,

[0099] If the target type of the control operation is a touch event, the operation message is transmitted to a virtual touch node, and the operation message is converted into a touch event control signal for the terminal device through the virtual touch node.

[0100] The kernel of the target projection device pre-creates a virtual event node of the target type of the control operation.

[0101] Specifically, the target projection device stores preset configuration data of virtual nodes. When the user wishes to create an event node, the user can send an event node creation instruction carrying the node type and node configuration data of the node to be created to the target projection device. For example, the user writes an event node creation instruction carrying the node type and node configuration data in the target projection device. When the kernel of the target projection device receives the event node creation instruction, it creates an event node according to the node configuration data and the preset configuration data of the virtual node, and configures the node's attribute information to obtain the event node. Among them, the attribute information includes the node name, the node's adaptive parameters, the node storage location, and the node's link, etc.

[0102] The target types of the control operation may include mouse events and touch events. Accordingly, the event nodes in the target screen projection device may include virtual mouse nodes and virtual touch nodes.

[0103] Furthermore, if the target type of the control operation is a mouse event, the target screen projection device can pass the operation message to the virtual mouse node, and convert the operation message into a mouse event control signal about the terminal device through the virtual mouse node.

[0104] If the target type of the control operation is a touch event, the target screen projection device can pass the operation message to the virtual touch node, and convert the operation message into a touch event control signal about the terminal device through the virtual touch node.

[0105] In this embodiment, the target projection device can pass the operation message to the event node corresponding to the target type of the control operation, and convert the operation message into a control signal about the terminal device through the event node. In this way, after the operation message is converted into a control signal, the control signal is passed to the terminal device, and the terminal device can directly respond to the control signal. Therefore, the target type of the control operation received by the terminal device and the projection screen is the same, achieving the reverse control effect of transmitting the control operation performed by the user on the projection screen to the terminal device in multiple levels.

[0106] As an implementation method of an embodiment of the present application, the target type of the control operation is a mouse event, and the position information corresponding to the control operation is the absolute coordinates of the mouse on the projection screen.

[0107] The above-mentioned step S402, i.e., the step of converting the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type, includes:

[0108] Calculate the ratio of the first coordinate in the first direction of the absolute coordinate to the resolution value in the first direction of the projection screen, and calculate the ratio of the second coordinate in the second direction of the absolute coordinate to the resolution value in the second direction of the projection screen to obtain normalized coordinates;

[0109] Calculate the product of the third coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the terminal device, and calculate the product of the fourth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the terminal device to obtain the absolute coordinates of the mouse in the screen of the terminal device.

[0110] When the target type of the control operation is a mouse event, the position information corresponding to the control operation is the absolute coordinates of the mouse corresponding to the control operation on the projection screen.

[0111] In order to facilitate the terminal device to directly respond to the mouse event control signal after receiving the mouse event control signal, the target projection device can convert the absolute coordinates of the projection screen corresponding to the control operation into the absolute coordinates of the mouse in the terminal device screen corresponding to the control operation, and encapsulate the absolute coordinates of the mouse in the terminal device screen corresponding to the control operation and the mouse event into an operation message supported by the kernel, and then pass the operation message to the virtual mouse node, and convert the operation message into a mouse event control signal.

[0112] The target projection device may pre-store the resolution of the projection screen and the resolution of the screen of the terminal device. When performing coordinate conversion, the target projection device may first calculate the ratio of the first coordinate in the first direction in the absolute coordinate to the resolution value in the first direction of the pre-stored resolution of the projection screen, and calculate the ratio of the second coordinate in the second direction in the absolute coordinate to the resolution value in the second direction of the projection screen, to obtain the normalized coordinates of the mouse corresponding to the control operation in the projection screen.

[0113] Afterwards, the product of the third coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the terminal device is calculated, and the product of the fourth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the terminal device is calculated to obtain the absolute coordinates of the mouse corresponding to the control operation in the screen of the terminal device.

[0114] For example, as shown in Figure 5 (a), in the teaching scenario, the projection screen is a large screen, and the target projection device has a product name, for example, it can be an intelligent control engine, which integrates recording, video distribution, face recognition, IoT control and other functions, and is the core device in the smart teaching system. The terminal device is a PC. The resolution of the large screen is W1×H1, that is, the absolute coordinates of the mouse on the large screen range from (0, 0) to (W1, H1), and the resolution of the PC screen is W2×H2, that is, the absolute coordinates of the mouse on the large screen range from (0, 0) to (W2, H2). The user controls the mouse to perform control operations at the absolute coordinates (x, y) of the large screen. In this way, after parsing the absolute coordinates of the mouse on the large screen, the intelligent control engine can calculate the ratio of the absolute coordinates to the resolution in each direction, and obtain the normalized coordinates (x / W1, y / H1) corresponding to the control operation, and mark the normalized coordinates as (x1, y1), where (x1, y1) belongs to the relative coordinate range of (0, 0) to (1, 1).

[0115] Furthermore, as shown in Figure 5 (b), the intelligent control engine can calculate the product of the normalized coordinates (x1, y1) and the resolution of the PC screen in each direction to obtain the absolute coordinates (x1×W2, y1×H2) on the PC screen corresponding to the control operation, thereby converting the normalized coordinates within the relative coordinate range of (0, 0) to (1, 1) into absolute coordinates on the PC screen ranging from (0, 0) to (W2, H2).

[0116] like Figure 6 As shown in the figure, the target type of the control operation is a mouse event, and the position information corresponding to the control operation is the absolute coordinates of the mouse on the large screen. The intelligent control engine converts the position information corresponding to the control operation as the absolute coordinates of the mouse on the large screen into the position information corresponding to the control operation as the absolute coordinates of the mouse on the PC screen through coordinate conversion. In this way, when the screen contents displayed on the large screen and the PC are both "important test points", through reverse control, the absolute coordinates of the mouse on the large screen correspond to the absolute coordinates of the mouse on the PC screen, and the user can intuitively feel the precise reverse control of the large screen over the PC.

[0117] In this embodiment, when the target type of the control operation is a mouse event, and the position information corresponding to the control operation is the absolute coordinates of the mouse on the projection screen, the target projection device can convert the absolute coordinates of the mouse on the projection screen corresponding to the control operation into the normalized coordinates of the mouse on the projection screen corresponding to the control operation according to the resolution of the projection screen, and then, according to the resolution of the screen of the terminal device, convert the normalized coordinates into the absolute coordinates of the mouse on the screen of the terminal device corresponding to the control operation. In this way, by converting the absolute coordinates of the mouse on the projection screen corresponding to the control operation into the absolute coordinates of the mouse on the screen of the terminal device corresponding to the control operation according to the corresponding method of the mouse event, and encapsulating the absolute coordinates of the mouse on the screen of the terminal device corresponding to the control operation as an operation message, and then converting it into a mouse event control signal and transmitting it to the terminal device, the terminal device can directly respond to the mouse event control signal according to the absolute position, so that the screen position of the terminal device responding to the mouse event control signal corresponds to the screen position of the user performing the control operation in the projection screen, thereby improving the accuracy of the reverse control of the terminal device and improving the user experience.

[0118] As an implementation method of an embodiment of the present application, the target type of the control operation is a mouse event, and the position information corresponding to the control operation is the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen.

[0119] The above-mentioned step S402, i.e., the step of converting the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type, includes:

[0120] Acquire the previous absolute coordinates of the mouse on the projection screen, and calculate the sum of the previous absolute coordinates and the relative coordinates to obtain the current absolute coordinates of the mouse on the projection screen;

[0121] Calculate the ratio of the first coordinate in the first direction of the current absolute coordinate to the resolution value in the first direction of the projection screen, and calculate the ratio of the second coordinate in the second direction of the current absolute coordinate to the resolution value in the second direction of the projection screen to obtain normalized coordinates;

[0122] Calculate the product of the third coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the terminal device, and calculate the product of the fourth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the terminal device to obtain the absolute coordinates of the mouse in the screen of the terminal device.

[0123] When the target type of the control operation is a mouse event, the position information corresponding to the control operation may be the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen.

[0124] Among them, the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen, that is, the change in the mouse position coordinates relative to the previous position of the mouse on the projection screen during the control operation.

[0125] The target projection device can convert the relative coordinates of the mouse corresponding to the control operation on the projection screen into the absolute coordinates of the mouse corresponding to the control operation on the screen of the terminal device, and encapsulate the absolute coordinates of the mouse corresponding to the control operation on the screen of the terminal device and the mouse event into an operation message supported by the kernel, and then pass the operation message to the virtual mouse node, and convert the operation message into a mouse event control signal.

[0126] The target projection device may pre-store the resolution of the projection screen and the resolution of the terminal device's screen. When performing coordinate conversion, the target projection device may first obtain the previous absolute coordinates of the mouse on the projection screen, and calculate the sum of the previous absolute coordinates and the relative coordinates to obtain the current absolute coordinates of the mouse on the projection screen.

[0127] As an implementation method, the target projection device may pre-store the absolute coordinates of the previous position of the mouse on the projection screen, that is, the previous absolute coordinates of the mouse on the projection screen.

[0128] As an implementation method, when the target projection device receives the back-control information and parses the back-control information to obtain the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen, the target projection device can send a previous absolute coordinate acquisition instruction to the projection screen to acquire the absolute coordinates of the previous position of the mouse on the projection screen, and receive the previous absolute coordinates of the mouse on the projection screen returned by the projection screen based on the previous absolute coordinate acquisition instruction.

[0129] Afterwards, the target projection device can calculate the ratio of the first coordinate in the first direction in the current absolute coordinate to the resolution value in the first direction of the pre-stored resolution of the projection screen, and calculate the ratio of the second coordinate in the second direction in the current absolute coordinate to the resolution value in the second direction of the projection screen to obtain the normalized coordinates of the mouse corresponding to the control operation in the projection screen.

[0130] Next, the product of the third coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the screen of the terminal device is calculated, and the product of the fourth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the screen of the terminal device is calculated to obtain the absolute coordinates of the mouse corresponding to the control operation in the screen of the terminal device.

[0131] In this embodiment, when the target type of the control operation is a mouse event, and the position information corresponding to the control operation is the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen, the target projection device can determine the current absolute position of the mouse in the projection screen according to the previous absolute position of the mouse in the projection screen, and convert the current absolute coordinates of the mouse on the projection screen corresponding to the control operation into the normalized coordinates of the mouse on the projection screen corresponding to the control operation according to the resolution of the projection screen, and then, according to the resolution of the screen of the terminal device, convert the normalized coordinates into the absolute coordinates of the mouse on the screen of the terminal device corresponding to the control operation. In this way, by converting the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen into the absolute coordinates of the mouse on the screen of the terminal device corresponding to the control operation in a corresponding manner to the mouse events, and encapsulating the absolute coordinates of the mouse on the screen of the terminal device corresponding to the control operation as an operation message, and then converting it into a mouse event control signal and transmitting it to the terminal device. The terminal device can directly respond to the mouse event control signal according to the absolute position. In this way, the screen position of the terminal device in response to the mouse event control signal corresponds to the screen position where the user performs the control operation on the projection screen, thereby improving the screen reverse control effect.

[0132] As an implementation method of an embodiment of the present application, the target type of the control operation is a touch event, and the position information corresponding to the control operation is the normalized coordinates of the touch operation on the projection screen.

[0133] The above-mentioned step S402, i.e., the step of converting the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type, includes:

[0134] Calculate the product of the fifth coordinate in the first direction of the normalized coordinates and the resolution value in the first direction of the resolution of the terminal device, and calculate the product of the sixth coordinate in the second direction of the normalized coordinates and the resolution value in the second direction of the resolution of the terminal device to obtain the absolute coordinates of the touch operation on the screen of the terminal device.

[0135] When the target type of the control operation is a touch event, the position information corresponding to the control operation is the normalized coordinates of the touch operation on the projection screen. The touch operation can be performed by the user using a finger or a touch device such as a stylus.

[0136] The target projection device can convert the normalized coordinates of the touch operation corresponding to the control operation on the projection screen into the absolute coordinates of the touch operation corresponding to the control operation on the screen of the terminal device, and encapsulate the absolute coordinates of the touch operation corresponding to the control operation on the screen of the terminal device and the mouse event into an operation message supported by the kernel, and then pass the operation message to the virtual mouse node, and convert the operation message into a mouse event control signal.

[0137] The target projection device may pre-store the resolution of the terminal device's screen. When performing coordinate conversion, the target projection device may calculate the product of the fifth coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the terminal device's screen, and calculate the product of the sixth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the terminal device's screen, to obtain the absolute coordinates of the touch operation on the terminal device's screen.

[0138] In this embodiment, when the target type of the control operation is a touch event, and the position information corresponding to the control operation is the normalized coordinates of the touch operation on the projection screen, the target projection device can convert the normalized coordinates into the absolute coordinates of the touch operation corresponding to the control operation on the screen of the terminal device according to the resolution of the screen of the terminal device. In this way, by converting the normalized coordinates of the touch operation on the projection screen into the absolute coordinates of the touch operation corresponding to the control operation on the screen of the terminal device in accordance with the touch event corresponding method, and encapsulating the absolute coordinates of the touch operation corresponding to the control operation on the screen of the terminal device as an operation message, and then converting it into a touch event control signal and transmitting it to the terminal device, the terminal device can directly respond to the touch event control signal according to the absolute position. In this way, the screen position of the terminal device responding to the touch event control signal corresponds to the screen position of the user performing the control operation on the projection screen, thereby improving the screen reverse control effect.

[0139] As an implementation method of an embodiment of the present application, the target projection device and the projection screen communicate through a network, the target projection device stores pre-configured IP information and port information of the projection screen, and the reverse control information also includes the IP information and port information of the projection screen for transmitting the reverse control information.

[0140] Before the above step S202, i.e. the step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation, the following may also be included:

[0141] Verify whether the IP information included in the counter-control information is consistent with the IP information of the projection screen stored in itself, and whether the port information included in the counter-control information is consistent with the port information of the projection screen stored in itself;

[0142] If the IP information included in the reverse control information is consistent with the IP information of the projection screen stored in itself, and the port information included in the reverse control information is consistent with the port information of the projection screen stored in itself, execute the step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation.

[0143] In the screen projection system, the target screen projection device and the projection screen can communicate through the network. The IP (Internet Protocol) information and port information for network communication are pre-configured for the projection screen, and the IP information and port information of the projection screen are stored in the target screen projection device. The target screen projection device and the projection screen can communicate through a wireless network or a wired network.

[0144] The reverse control information fed back by the projection screen may also include the IP information and port information of the projection screen for transmitting the reverse control information.

[0145] When the target projection device receives the reverse control information fed back by the projection screen, it can parse the reverse control information to obtain the IP information and port information of the projection screen carried in the reverse control information. Afterwards, for the sake of communication security, the target projection device can verify whether the IP information included in the reverse control information is consistent with the IP information of the projection screen stored in itself, and verify whether the port information included in the reverse control information is consistent with the port information of the projection screen stored in itself.

[0146] Furthermore, if the IP information included in the reverse control information is consistent with the IP information of the projection screen stored in itself, and the port information included in the reverse control information is consistent with the port information of the projection screen stored in itself, the target projection device can execute the step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation.

[0147] On the contrary, if the IP information included in the counter-control information is inconsistent with the IP information of the projection screen stored by itself, and / or the port information included in the counter-control information is inconsistent with the port information of the projection screen stored by itself, then the target projection device can block the counter-control information and report the device verification instruction including the counter-control information, so that the staff can verify the device sending the counter-control information based on the device verification instruction to eliminate communication security risks.

[0148] In this embodiment, when the target screen projection device receives the reverse control information, it can compare the IP information and port information carried by the reverse control information with the IP information and port information of the projection screen stored by itself to see if they are consistent. When the IP information included in the reverse control information is consistent with the IP information of the projection screen stored by itself, and the port information included in the reverse control information is consistent with the port information of the projection screen stored by itself, the step of encapsulating the reverse control information into an operation message supported by the kernel is performed according to the target type of the control operation. In this way, by verifying the reverse control information, the security of the information source of the reverse control information is verified, and untrusted devices are prevented from reverse controlling the terminal device and tampering with the screen data of the terminal device, thereby improving the information security of the target screen projection device and the terminal device.

[0149] As an implementation method of an embodiment of the present application, the terminal device and the target screen projection device are connected via an HDMI cable and a Touch cable respectively, and the HDMI cable is used to transmit the screen projection data.

[0150] The step of sending the control signal to the terminal device in the above step S204 may include:

[0151] The control signal is sent to the terminal device through the Touch line.

[0152] The terminal device and the target screen projection device can be connected via an HDMI cable and a Touch (connection) cable respectively, wherein the HDMI cable is used to transmit screen projection data, and the Touch cable is used to transmit control signals. In addition, the above-mentioned Touch cable can be a USB Touch or a Type-C (Type-c interface) Touch. Of course, it can also be other Touch cables, which are all reasonable and are not limited here.

[0153] After the target projection device converts the feedback control information from the projection screen into a control signal about the terminal device, it can send the control signal to the terminal device through the Touch line between it and the terminal device, so that the terminal device receives and responds to the control signal.

[0154] like Figure 7 As shown, notebook A is the terminal device, intelligent control engine B is the target screen projection device, and large screen C is the projection screen. Notebook A and intelligent control engine B are connected via HDMI cable and Touch cable respectively. Notebook A transmits the screen data to intelligent control engine B via HDMI cable. Intelligent control engine B receives the reverse control information fed back by large screen C, encapsulates the reverse control information into operation messages through message processing, and converts the operation messages into mouse event control signals or touch event control signals through internal mouse nodes or touch nodes, and transmits them to notebook A via Touch cable.

[0155] In this embodiment, the target projection device transmits the control signal to the terminal device through the Touch line. In this way, for the terminal signal, the control signal transmitted by the target projection device and received through the Touch line simulates the control signal directly sent by the user to the terminal device. Thus, the user can control the projection screen and control the terminal device synchronously, thereby improving the user experience.

[0156] As an implementation method of the present application, Figure 8 As shown, before the above step S201, i.e., before receiving the reverse control information sent by the projection screen, the following may also be included:

[0157] S205: When it is detected that an HDMI cable connection is established between the terminal device and a screen projection operation of the user is detected, a screen projection start instruction is sent to the projection screen, and a signal source of the projection screen is configured as an HDMI channel, so that the projection screen is switched to the projection mode;

[0158] S206: Acquire the picture data of the picture displayed by the terminal device through the HDMI cable connection with the terminal device, and send the picture data to the projection screen through the HDMI channel, so that the projection screen receives the picture data and displays the picture corresponding to the picture data.

[0159] The terminal device and the target projection device can be connected via an HDMI cable and a Touch cable respectively, wherein the HDMI cable is used to transmit projection data. In order to project the screen of the terminal device, an HDMI cable connection between the terminal device and the target projection device can be established, as well as a communication connection between the projection screen and the target projection device.

[0160] In this way, when an HDMI cable connection is established between the terminal device and the target projection device, the user can perform the projection operation. Among them, the user can perform the projection operation on the target projection device or on the terminal device.

[0161] As an implementation method, the user can click on the terminal device displayed on the target screen projection device that has an HDMI cable connection with the target screen projection device, select the screen projection option, and send a screen projection instruction to the target screen projection device.

[0162] As an implementation method, the user can click on the screen projection option displayed by the terminal device and send a screen projection instruction to the terminal device. The terminal device sends the screen projection instruction to the target screen projection device, and the target screen projection device can receive the screen projection instruction.

[0163] Based on this, when the target projection device detects that an HDMI cable connection is established between it and the terminal device and detects the user's projection operation, it can send a projection start command to the projection screen and configure the signal source of the projection screen to the HDMI channel. When the projection screen receives the projection start command, it can switch the working mode to projection mode.

[0164] As an implementation method, the target screen projection device can configure the signal source of the projection screen as an HDMI channel, that is, the target screen projection device can send a projection start instruction to the projection screen and configure the signal source of the projection screen as an HDMI channel.

[0165] As an implementation method, the projection screen can receive a projection start instruction sent by the target projection device and configure its own signal source as an HDMI channel.

[0166] Then, the target projection device can obtain the picture data of the picture displayed by the terminal device through the HDMI cable connection between the terminal device, and send the picture data to the HDMI channel of the projection screen through the network communication connection, so that the projection screen can obtain the picture data through the HDMI channel and display the picture corresponding to the picture data.

[0167] In addition, users can end screen projection through the target projection device, terminal device or projection screen. When the target projection mode detects the end of screen projection, it will no longer transmit the screen data of the terminal device to the projection screen.

[0168] As an implementation method, when the target screen projection device detects that the HDMI cable between the target screen projection device and the terminal device is disconnected and / or detects the user's screen projection end operation, it stops acquiring the screen data of the terminal device and sends a screen projection end instruction to the projection screen to turn off the projection mode of the projection screen. When the projection screen is to be ended, the user can directly disconnect the HDMI cable connection between the target screen projection device and the terminal device; or the user can keep the HDMI cable connection between the target screen projection device and the terminal device, but perform a screen projection end operation on the target screen projection device or the terminal device. The user can send a screen projection end instruction to the terminal device by clicking the screen projection end option displayed on the screen of the terminal device, or click the screen projection end button set on the target screen projection device to send a screen projection end instruction to the target screen projection device. When the user ends the screen projection operation on the terminal device, the terminal device can generate a screen projection end message based on the end of the screen projection operation, and send the screen projection end notification message to the target screen projection device, so that the target screen projection device stops obtaining the screen data of the terminal device based on the screen projection end message, and sends a screen projection end instruction to the projection screen, so that the projection screen turns off the projection mode based on the screen projection end instruction.

[0169] As an implementation method, when the target projection device receives the projection end counter-control message sent by the projection screen, it stops sending the screen data of the terminal device to the projection screen. When the user wants to end the projection, the user can click the projection end option in the projection screen, or click the projection end button set on the projection screen. When the projection screen detects the user's projection end operation, it can generate a projection end counter-control message about the target projection device, send the projection end counter-control message to the target projection device, and turn off the projection mode. When the target projection device receives the projection end counter-control message sent by the projection screen, it can stop sending the terminal device's screen data to the projection screen.

[0170] In this embodiment, the target projection device establishes communication connections with the terminal device and the projection screen respectively, and when receiving the projection operation of the user, the target projection device can send a projection start instruction to the projection screen and configure the signal source of the projection screen to be the HDMI channel. When the projection screen is in projection mode, the target projection device can obtain the picture data of the terminal device through the HDMI cable, and transmit the picture data of the terminal device to the projection screen through the communication connection, so as to realize efficient transmission of picture data.

[0171] As an implementation of the embodiment of the present application, a screen projection reverse control method provided in the embodiment of the present application may also include:

[0172] In the process of acquiring the picture data of the picture displayed by the terminal device through the HDMI line connection between the terminal device and the terminal device, the acquired picture data is recorded.

[0173] In a screen projection system provided in the present application, the target screen projection device may be an electronic device that integrates functions such as recording, video distribution, face recognition, and IoT control.

[0174] In order to record the projection content and facilitate subsequent viewing by users, the target projection device can record the acquired screen data while obtaining the screen data of the screen displayed by the terminal device through the HDMI cable connection between the terminal device and the terminal device.

[0175] As an implementation method, the target screen projection device can record the picture data obtained through the HDMI cable connection between it and the terminal device, and after the screen projection is completed, the recorded picture data can be stored in a preset storage area for archiving, or the recorded picture data can be sent to a preset receiving address so that the user can view the picture data during this screen projection process.

[0176] For example, in a teaching scenario, the teacher projects the courseware on the PC to the projection screen through the HDMI cable connection between the PC and the target projection device and the communication connection between the target projection device and the projection screen, and uses the projection screen to explain the courseware. During the projection process, the target projection device can record the screen data of the courseware obtained through the HDMI cable connection between it and the terminal device, and store the recorded screen data in the courseware database after the projection is completed, so that students can download the courseware in the database for review.

[0177] In this embodiment, the target screen projection device can record the image data of the screen displayed by the terminal device obtained through the HDMI cable connection between it and the terminal device during the screen projection process, so that the user can view the image data of the screen projection again later, thereby improving the user experience.

[0178] As an implementation manner of an embodiment of the present application, the at least one screen projection device includes the target screen projection device and at least one relay screen projection device, and the target screen projection device is connected to the projection screen through the at least one relay screen projection device.

[0179] The above-mentioned step S201, i.e., the step of receiving the reverse control information sent by the projection screen, may include: receiving the reverse control information forwarded by the relay projection device connected to the target projection device, wherein the reverse control information is sent by the projection screen to the relay projection device, or the reverse control information is the reverse control information sent by the projection screen and forwarded by other relay projection devices.

[0180] As shown in FIG. 9 ( a ), at least one projection device 320 in a projection system provided in an embodiment of the present application may include a target projection device 3201 and at least one relay projection device 3202 , and the target projection device 3201 is connected to the projection screen 330 via at least one relay projection device 3202 .

[0181] After the projection screen generates the counter-control information, it can send the counter-control information to the relay projection screen device that is connected to the projection screen. The relay projection screen device that is connected to the projection screen receives the counter-control information sent by the projection screen and forwards the counter-control information to the target projection screen device.

[0182] Among them, the number of relay screen projection devices between the target screen projection device and the projection screen can be set according to the actual application scenario.

[0183] As an implementation method, there is a relay screen projection device between the target screen projection device and the projection screen, and the reverse control information received by the target screen projection device is sent by the projection screen to the relay screen projection device that is in communication with the target screen projection device, and forwarded by the relay screen projection device to the target screen projection device. For example, as shown in FIG9 (b), the target screen projection device 3201 and the projection screen 330 are in different network segments, and there is a relay screen projection device 3202 between the two, which is a gateway device connecting the two network segments. The projection screen 330 sends the reverse control information to the relay screen projection device 3202, and then the relay screen projection device 3202 forwards it to the target screen projection device 3201.

[0184] As an implementation method, there are multiple relay screen projection devices between the target screen projection device and the projection screen, and the reverse control information received by the target screen projection device is the reverse control information sent by the projection screen forwarded by other relay screen projection devices. That is, the projection screen sends the reverse control information to the relay screen projection device connected to it in communication, and the relay screen projection device connected to the projection screen in communication receives the reverse control information and forwards the reverse control information to the next relay screen projection device until it is forwarded to the relay screen projection device connected to the target screen projection device in communication, and the relay screen projection device connected to the target screen projection device in communication forwards the reverse control information to the target screen projection device. For example, as shown in FIG9 (c), there are multiple relay screen projection devices 3202 between the target screen projection device 3201 and the projection screen 330, and the projection screen 330 only supports communication with the relay screen projection device A, and the target screen projection device 3201 only supports communication with the relay screen projection device B, and the relay screen projection device A is in communication connection with the relay screen projection device B. The projection screen 330 sends the reverse control information to the relay projection device A, the relay projection device A receives the reverse control information and forwards it to the relay projection device B, the relay projection device B receives the reverse control information and forwards it to the target projection device 3201.

[0185] In this embodiment, the target projection device and the projection screen are communicated through at least one relay projection device. The projection screen sends the reverse control information to the relay projection device, which is forwarded to the target projection device by at least one relay projection device and finally transmitted to the terminal device. In this way, the reverse control of the terminal device is finally achieved through the multi-level transmission of the reverse control information between multiple devices.

[0186] In order to facilitate understanding of a screen projection reverse control method provided in an embodiment of the present application, Fig.10 and Fig.11 The specific embodiment shown explains the connection relationship between the various devices in the screen projection system, and specifically explains the screen projection control and screen projection reverse control process achieved by the screen projection system.

[0187] The screen projection system includes a PC, an intelligent control engine and a large screen; wherein the PC is the terminal device in the embodiment of the present application, the intelligent control engine is the target screen projection device in the embodiment of the present application, and the large screen is the projection screen in the embodiment of the present application.

[0188] like Fig.10 As shown, the PC and the intelligent control engine are directly connected via an HDMI cable and Type-c, the intelligent control engine is connected to the large screen via an HDMI cable, and reverse control information is transmitted between the intelligent control engine and the large screen via the network, wherein IP and port (interface) for network communication are pre-set for the intelligent control engine and the large screen, that is, IP1 and port1 are set for the intelligent control engine, and IP2 and port2 are set for the large screen.

[0189] The intelligent control engine obtains the video signal of HDMI input through the HDMI cable connection between it and the PC, and sends the video signal to the large screen through the HDMI cable connection between it and the large screen, so that the large screen can obtain the HDMI signal and display the video picture.

[0190] When the large screen receives a control operation, it generates a counter-control message and feeds the counter-control message back to the intelligent control engine through the network connection with the intelligent control engine. The intelligent control engine can receive the counter-control message sent by the large screen and verify the IP2 and port2 included in the counter-control message. When the counter-control message passes the verification, the intelligent control engine uses its own event node to process the counter-control message into a control signal about the PC, and feeds the control signal back to the PC through its Type-c connection with the PC, so that the PC receives the control signal and responds to the control signal.

[0191] Specifically, Fig.11 As shown, the user can pre-configure the IP and port of the large screen, configure the IP and port of the intelligent control engine, and establish a connection between the PC and the intelligent control engine, that is, connect the PC and the intelligent control engine through an HDMI cable and a Touch cable.

[0192] When the user wants the large screen to reversely control the PC, the user can set the large screen to enable the reverse control function.

[0193] When the user wants to project the PC screen, the user can control the large screen to display the PC screen through the intelligent control engine, that is, the user turns on the projection operation on the intelligent control engine, so that the intelligent control engine configures the large screen signal source HDMI when the projection operation is detected.

[0194] Then the intelligent control engine obtains the screen data of the PC screen through the HDMI cable, and transmits the screen data to the large screen through the HDMI network connection with the large screen. The large screen obtains the HDMI channel screen and sets the stream screen as the current display content.

[0195] When the large screen displays the streaming image, the user can control the large screen through touch and mouse. When the large screen detects the control operation, it encapsulates the touch information and mouse information as reverse control information, and sends the reverse control information to the intelligent control engine through network communication based on its own IP and port.

[0196] The intelligent control engine receives the reverse control information and verifies the source of the information by verifying the IP information and port information in the reverse control information. After that, the position information of the large screen corresponding to the control operation in the reverse control information is converted into the position information of the PC screen corresponding to the control operation, and the reverse control information is encapsulated into an operation message supported by the kernel, and then the operation message is converted into a mouse signal or a touch signal through a virtual node. The mouse signal is sent to the PC via a Type-c connection, or the touch signal is sent to the PC via a Type-c connection.

[0197] The PC receives and responds according to the control signal, that is, it performs the operation indicated by the control signal on its own display.

[0198] When you want to control the large screen to display its own picture, you can configure the large screen signal source as its own picture channel through the intelligent control engine to control the large screen to display its own picture. When the large screen displays its own picture, the user can control the large screen by touching or using the mouse. When a control operation is detected, the large screen responds to the control operation and does not feedback the control information to the intelligent control engine.

[0199] If you do not want the large screen to reverse control the PC, you can disable the reverse control function. If you want to control the large screen to display the PC signal, you can configure the large screen signal source HDMI through the intelligent control engine so that the large screen can take the HDMI channel image. When the large screen displays the PC image, the user can control the large screen through touch or mouse. When the control operation is detected, the large screen responds to the control operation, but will not reverse control the PC.

[0200] Corresponding to the above-mentioned screen projection reverse control method, the embodiment of the present application also provides a screen projection reverse control device. The following is an introduction to a screen projection reverse control device provided by the embodiment of the present application. The screen projection reverse control device is applied to a target screen projection device directly connected to a terminal device in at least one screen projection device included in the screen projection system. The screen projection system also includes a terminal device and a screen projection screen, and at least one screen projection device is in communication connection with the screen projection screen. A screen projection reverse control device includes:

[0201] An information receiving module, used to receive the reverse control information sent by the projection screen, wherein the reverse control information is generated when the projection screen detects a control operation in the projection mode, and the reverse control information includes a target type of the control operation and position information corresponding to the control operation;

[0202] An encapsulation module, configured to encapsulate the reverse control information into an operation message supported by the kernel according to a target type of the control operation, wherein the operation message includes location information corresponding to the control operation;

[0203] a conversion module, configured to transmit the operation message to an event node corresponding to the type of the control operation, and convert the operation message into a control signal for the terminal device through the event node, wherein the event node is a virtual node pre-created by the kernel for responding to the operation message;

[0204] A signal sending module is used to send the control signal to the terminal device, so that the terminal device receives the control signal and performs the reverse control operation of the target type based on the position information.

[0205] In the solution provided in the embodiment of the present application, the reverse control information of the projection screen on the terminal device is transmitted to the terminal device by at least one projection device between the two, that is, the reverse control information of the projection screen on the terminal device is finally realized through the multi-level transmission of the reverse control information between the projection screen, at least one projection device and the terminal device. Compared with the projection screen reverse control method that relies on the direct communication connection between the terminal device and the projection screen, the projection screen reverse control method provided in the embodiment of the present application does not require direct communication between the terminal device and the projection screen, and is applicable to various projection screen reverse control scenarios. In addition, since the projection screen device can simulate itself as a control component through its own event node, convert the reverse control information into a control signal and send the control signal directly to the terminal device through a direct connection with the terminal device, it simulates the reverse control effect of transmitting the control signal received by the projection screen to the terminal device.

[0206] Corresponding to the above-mentioned screen projection reverse control method, the embodiment of the present application also provides a screen projection system, such as Figure 3 As shown, the screen projection system may include a terminal device 310, at least one screen projection device 320 and a projection screen 330, wherein at least one screen projection device 320 is communicatively connected with the projection screen 330, wherein:

[0207] The projection screen 330 is used to generate counter-control information when a control operation is detected in the projection mode, and send the counter-control information to a target projection device 3201 directly connected to the terminal device in the at least one projection device 320, wherein the counter-control information includes a target type of the control operation and position information corresponding to the control operation;

[0208] The target screen projection device 3201 is used to implement the screen projection reverse control method provided in the embodiment of the present application;

[0209] The terminal device 310 is used to receive the control signal sent by the target screen projection device 3201 and perform a reverse control operation of the target type of the control operation based on the position information corresponding to the control operation in the control signal.

[0210] It should be noted that the connection method of multiple screen projection devices in the screen projection system can also be that the multiple screen projection devices except the target screen projection device directly connected to the terminal device are connected in parallel, and the screen projection devices connected in parallel communicate with the target screen projection device and the projection screen respectively. This is all reasonable and no restriction is imposed on the connection method of each screen projection device in the screen projection system. Various connection methods that can realize the screen projection reverse control method provided by this application are within the protection scope of this application.

[0211] In the solution provided in the embodiment of the present application, the reverse control information of the projection screen on the terminal device is transmitted to the terminal device by at least one projection device between the two, that is, the reverse control information of the projection screen on the terminal device is finally realized through the multi-level transmission of the reverse control information between the projection screen, at least one projection device and the terminal device. Compared with the projection screen reverse control method that relies on the direct communication connection between the terminal device and the projection screen, the projection screen reverse control method provided in the embodiment of the present application does not require direct communication between the terminal device and the projection screen, and is applicable to various projection screen reverse control scenarios. In addition, since the projection screen device can simulate itself as a control component through its own event node, convert the reverse control information into a control signal and send the control signal directly to the terminal device through a direct connection with the terminal device, it simulates the reverse control effect of transmitting the control signal received by the projection screen to the terminal device.

[0212] The present application also provides a screen projection device, such as Fig.12 As shown, including:

[0213] Memory 1201, used for storing computer programs;

[0214] The processor 1202 is used to implement any of the above-mentioned embodiments when executing the program stored in the memory 1201, and to apply the screen projection reverse control method of the target screen projection device directly connected to the terminal device in at least one screen projection device included in the screen projection system.

[0215] In addition, the above-mentioned screen projection device may also include a communication bus and / or a communication interface, and the processor 1202, the communication interface, and the memory 1201 communicate with each other through the communication bus.

[0216] In the solution provided in the embodiment of the present application, the reverse control information of the projection screen on the terminal device is transmitted to the terminal device by at least one projection device between the two, that is, the reverse control information of the projection screen on the terminal device is finally realized through the multi-level transmission of the reverse control information between the projection screen, at least one projection device and the terminal device. Compared with the projection screen reverse control method that relies on the direct communication connection between the terminal device and the projection screen, the projection screen reverse control method provided in the embodiment of the present application does not require direct communication between the terminal device and the projection screen, and is applicable to various projection screen reverse control scenarios. In addition, since the projection screen device can simulate itself as a control component through its own event node, convert the reverse control information into a control signal and send the control signal directly to the terminal device through a direct connection with the terminal device, it simulates the reverse control effect of transmitting the control signal received by the projection screen to the terminal device.

[0217] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0218] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus, system, screen projection device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiment of the target screen projection device directly connected to the terminal device in at least one screen projection device included in the screen projection system, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0219] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A screen projection reverse control method, characterized in that: The method is applied to a target screen projection device directly connected to a terminal device in at least one screen projection device included in a screen projection system, wherein the screen projection system further includes a terminal device and a screen projection screen, and the at least one screen projection device is communicatively connected to the screen projection screen. The method includes: Receiving reverse control information sent by the projection screen, wherein the reverse control information is generated when the projection screen detects a control operation in the projection mode, and the reverse control information includes a target type of the control operation and position information corresponding to the control operation; According to the target type of the control operation, encapsulate the reverse control information into an operation message supported by the kernel, wherein the operation message includes position information corresponding to the control operation on the screen of the terminal device; Passing the operation message to an event node corresponding to the type of the control operation, and converting the operation message into a control signal for the terminal device through the event node, wherein the event node is a virtual node pre-created by the kernel for responding to the operation message; The control signal is sent to the terminal device, so that the terminal device receives the control signal and performs the reverse control operation of the target type based on the position information on the screen of the terminal device corresponding to the control operation.

2. The method according to claim 1, characterized in that The step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation comprises: Parsing the reverse control information to obtain the target type of the control operation and the position information corresponding to the control operation; According to the position information conversion method corresponding to the target type, the position information of the projection screen corresponding to the control operation is converted into the position information in the screen of the terminal device corresponding to the control operation; The location information on the screen of the terminal device and the target type are encapsulated into an operation message supported by the kernel according to the format of the operation message supported by the kernel.

3. The method according to claim 2, characterized in that The target types of the control operation include mouse events and touch events, and the event nodes include virtual mouse nodes and virtual touch nodes; The step of transmitting the operation message to an event node corresponding to the type of the control operation, and converting the operation message into a control signal for the terminal device through the event node, comprises: If the target type of the control operation is a mouse event, the operation message is transmitted to a virtual mouse node, and the operation message is converted into a mouse event control signal for the terminal device through the virtual mouse node; or, If the target type of the control operation is a touch event, the operation message is transmitted to a virtual touch node, and the operation message is converted into a touch event control signal for the terminal device through the virtual touch node.

4. The method according to claim 3, characterized in that: The target type of the control operation is a mouse event, and the position information corresponding to the control operation is the absolute coordinates of the mouse on the projection screen; The step of converting the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type includes: Calculate the ratio of the first coordinate in the first direction of the absolute coordinate to the resolution value in the first direction of the projection screen, and calculate the ratio of the second coordinate in the second direction of the absolute coordinate to the resolution value in the second direction of the projection screen to obtain normalized coordinates; Calculate the product of the third coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the terminal device, and calculate the product of the fourth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the terminal device to obtain the absolute coordinates of the mouse in the screen of the terminal device.

5. The method according to claim 3, characterized in that: The target type of the control operation is a mouse event, and the position information corresponding to the control operation is the relative coordinates of the current position of the mouse on the projection screen and the previous position of the mouse on the projection screen; The step of converting the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type includes: Acquire the previous absolute coordinates of the mouse on the projection screen, and calculate the sum of the previous absolute coordinates and the relative coordinates to obtain the current absolute coordinates of the mouse on the projection screen; Calculate the ratio of the first coordinate in the first direction of the current absolute coordinate to the resolution value in the first direction of the projection screen, and calculate the ratio of the second coordinate in the second direction of the current absolute coordinate to the resolution value in the second direction of the projection screen to obtain normalized coordinates; Calculate the product of the third coordinate in the first direction in the normalized coordinates and the resolution value in the first direction in the resolution of the terminal device, and calculate the product of the fourth coordinate in the second direction in the normalized coordinates and the resolution value in the second direction in the resolution of the terminal device to obtain the absolute coordinates of the mouse in the screen of the terminal device.

6. The method according to claim 3, characterized in that The target type of the control operation is a touch event, and the position information corresponding to the control operation is the normalized coordinates of the touch operation on the projection screen; The step of converting the position information of the projection screen corresponding to the control operation into the position information on the screen of the terminal device corresponding to the control operation according to the position information conversion method corresponding to the target type includes: Calculate the product of the fifth coordinate in the first direction of the normalized coordinates and the resolution value in the first direction of the resolution of the terminal device, and calculate the product of the sixth coordinate in the second direction of the normalized coordinates and the resolution value in the second direction of the resolution of the terminal device to obtain the absolute coordinates of the touch operation on the screen of the terminal device.

7. The method according to any one of claims 1 to 6, characterized in that: The target projection device and the projection screen communicate via a network, the target projection device stores pre-configured IP information and port information of the projection screen, and the reverse control information also includes IP information and port information of the projection screen for transmitting the reverse control information; Before the step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation, the method further includes: Verify whether the IP information included in the counter-control information is consistent with the IP information of the projection screen stored in itself, and whether the port information included in the counter-control information is consistent with the port information of the projection screen stored in itself; If the IP information included in the reverse control information is consistent with the IP information of the projection screen stored in itself, and the port information included in the reverse control information is consistent with the port information of the projection screen stored in itself, execute the step of encapsulating the reverse control information into an operation message supported by the kernel according to the target type of the control operation.

8. The method according to claim 7, characterized in that Before receiving the reverse control information sent by the projection screen, the method further includes: When it is detected that an HDMI cable connection is established between the terminal device and the user's screen projection operation is detected, a screen projection start instruction is sent to the projection screen, and the signal source of the projection screen is configured as an HDMI channel, so that the projection screen is switched to the projection mode; By connecting the terminal device through an HDMI cable, the picture data of the picture displayed by the terminal device is obtained, and the picture data is sent to the projection screen through the HDMI channel, so that the projection screen receives the picture data and displays the picture corresponding to the picture data.

9. The method according to any one of claims 1 to 6, characterized in that: The at least one screen projection device includes the target screen projection device and at least one relay screen projection device, and the target screen projection device is communicatively connected with the projection screen through the at least one relay screen projection device; The step of receiving the reverse control information sent by the projection screen includes: Receive the reverse control information forwarded by the relay projection device that is communicatively connected to the target projection device, wherein the reverse control information is sent by the projection screen to the relay projection device, or the reverse control information is sent by the projection screen and forwarded by other relay projection devices.

10. A screen projection system, characterized in that: The screen projection system includes a terminal device, at least one screen projection device and a screen projection screen, wherein the at least one screen projection device is communicatively connected with the screen projection screen, wherein: The projection screen is used to generate counter-control information when a control operation is detected in the projection mode, and send the counter-control information to a target projection device directly connected to the terminal device among the at least one projection device, wherein the counter-control information includes a target type of the control operation and position information corresponding to the control operation; The target screen projection device is used to implement the method described in any one of claims 1 to 9 above; The terminal device is used to receive the control signal sent by the target screen projection device and perform a reverse control operation of the target type of the control operation based on the position information in the screen of the terminal device corresponding to the control operation in the control signal.

11. A screen projection device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 9 when executing a program stored in a memory.

Images