Linkage control method, device and system, computer equipment and storage medium

Abstract

The invention relates to a linkage control method, device and system, computer equipment and a storage medium. The method is applied to a first device, a back-end process of the first device and a back-end process of a second device linked with the first device are deployed in the first device, and the method comprises the following steps: receiving a linkage control instruction triggered on a target device; respectively sending the linkage control instruction to a back-end process of the first equipment and a back-end process of the second equipment by adopting a message forwarding mechanism; and performing linkage control on the first equipment and the second equipment based on the linkage control instruction, the back-end process of the first equipment and the back-end process of the second equipment. By adopting the method provided by the invention, when one end device triggers the linkage control operation, the multi-end device can be simultaneously controlled to synchronously respond to the linkage control operation, so that the linkage control of the multi-end device is realized, the display interfaces of the multi-end device are ensured to have interface changes corresponding to the linkage control operation, and the user experience is improved. And synchronous change of interface contents of the multi-terminal equipment is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a linkage control method, device, system, computer equipment and storage medium. Background Art

[0002] With the development of medical technology and virtual reality technology, virtual reality technology has gradually been applied to the medical field. For example, a surgical navigation system may include a computer device (such as a Personal Computer, PC for short) and a virtual reality device (such as Augmented Reality Glasses, AR glasses for short). Usually, after the preoperative planning is completed on the PC, the PC will send the patient's imaging data and planning results to the AR glasses. Corresponding display interfaces are respectively provided on the PC and AR glasses. The display interface not only provides content display to users, but also provides users with some interface operation functions.

[0003] Traditionally, users can perform interface operations on the PC or AR glasses separately. However, the traditional interface operation method cannot achieve linkage control between the PC and AR glasses, resulting in inconsistency between what users see on the PC and AR glasses. Summary of the invention

[0004] Based on this, it is necessary to provide a linkage control method, device, system, computer equipment, computer-readable storage medium and computer program product that can realize multi-terminal linkage control to achieve content synchronization in response to the above technical problems.

[0005] In a first aspect, the present application provides a linkage control method. Applied to a first device, the first device is deployed with a backend process of the first device and a second device linked with the first device, the method comprising:

[0006] Receiving a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device;

[0007] Using a message forwarding mechanism, the linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively;

[0008] Based on the linkage control instruction, the backend process of the first device and the backend process of the second device, the first device and the second device are linkage controlled.

[0009] In one of the embodiments, a message forwarding service is deployed on the first device; receiving a linkage control instruction triggered on the target device includes:

[0010] Receiving linkage control instructions triggered on a target interface of a target device through a message forwarding service;

[0011] Accordingly, a message forwarding mechanism is adopted to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively, including:

[0012] The linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

[0013] In one embodiment, if the linkage control instruction is an interface control instruction for a target interface, the linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively through a message forwarding service, including:

[0014] Sending the interface control instruction to other devices through the message forwarding service to instruct the other devices to trigger the same interface control instruction; the other devices include the first device and the second device except the target device;

[0015] Through the message forwarding service, the interface control instructions triggered by the target device are sent to the backend process of the target device, and the interface control instructions triggered by other devices are sent to the backend processes of other devices.

[0016] In one embodiment, if the linkage control instruction is a data processing instruction for the target interface, the linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively through the message forwarding service, including:

[0017] The data processing instructions are sent to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

[0018] In one embodiment, a system control program is further deployed on the first device; the target device further includes a third device; and receiving a linkage control instruction triggered on the target device includes:

[0019] Receiving, through the system control program, a linkage control instruction triggered on a third device;

[0020] Accordingly, a message forwarding mechanism is adopted to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively, including:

[0021] The linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively through the system control program.

[0022] In one embodiment, based on the linkage control instruction, the backend process of the first device and the backend process of the second device, linkage control is performed on the first device and the second device, including:

[0023] Processing the linkage control instruction through the backend process of the first device to generate a processing result of the first device;

[0024] Processing the linkage control instruction through the backend process of the second device to generate a processing result of the second device;

[0025] The first device and the second device are controlled in linkage according to the processing result of the first device and the processing result of the second device.

[0026] In one embodiment, the first device and the second device are controlled in linkage according to the processing result of the first device and the processing result of the second device, including:

[0027] Through the message forwarding service, the processing result of the first device is sent to the front-end process of the first device, and the processing result of the second device is sent to the front-end process of the second device, so as to instruct the front-end process of the first device and the front-end process of the second device to execute the linkage control instruction.

[0028] In a second aspect, the present application further provides a linkage control device. Applied to a first device, the first device is deployed with a backend process of the first device and a backend process of a second device linked with the first device, the device comprising:

[0029] A receiving module, used to receive a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device;

[0030] A sending module, used to use a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively;

[0031] The control module is used to perform linkage control on the first device and the second device based on the linkage control instruction, the backend process of the first device and the backend process of the second device.

[0032] In a third aspect, the present application also provides a linkage control system, including a first device and a second device linked with the first device; wherein the first device has a backend process of the first device and a backend process of the second device deployed therein;

[0033] The first device is used to receive a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device;

[0034] The first device is also used to adopt a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively; and based on the linkage control instruction, the backend process of the first device and the backend process of the second device, the first device and the second device are linkage controlled.

[0035] In one embodiment, the first device includes a PC device, and the second device includes an AR glasses device.

[0036] In a fourth aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the linkage control method in the first aspect when executing the computer program.

[0037] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the linkage control method in the first aspect when executed by a processor.

[0038] In a sixth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the linkage control method in the first aspect are implemented.

[0039] The linkage control method, device, system, computer device, storage medium and computer program product are configured to deploy the backend process of the first device and the backend process of the second device linked to the first device on the first device, so that after receiving the linkage control instruction triggered on the target device, the first device can use the message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively; then, the first device performs linkage control on the first device and the second device based on the linkage control instruction, the backend process of the first device and the backend process of the second device. The target device includes any one of the first device and the second device. That is to say, when the backend processes of multiple devices linked to each other are deployed on the first device, after receiving the linkage control instruction sent by any one of the devices, the first device can send the linkage control instruction to the backend processes of each device linked to each other through the message forwarding mechanism, so that each device can respond to the linkage control operation triggered on the target device, thereby realizing linkage control of each device. By adopting the linkage control method proposed in the present application, when a linkage control operation is triggered on a device at one end, multiple devices at the same time can be controlled to synchronously respond to the linkage control operation, thereby realizing linkage control of multiple devices, ensuring that the display interfaces of the multiple devices all undergo interface changes corresponding to the linkage control operation, and realizing synchronous changes in the interface content of the multiple devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG. 1( a ) is a schematic diagram of a display interface of navigation control software on a PC in one embodiment;

[0041] FIG1( b ) is a schematic diagram of a display interface of navigation control software on an AR glasses end in one embodiment;

[0042] Figure 2 An application environment diagram of a linkage control method in an embodiment;

[0043] Figure 3 A schematic diagram of a process flow of a linkage control method in an embodiment;

[0044] Figure 4 A schematic diagram of a system structure of multi-terminal linkage in an embodiment;

[0045] Figure 5 A schematic flow chart of a linkage control method in another embodiment;

[0046] Figure 6 A schematic flow chart of a linkage control method in another embodiment;

[0047] Figure 7 It is a schematic diagram of a multi-terminal linkage control process corresponding to an interface control operation triggered on a display interface in one embodiment;

[0048] Figure 8 It is a schematic diagram of a multi-terminal linkage control flow corresponding to a data processing operation triggered on a display interface in one embodiment;

[0049] Fig. 9 A schematic diagram of a multi-terminal linkage system structure in another embodiment;

[0050] Fig.10 A schematic flow chart of a linkage control method in another embodiment;

[0051] Fig.11 is a schematic diagram of a multi-terminal linkage control flow corresponding to a data processing operation triggered by a third device in an embodiment;

[0052] Fig.12 is a structural block diagram of a linkage control device in an embodiment;

[0053] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] With the development of medical technology and virtual reality technology, virtual reality technology is gradually applied to the medical field. For example, for the surgical navigation system in the medical field, it can include a PC side and an AR glasses side; both the PC side and the AR glasses side are equipped with corresponding navigation software interfaces, and users can wear AR glasses and control handles to cooperate with the PC side to complete the registration and navigation steps.

[0056] Usually, the preoperative planning process is first performed on the PC to complete the tissue segmentation, path planning, and marker point extraction of the patient's image data, and generate the planning results. When entering the navigation step, the PC and AR glasses can interact at the same time; the AR glasses will load the patient's image data and planning results, and enter the patient registration process to complete the registration preparation, marker point collection, and accuracy verification processes. After registration is completed, the navigation and positioning process is entered to complete the coarse and fine adjustment of the guide, and then the path navigation positioning is realized according to the planned path.

[0057] In the registration and navigation steps, both the PC and AR glasses can observe the patient's two-dimensional and three-dimensional images, and can also complete the control of some interface processes; among them, for the PC, it can display the complete navigation control software interface, and for the AR glasses, it can only display some major interface elements, such as process control, patient images, registration pens, navigation probes, etc. In addition, for the AR glasses and PC, the interface content and interface layout of the navigation control software interface of the two can be different. As shown in reference Figure 1, Figure 1 (a) shows an interface diagram of one of the interfaces of the navigation control software on the PC, and Figure 1 (b) shows an interface diagram corresponding to the current interface displayed by the AR glasses and the PC, including the AR left eye interface and the AR right eye interface.

[0058] For users, they can perform related interface operations in the navigation control software interface on the PC side, and they can also perform related interface operations in the navigation control software interface on the AR glasses side; however, currently, when users perform interface operations on the PC side or the AR glasses side, they cannot achieve linkage control between the PC side and the AR glasses side, resulting in inconsistent content seen by users on the PC side and the AR glasses side.

[0059] In the related art, if you want to share multiple screens, you need to turn on the remote sharing function. The client is remotely connected, and the host authorizes the client to view and operate. The remote user can operate the shared desktop to realize the functions of remote viewing and control. However, this screen sharing method is only applicable to the synchronous linkage of multiple PC terminals, and users operate and use the same software interface; because the PC terminal and the AR glasses terminal in this application use different software interfaces respectively, the traditional screen sharing method is not applicable to the synchronous linkage operation of multiple devices using different software interfaces.

[0060] Based on this, the present application proposes a linkage control method, which can realize the linkage control of multi-terminal devices, so that after the user performs an interface operation on one end, it will be synchronously fed back to the other end; in this way, in the surgical navigation system, the PC end and the AR glasses end can synchronously display the same picture. Through the joint use of the PC end and the AR glasses end in surgical operations, the user experience can be greatly improved to better assist users in performing related surgical operations.

[0061] The linkage control method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown, the first device 102 is connected to at least one second device 104 in communication; when the first device 102 and the second device 104 access the target interface of the same software, the target interface displayed on the first device 102 is not completely the same as the target interface displayed on the second device 104; the target interfaces displayed by each second device 104 may be completely the same or may not be completely the same.

[0062] In other words, the first device 102 and the second device 104 may use different front-end operating programs and different back-end operating programs to access the same software, so that the display interfaces of the first device 102 and the second device 104 are not completely the same. For each second device 104, it may use the same front-end operating program and back-end operating program, or different front-end operating programs and back-end operating programs; that is, the display interfaces of each second device 104 may be completely the same, or may not be completely the same.

[0063] Exemplarily, the first device 102 can access the target interface of the software by running the installed software; the second device 104 can access the target interface by inputting a specific URL through a browser.

[0064] Furthermore, by deploying the backend running program of the first device 102 and the backend running programs of each second device 104 in the first device 102, when the first device 102 accesses the target interface, the backend process of the first device 102 can be created in the first device 102; when the second device 104 accesses the target interface, the backend process of the second device 104 can also be created in the first device 102. In this way, the first device 102 can control the backend processes of the first device 102 and each second device 104, thereby realizing the linkage control of the first device 102 and each second device 104.

[0065] Among them, the first device 102 and the second device 104 can be different types of terminal devices, including but not limited to various personal computers, laptops, smart phones, tablets, virtual reality devices, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc.; portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0066] In one embodiment, Figure 3 As shown, a linkage control method is provided, which is applied to Figure 2 Taking the first device in the example as an example, the method includes the following steps:

[0067] Step 302: receiving a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device.

[0068] That is to say, when the first device is performing linkage control with each second device, the linkage control instruction can be triggered by the first device, that is, each second device synchronously performs the same control operation according to the linkage control operation of the first device, so that the display interface of each second device changes synchronously with the display interface of the first device; in addition, when performing linkage control, the linkage control instruction can also be triggered by any second device, that is, when one of the second devices triggers the linkage control operation, the first device and the other second devices can also synchronously perform the same control operation, so that the display interface of the first device and the display interface of each second device can change synchronously.

[0069] Exemplarily, the first device can receive a linkage control instruction sent by any target device; the linkage control instruction is used to perform linkage control on the target device and at least one linkage device; wherein, the at least one linkage device may include at least one device of the first device and the second device except the target device.

[0070] For example, when performing device linkage, all the mutually related devices can be controlled in linkage, or some of the mutually related devices can be controlled in linkage, so as to realize the device linkage requirements in multiple scenarios. For this case, for example, the linkage control instruction can also carry the identification of the linkage device; of course, when the linkage control instruction carries the ALL identification, it can indicate that all the mutually related devices are controlled in linkage.

[0071] Step 304: Use a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively.

[0072] Exemplarily, the backend process of the first device and the backend processes of each second device are running in the first device; then, when the first device receives a linkage control instruction sent by the target device, the linkage control instruction can be sent to the backend process of the first device and the backend process of each second device respectively based on the message forwarding mechanism; thereby, the first device and each second device can both execute the linkage control operation triggered on the target device.

[0073] Exemplarily, when the joint control instruction carries the identifier of at least one linkage device, the first device can send the joint control instruction to the backend process of the target device and the backend process of each linkage device corresponding to the identifier of each linkage device based on the message forwarding mechanism according to the identifier of each linkage device, so that the target device and each linkage device can synchronously execute the joint control operation triggered on the target device. In this way, the user can be provided with a selective device linkage function and the flexibility of device linkage can be improved.

[0074] Step 306: Based on the linkage control instruction, the backend process of the first device and the backend process of the second device, linkage control is performed on the first device and the second device.

[0075] Exemplarily, after the first device sends the linkage control instruction to the back-end process of the first device and the back-end process of the second device respectively, the linkage control instruction can be processed by the back-end process of the first device to generate a processing result of the first device; and the linkage control instruction can be processed by the back-end process of the second device to generate a processing result of the second device; then, the first device and the second device can be linkage controlled according to the processing result of the first device and the processing result of the second device.

[0076] Exemplarily, the first device can send the processing result of the first device to the front-end process of the first device to instruct the front-end process of the first device to execute the linkage control instruction and update the display interface of the first device; and send the processing result of the second device to the front-end process of the second device to instruct the front-end process of the second device to execute the linkage control instruction and update the display interface of the second device. In this way, the control operation triggered on the target device can be linked to control other devices to synchronously execute the control operation; that is, linkage control of multiple devices is realized, so that the display content in the display interface of the multiple devices remains consistent.

[0077] In the above linkage control method, by deploying the backend process of the first device and the backend process of the second device linked with the first device on the first device, the first device can adopt a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively after receiving the linkage control instruction triggered on the target device; then, the first device performs linkage control on the first device and the second device based on the linkage control instruction, the backend process of the first device and the backend process of the second device. The target device includes any one of the first device and the second device. That is to say, when the backend processes of multiple devices linked with each other are deployed on the first device, after receiving the linkage control instruction sent by any one of the devices, the first device can send the linkage control instruction to the backend processes of each device linked with each other through the message forwarding mechanism, so that each device can respond to the linkage control operation triggered on the target device, thereby realizing linkage control of each device. By adopting the linkage control method proposed in the present application, when a linkage control operation is triggered on a device at one end, multiple devices at the same time can be controlled to synchronously respond to the linkage control operation, thereby realizing linkage control of multiple devices, ensuring that the display interfaces of the multiple devices all undergo interface changes corresponding to the linkage control operation, and realizing synchronous changes in the interface content of the multiple devices.

[0078] In one embodiment, a message forwarding service may also be deployed in the first device. Based on the message forwarding service, the communication connection between the front-end process of the first device and the back-end process of the first device may be realized, and the communication connection between the front-end process of the second device and the back-end process of the second device may also be realized; that is, the message forwarding service may simultaneously connect the front-end process of the first device and the back-end process of the first device, as well as the front-end process of the second device and the back-end process of the second device; the system structure diagram of the first device may refer to Figure 4 As shown. Exemplarily, the message forwarding service may be a Web service or other types of message forwarding services.

[0079] Based on this, Figure 5 As shown, for the above step 302, receiving the linkage control instruction triggered on the target device may include:

[0080] Step 302a: receiving the linkage control instruction triggered on the target interface of the target device through the message forwarding service.

[0081] Accordingly, for the above step 304, using a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively may include:

[0082] Step 304a: Send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

[0083] That is to say, when the user triggers the linkage control operation on the display interface of any target device, the front-end process of the target device monitors the linkage control operation and sends a linkage control instruction to the message forwarding service deployed on the first device; then, the message forwarding service sends the linkage control instruction to the back-end process of the first device and the back-end process of the second device respectively; the back-end process of the first device processes the linkage control instruction to generate a processing result of the first device; the back-end process of the second device processes the linkage control instruction to generate a processing result of the second device; then, the back-end process of the first device feeds back the processing result of the first device to the message forwarding service, and sends the processing result of the first device to the front-end process of the first device through the message forwarding service; similarly, the back-end process of the second device feeds back the processing result of the second device to the message forwarding service, and sends the processing result of the second device to the front-end process of the second device through the message forwarding service; thereby instructing the front-end process of the first device and the front-end process of the second device to execute the linkage control instruction, and finally realizing the linkage control of the first device and the second device.

[0084] In this embodiment, by deploying a message forwarding service in the first device, and connecting with the front-end process and the back-end process of the first device through the message forwarding service, and connecting with the front-end process and the back-end process of the second device through the message forwarding service, the first device can communicate and connect with the back-end processes of each second device respectively, so as to realize the joint control of each second device by the first device; in addition, through the message forwarding service, the first device can also respond to the linkage control operation of each second device, so that the second device can also realize the linkage control of the first device; the message forwarding service in this embodiment can provide a communication connection basis for realizing multi-terminal linkage, and provide technical support for the mutual linkage of multiple terminals.

[0085] In one embodiment, the above-mentioned linkage control instructions may include interface control instructions for the target interface, and data processing instructions for the target interface; wherein the interface control instructions include but are not limited to control instructions corresponding to interface control operations such as interface refresh and interface jump; data processing instructions include but are not limited to instructions corresponding to any processing operations on the data in the interface, such as: when the interface includes image data, the processing operation may include rotating, translating, scaling and other processing operations on the image.

[0086] The following describes the linkage control process in detail for different linkage control instructions.

[0087] The first type: When the linkage control instruction is an interface control instruction for the target interface, such as Figure 6 As shown, the above step 302a, sending the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service, may include:

[0088] Step 602: Send the interface control instruction to other devices through the message forwarding service to instruct other devices to trigger the same interface control instruction.

[0089] The other devices include the devices other than the target device in the first device and the second device. Of course, when the linkage control instruction includes the identification of the linkage device, the other devices may also include the linkage devices other than the target device in the first device and the second device.

[0090] That is to say, when the user triggers an interface control operation for the target interface of the target device on the target device, since the interface update may also involve the update of interface content such as interface elements, layout, color, etc., and the update of interface content requires the processing of the device front-end process; therefore, when the linkage control instruction is an interface control instruction, after receiving the interface control instruction sent by the target device, the message forwarding service needs to send the interface control instruction to other devices, and instruct the other devices to also trigger the same interface control instruction; that is, control other devices to simulate triggering the interface control operation, and send the interface control instruction to the message forwarding service through the front-end process of other devices.

[0091] Step 604: send the interface control instruction triggered by the target device to the backend process of the target device through the message forwarding service, and send the interface control instruction triggered by other devices to the backend processes of other devices.

[0092] When controlling other devices also triggers the same interface control operation, at this time, the message forwarding service not only receives the interface control instructions sent by the target device, but also receives the interface control instructions sent by other devices; and the interface control instructions sent by each device are interface control instructions triggered by the same interface control operation; which is equivalent to that each device triggers the same interface control operation at the same time.

[0093] Based on this, the message forwarding service can send the interface control instructions triggered by the target device to the backend process of the target device, and send the interface control instructions triggered by other devices to the backend processes of other devices respectively.

[0094] For example, in the case where the first device is a PC device, the second device is an AR device, and the message forwarding service is a Web service, it is assumed that the user triggers an interface control operation in the display interface of the PC device (as the target device). Figure 7 As shown, the corresponding linkage control process may include:

[0095] Step 1a: The PC front end detects the interface control operation triggered by the user and sends an interface control instruction to the Web service.

[0096] Step 0: The Web service sends the interface control instruction to the AR front end to instruct the AR front end to trigger the same interface control operation.

[0097] Step 1b: The AR front end sends an interface control instruction to the Web service.

[0098] Step 2a, the Web service sends the interface control instruction sent by the PC front end to the PC back end.

[0099] Step 2b: The Web service sends the interface control instructions sent by the AR front end to the AR back end.

[0100] Step 3a: The Web service receives the processing result of the PC device sent by the PC backend.

[0101] Step 3b: The Web service receives the processing result of the AR device sent by the AR backend.

[0102] In step 4a, the Web service sends the processing result of the PC device to the PC front end to instruct the PC front end to update the display interface of the PC device based on the processing result of the PC device.

[0103] In step 4b, the Web service sends the processing result of the AR device to the AR front end to instruct the AR front end to update the display interface of the AR device based on the processing result of the AR device.

[0104] Based on the above steps, when the user performs interface operations on the PC device, the AR device can be controlled to perform the same operations simultaneously, so that the display interfaces of the PC device and the AR device undergo the same interface changes at the same time.

[0105] Referring to the above steps, it is also possible to implement interface operations on the AR device and control the PC device to undergo the same interface changes. That is to say, using the above method, after the PC front end or the AR front end controls the process, it will send an interface control instruction to the Web service, and the Web service will forward the interface control instruction to other front ends. Other front ends passively trigger the same interface control instruction to the Web service, and then the Web service will send the interface control instruction to their respective backend processes. Figure 7 In the example, the message flow corresponding to step 0 is the message flow forwarded by the Web service to other front ends.

[0106] In addition, it should be noted that when the Web service executes the above step 2a, the Web service can execute step 2a after executing step 1a; or it can synchronously execute step 2a and step 2b after executing step 1b; that is, the Web service can send the interface control instruction to the back-end process corresponding to the target device after receiving the interface control instruction sent by the target device, so as to perform the interface control operation on the target device; the Web service can also synchronously execute the step of sending the interface control instruction of each device to the back-end process of each device respectively after receiving the interface control instruction sent by other devices, so as to synchronously control each device to perform the interface control operation. In this way, the consistency and synchronization of the linkage control of multiple devices can be improved, and the response delay between devices can be reduced.

[0107] The following describes the linkage control process using a specific interface control operation.

[0108] Assume that the confirmation button is clicked on the PC or AR side. Here, the PC operation is taken as an example. If the confirmation button is clicked on the PC, the display interface of the PC will switch from the current interface to the next interface, and send a command to switch the image layout to its backend through the Web service.

[0109] a. The Web service receives the confirmation button command sent by the PC and forwards the confirmation button command to other AR front ends.

[0110] b. After the AR front end receives the confirmation button command, it also executes the confirmation button command.

[0111] c. Both device front ends send instructions to switch image layout to the Web service.

[0112] d. The Web service forwards the layout switching instruction to the respective backend process.

[0113] e. The backend process switches the layout, updates the image display, and sends the image data to the Web service.

[0114] f. The Web service forwards the image data to the respective front-end processes.

[0115] g. The front-end process parses and displays the image data, so that the two device front-ends display images with the same layout.

[0116] Second: When the linkage control instruction is a data processing instruction for the target interface, the above step 302a, sending the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service, may include:

[0117] The data processing instructions are sent to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

[0118] That is to say, when the user processes the data in the target interface, the data displayed in the interface is provided to the front-end interface by the back-end process, and when the data changes, only the data in the front-end interface changes, and no other content in the interface changes; therefore, for data processing instructions, the message forwarding service can directly send the data processing instructions to the back-end process of each device, so that after the back-end process of each device processes the data processing instructions respectively, the processing results of each device are sent to the front-end process of each device respectively, so that the front-end process of each device can update the data in the display interface of each device based on the processing results of each device.

[0119] For example, in the case where the first device is a PC device, the second device is an AR device, and the message forwarding service is a Web service, it is assumed that the user triggers a data processing operation in the display interface of the PC device (as the target device). Figure 8 As shown, the corresponding linkage control process may include:

[0120] Step 1a: The PC front-end process detects the data processing operation triggered by the user and sends a data processing instruction to the Web service.

[0121] Step 2a, the Web service sends the data processing instruction sent by the PC front-end process to the PC back-end process.

[0122] Step 2b: The Web service sends the data processing instructions sent by the PC front-end process to the AR back-end process.

[0123] Step 3a: The Web service receives the processing result of the PC device sent by the PC backend process.

[0124] Step 3b: The Web service receives the processing result of the AR device sent by the AR backend process.

[0125] In step 4a, the Web service sends the processing result of the PC device to the PC front-end process to instruct the PC front-end process to update the data in the display interface of the PC device based on the processing result of the PC device.

[0126] In step 4b, the Web service sends the processing result of the AR device to the AR front-end process to instruct the AR front-end process to update the data in the display interface of the AR device based on the processing result of the AR device.

[0127] Based on the above steps, when the user operates the data on the PC device, the data in the display interface of the AR device can be controlled to undergo the same change operation at the same time, so that the data in the display interfaces of the PC device and the AR device undergo the same change at the same time.

[0128] With reference to the above steps, it is also possible to implement data processing operations on the AR device and control the PC device to undergo the same data processing changes. That is to say, using the above method, after the PC front end or the AR front end performs operations such as translation, scaling, and rotation on the image, it will send data processing instructions to the Web service. Since image operations are mainly processed on the back end and updated to the front end, the Web service directly forwards the data processing instructions to all back ends. After processing the message, the back end completes image rendering and refresh, and forwards the generated new impact data to the Web service, which then forwards it to their respective front end processes, and the front end parses the image data for display. Among them, the above Figure 8 In the example, steps 2a and 2b are the message flows forwarded by the Web service to all backends.

[0129] The following describes the process of linkage control using a specific data processing operation.

[0130] Assume that a PC or AR terminal performs a flip operation on a three-dimensional image. Here, the operation on the PC is taken as an example.

[0131] a. The user flips the 3D image on the PC front-end interface, and the PC front-end process sends the coordinate data of the starting point before the 3D image flips and the coordinate data of the ending point after the flip to the Web service.

[0132] b. The Web service forwards the starting point coordinate data and the ending point coordinate data of the 3D image to all backend processes.

[0133] c. All backend processes parse the starting point coordinate data and the ending point coordinate data of the three-dimensional image respectively, perform a flip operation on the three-dimensional image, and generate new three-dimensional image data.

[0134] d. Each backend process sends the generated new 3D image data to the Web service.

[0135] e. The Web service forwards the new 3D image data sent by each backend process to the frontend process corresponding to each backend process, and each frontend process outputs the new 3D image data to each frontend interface for display; in this way, the 3D image on the PC side is flipped, and the 3D image on the AR side is also flipped.

[0136] The above content describes the implementation steps and control flow of linkage control of multiple devices based on the operations performed by the user on the front-end interface of each device. When a user triggers an operation on the display interface of a device at one end, not only can the display interface of the device be controlled to change, but the display interface of other devices can also be controlled to change in linkage, so that the content viewed by the user on the display interface of multiple devices is consistent.

[0137] In one embodiment, the data displayed in the display interface of the first device and the second device may also come from a third device, that is, the data required in the display interface of the first device and the second device is obtained through the third device, and the data is sent to the first device and the second device through the third device. Assuming that the data obtained by the third device is real-time changing data, then, when the data changes, the third device should synchronously send the changed data to the display interface of the first device for display, and to the display interface of the second device for display; to ensure that the data displayed on the display interface of the first device and the display interface of the second device are consistent. In this way, it can be ensured that the data viewed by the user on the first device and the second device are consistent, avoiding the inability of the user to accurately know which is the correct data when the data is inconsistent.

[0138] Exemplarily, a system control program may be deployed on the first device; the system control program may be used to establish a communication connection with the third device and receive data sent by the third device. Exemplarily, a lower-level computer software may be deployed on the third device, and the lower-level computer software may communicate with the system control software in the first device; for example, the lower-level computer software in the third device may perform two-way communication and two-way data transmission with the system control software in the first device.

[0139] In some implementations, the slave software in the third device can obtain the data required in the display interfaces of the first device and the second device in real time to provide real-time data support for users; for example: in a surgical navigation system, the slave software in the third device can monitor the position, status and other data of surgical instruments in real time, and transmit these data to the first device and the second device in real time, and output them for display; allowing users to grasp information such as the position and status of surgical instruments in real time.

[0140] refer to Fig. 9As shown, it shows a schematic diagram of the structure of a first device. On the basis of the above embodiment, the first device is also deployed with system control software for communicating with the lower computer software in the third device. In addition, the system control software can also communicate with the back-end process of the first device and the second device respectively; for example, the system control software can communicate with the back-end process of the PC device and the back-end process of the AR device respectively.

[0141] Based on this, in this embodiment, the target device may further include a third device; Fig.10 As shown, in the above step 302, receiving the linkage control instruction triggered on the target device may also include:

[0142] Step 302b: receiving the linkage control instruction triggered on the third device through the system control program.

[0143] Accordingly, in the above step 304, a message forwarding mechanism is adopted to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively, which may also include:

[0144] Step 304b: Send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the system control program.

[0145] That is to say, in this embodiment, the first device can also receive the linkage control instructions sent by the lower computer software in the third device through the system control software deployed thereon; the linkage control instructions can carry the target data monitored by the third device; then, the system control software can send the received linkage control instructions to the back-end process of the first device and the back-end process of the second device respectively; so that after the back-end process of the first device processes the linkage control instructions, the processing result of the first device is obtained; and after the back-end process of the second device processes the linkage control instructions, the processing result of the second device is obtained.

[0146] Furthermore, the backend process of the first device can send the processing result of the first device to the message forwarding service; and send the processing result of the first device to the front-end process of the first device through the message forwarding service; and then update the display interface of the first device based on the processing result of the first device through the front-end process of the first device.

[0147] Similarly, the back-end process of the second device can send the processing result of the second device to the message forwarding service; and send the processing result of the second device to the front-end process of the second device through the message forwarding service; and then update the display interface of the second device based on the processing result of the second device through the front-end process of the second device.

[0148] At this point, linkage control of the first device and the second device can be achieved, so that the display content in the display interface of the first device is consistent with the display content in the display interface of the second device.

[0149] For example, when the first device is a PC device, the second device is an AR device, and the message forwarding service is a Web service, assuming that the position or status data of the surgical instrument can be monitored in real time through the lower computer software in the third device, then, reference Fig.11 As shown, the control process for synchronously updating data between the PC and AR terminals may include:

[0150] Step 1: The lower computer software actively reports some device position data or status data to the system control software;

[0151] Step 2a, the system control software sends this data to the PC backend process.

[0152] In step 2b, the system control software sends this data to the AR backend process.

[0153] Step 3a: After the PC backend process processes the data, the processing result corresponding to the PC device is sent to the Web service.

[0154] Step 3b: After the AR backend process processes the data, the processing result corresponding to the AR device is sent to the Web service.

[0155] In step 4a, the Web service sends the processing result corresponding to the PC device to the PC front-end process, so that the PC front-end process outputs the data on the display interface of the PC device.

[0156] In step 4b, the Web service sends the processing result corresponding to the AR device to the AR front-end process, so that the AR front-end process outputs the data on the display interface of the AR device.

[0157] For example, in the above steps 2a and 2b, the system control software can forward the data to all backend processes in the form of events; then, after the backend processes have processed the data, they are sent to the Web service, which then forwards the data to the respective frontend processes for device position update or status update. Fig.11 In the example, the message flows corresponding to step 2a and step 2b are the message flows forwarded by the system control software to all backends.

[0158] The following describes the process of linkage control using a specific data operation.

[0159] Assume that the lower computer reports the probe position data.

[0160] a. The lower computer collects the probe position and reports the probe position data to the system control software; wherein the probe position data may include the probe head coordinate data and the probe tail coordinate data.

[0161] b. The system control software forwards the probe position data to all backend processes in the form of events.

[0162] c. Each backend process parses the probe position data respectively, and updates the position of the probe displayed on the image according to the probe head coordinate data and the probe tail coordinate data, and generates new image data.

[0163] d. Each backend process sends the generated new image data to the Web service.

[0164] e. The Web service forwards the new image data sent by each backend process to the frontend process corresponding to each backend process, and each frontend process outputs the new image data to each frontend interface for display; in this way, both the PC and AR ends can update and display the position of the probe in real time.

[0165] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0166] Based on the same inventive concept, the embodiment of the present application also provides a linkage control device for implementing the linkage control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more linkage control device embodiments provided below can refer to the limitations on the linkage control method above, and will not be repeated here.

[0167] In one embodiment, Fig.12 As shown, a linkage control device is provided, which is applied to a first device, in which a backend process of the first device and a backend process of a second device linked with the first device are deployed. The device includes: a receiving module 1202, a sending module 1204 and a control module 1206, wherein:

[0168] The receiving module 1202 is used to receive the linkage control instruction triggered on the target device; the target device includes any one of the first device and the second device.

[0169] The sending module 1204 is used to adopt a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively.

[0170] The control module 1206 is used to perform linkage control on the first device and the second device based on the linkage control instruction, the backend process of the first device and the backend process of the second device.

[0171] In one embodiment, a message forwarding service is deployed on the first device; the receiving module 1202 includes:

[0172] A first receiving submodule, configured to receive a linkage control instruction triggered on a target interface of a target device through a message forwarding service;

[0173] Accordingly, the sending module 1204 includes:

[0174] The first sending submodule is used to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

[0175] In one embodiment, if the linkage control instruction is an interface control instruction for a target interface, the first sending submodule includes:

[0176] A first sending unit, configured to send the interface control instruction to other devices through a message forwarding service, so as to instruct the other devices to trigger the same interface control instruction; the other devices include devices other than the target device in the first device and the second device;

[0177] The second sending unit is used to send the interface control instruction triggered by the target device to the backend process of the target device through the message forwarding service, and send the interface control instruction triggered by other devices to the backend processes of other devices.

[0178] In one embodiment, if the linkage control instruction is a data processing instruction for the target interface, the first sending submodule includes:

[0179] The third sending unit is used to send the data processing instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

[0180] In one embodiment, a system control program is also deployed on the first device; the target device also includes a third device; the receiving module 1202 includes:

[0181] A second receiving submodule, used for receiving a linkage control instruction triggered on a third device through a system control program;

[0182] Accordingly, the sending module 1204 includes:

[0183] The second sending submodule is used to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the system control program.

[0184] In one embodiment, the control module 1206 includes:

[0185] A first processing submodule, configured to process the linkage control instruction through a backend process of the first device to generate a processing result of the first device;

[0186] A second processing submodule, used for processing the linkage control instruction through a backend process of the second device to generate a processing result of the second device;

[0187] The control submodule is used to perform linkage control on the first device and the second device according to the processing result of the first device and the processing result of the second device.

[0188] In one of the embodiments, the control submodule is used to send the processing result of the first device to the front-end process of the first device through a message forwarding service, and send the processing result of the second device to the front-end process of the second device, so as to instruct the front-end process of the first device and the front-end process of the second device to execute linkage control instructions.

[0189] Each module in the above linkage control device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0190] In one embodiment, a linkage control system is provided. Figure 2 The linkage control system includes a first device and a second device linked with the first device; wherein the backend process of the first device and the backend process of the second device are deployed in the first device.

[0191] Exemplarily, the first device is used to receive a linkage control instruction triggered on a target device; wherein the target device may include any one of the first device and the second device; the first device is also used to adopt a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively; and based on the linkage control instruction, the backend process of the first device and the backend process of the second device, the first device and the second device are linkage controlled.

[0192] That is to say, for multiple devices in a linkage control system, when the back-end processes of each device are deployed on the same device in the system, each device can trigger a linkage control instruction and send the linkage control instruction to the same device; at this time, the linkage control instruction can be synchronously sent to the back-end process of each device through the same device, so that each device can respond to the linkage control instruction, thereby realizing linkage control of multiple terminal devices.

[0193] For specific implementation methods, please refer to the above Figure 3 The illustrated embodiments will not be repeated here.

[0194] In one implementation scenario, the linkage control system may be a surgical navigation linkage control system, the first device in the linkage control system may include a PC device, and the second device in the linkage control system may include an AR glasses device. That is, in this example, the backend process of the PC device and the backend process of the AR glasses device may both be deployed in the PC device.

[0195] For example, for navigation control software suitable for surgical navigation linkage control system, it can be deployed on PC-side device. After the PC-side device is started, the navigation control software can be run and the navigation control interface for the PC-side device can be entered. At this time, the PC-side device can create a backend process for the PC-side device. In addition, the AR glasses-side device can access the navigation control software and enter the navigation control interface of the AR glasses-side device by opening a specific URL in the browser. At the same time, a new backend process for the AR glasses-side device will be created on the PC-side device.

[0196] Based on this, the user can freely operate the navigation control interface on the PC device or the AR glasses device to realize the linkage control of the PC device and the AR glasses device. For example: the user can operate the navigation control interface on the PC device, and control the AR glasses device to perform the operation in linkage, so that the AR glasses device and the PC device maintain the same interface content. Alternatively, the user can operate the navigation control interface on the AR glasses device, and control the PC device to perform the operation in linkage, so that the PC device and the AR glasses device maintain the same interface content.

[0197] In this embodiment, the backend process of each device in the linkage control system is deployed on the first device in the linkage control system, and the linkage control instruction sent by any target device in the linkage control system is received through the first device; then, the first device sends the linkage control instruction to the backend processes of each device that is linked to each other through a message forwarding mechanism, so that each device can respond to the linkage control operation triggered on the target device, thereby realizing linkage control of each device. That is, when the linkage control operation is triggered by a device at one end, the linkage control system can simultaneously control multiple devices to synchronously respond to the linkage control operation, thereby realizing linkage control of multiple devices, ensuring that the display interfaces of the multiple devices all undergo interface changes corresponding to the linkage control operation, and realizing synchronous changes in the interface content of the multiple devices.

[0198] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a linkage control method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the computer device housing, or an external keyboard, touch pad or mouse, etc.

[0199] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the linkage control method in any of the above embodiments are implemented.

[0201] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the linkage control method in any of the above embodiments are implemented.

[0202] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the linkage control method in any of the above embodiments when executed by a processor.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A linkage control method, characterized in that: Applied to a first device, in which a backend process of the first device and a backend process of a second device linked with the first device are deployed, the method includes: Receiving a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device; Using a message forwarding mechanism, the linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively; Based on the linkage control instruction, the backend process of the first device and the backend process of the second device, the first device and the second device are linkage controlled.

2. The method according to claim 1, characterized in that: A message forwarding service is deployed on the first device; and the receiving of a linkage control instruction triggered on the target device includes: Receiving, through the message forwarding service, a linkage control instruction triggered on a target interface of a target device; Correspondingly, the adopting of a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively includes: The linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

3. The method according to claim 2, characterized in that If the linkage control instruction is an interface control instruction for the target interface, sending the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service includes: Sending the interface control instruction to other devices through the message forwarding service to instruct the other devices to trigger the same interface control instruction; the other devices include devices other than the target device in the first device and the second device; Through the message forwarding service, the interface control instruction triggered by the target device is sent to the backend process of the target device, and the interface control instruction triggered by the other device is sent to the backend process of the other device.

4. The method according to claim 2, characterized in that: If the linkage control instruction is a data processing instruction for the target interface, sending the linkage control instruction to the backend process of the first device and the backend process of the second device respectively through the message forwarding service includes: The data processing instruction is sent to the backend process of the first device and the backend process of the second device respectively through the message forwarding service.

5. The method according to claim 2, characterized in that: The first device is also deployed with a system control program; the target device also includes a third device; and the receiving of the linkage control instruction triggered on the target device includes: Receiving, through the system control program, a linkage control instruction triggered on the third device; Correspondingly, the adopting of a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively includes: The linkage control instruction is sent to the backend process of the first device and the backend process of the second device respectively through the system control program.

6. The method according to any one of claims 2 to 5, characterized in that The step of performing linkage control on the first device and the second device based on the linkage control instruction, the backend process of the first device, and the backend process of the second device includes: Processing the linkage control instruction through a backend process of the first device to generate a processing result of the first device; Processing the linkage control instruction through a backend process of the second device to generate a processing result of the second device; The first device and the second device are controlled in linkage according to the processing result of the first device and the processing result of the second device.

7. The method according to claim 6, characterized in that The linking control of the first device and the second device according to the processing result of the first device and the processing result of the second device includes: Through the message forwarding service, the processing result of the first device is sent to the front-end process of the first device, and the processing result of the second device is sent to the front-end process of the second device, so as to instruct the front-end process of the first device and the front-end process of the second device to execute the linkage control instruction.

8. A linkage control device, characterized in that: Applied to a first device, in which a backend process of the first device and a backend process of a second device linked with the first device are deployed, the apparatus comprises: A receiving module, configured to receive a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device; A sending module, used to use a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively; A control module is used to perform linkage control on the first device and the second device based on the linkage control instruction, the backend process of the first device and the backend process of the second device.

9. A linkage control system, characterized in that: It includes a first device and a second device linked with the first device; wherein the backend process of the first device and the backend process of the second device are deployed in the first device; The first device is used to receive a linkage control instruction triggered on a target device; the target device includes any one of the first device and the second device; The first device is also used to adopt a message forwarding mechanism to send the linkage control instruction to the backend process of the first device and the backend process of the second device respectively; and based on the linkage control instruction, the backend process of the first device and the backend process of the second device, the first device and the second device are linkage controlled.

10. The system according to claim 9, characterized in that The first device includes a PC-side device, and the second device includes an AR glasses-side device.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images