Data synchronization method, collaboration system, head-mounted display device, medium and product

Abstract

The embodiment of the invention discloses a data synchronization method, a collaboration system, a head-mounted display device, a medium and a product. According to one specific embodiment, the method comprises the steps that in response to detection of an entry operation for a target cooperation scene, a space initialization request is sent to a server side included in the cooperation system, and the space initialization request is sent in the form of an information message; receiving space resource information corresponding to the target cooperation scene sent by the server; the space resource information is loaded in the head-mounted display device; and in response to the detected interaction operation acting on the virtual element in the displayed target collaborative scene, synchronizing an operation result corresponding to the interaction operation to each head-mounted display device included in the collaborative system through the server. According to the embodiment, the customization difficulty during state synchronization processing can be reduced, resource occupation can be reduced, and the method and the device can be applied to synchronization scenes with weak real-time performance and interactivity.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a data synchronization method, a collaborative system, a head-mounted display device, a medium, and a product. Background Art

[0002] Current mixed reality and augmented reality technologies usually support virtual object interaction and operation in single-user scenarios, but in multi-user collaborative scenarios, it is difficult to handle state synchronization. The usual approach is to introduce a game synchronization framework (such as Photon, Mirror, Unity Netcode for GameObjects, etc.).

[0003] However, when the above method is adopted, the following technical problems often occur: the method of introducing the game synchronization framework is difficult to customize and has high resource consumption. It is more suitable for application scenarios with high real-time requirements, delay sensitivity and rapid development, but not suitable for application in some synchronization scenarios with weak real-time and interactivity.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a data synchronization method, a collaborative system, a head-mounted display device, a computer-readable medium, and a computer program product in a collaborative scenario to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a data synchronization method in a collaborative scenario, which is applied to a head-mounted display device included in a collaborative system, and the method includes: in response to detecting an entry operation for a target collaborative scene, sending a space initialization request to a server included in the above-mentioned collaborative system, wherein the above-mentioned space initialization request is sent in the form of a message message; receiving space resource information corresponding to the above-mentioned target collaborative scene sent by the above-mentioned server; loading the above-mentioned space resource information in the above-mentioned head-mounted display device; in response to detecting an interactive operation acting on a virtual element in the above-mentioned target collaborative scene displayed, synchronizing the operation results corresponding to the above-mentioned interactive operation to each head-mounted display device included in the above-mentioned collaborative system through the above-mentioned server.

[0008] Optionally, the method also includes: periodically sending the heartbeat request of the head-mounted display device corresponding to the target collaborative scene to the server to obtain the heartbeat status information sent by the server, wherein the heartbeat status information is stored in a heartbeat status information queue maintained by the server, the heartbeat status information queue corresponds to the head-mounted display device and the target collaborative scene, and the heartbeat request is sent in the form of a message message.

[0009] Optionally, the method also includes: in response to detecting an exit operation corresponding to the above-mentioned target collaborative scene, sending a user exit request to the above-mentioned server, wherein the above-mentioned user exit request is sent in the form of a message message, and the above-mentioned user exit request includes the request time, the scene identifier of the above-mentioned target collaborative scene and the device identifier of the above-mentioned head-mounted display device.

[0010] Optionally, the operation result corresponding to the above-mentioned interactive operation includes the posture information corresponding to the virtual element; and the above-mentioned synchronizing the operation result corresponding to the above-mentioned interactive operation to the various head-mounted display devices included in the above-mentioned collaborative system through the above-mentioned server, including: determining the interaction time corresponding to the above-mentioned interactive operation; determining the element identifier of the virtual element corresponding to the above-mentioned interactive operation; generating a synchronization request according to the above-mentioned interaction time, the above-mentioned element identifier and the above-mentioned posture information, wherein the above-mentioned synchronization request includes the above-mentioned interaction time, the above-mentioned element identifier and the above-mentioned posture information; synchronizing the above-mentioned synchronization request to the various head-mounted display devices included in the above-mentioned collaborative system through the above-mentioned server, wherein the above-mentioned synchronization request is sent in the form of a message message.

[0011] Optionally, in response to detecting an interactive operation acting on a virtual element in the target collaborative scene displayed, synchronizing the operation result corresponding to the interactive operation to each head-mounted display device included in the collaborative system through the server, including: in response to detecting an interactive operation acting on a virtual element in the target collaborative scene displayed, determining whether the interactive operation satisfies a preset locking condition; in response to determining that the interactive operation satisfies the preset locking condition, sending a locking request to the server, wherein the locking request is sent in the form of a message message; receiving the locking result sent by the server; in response to determining that the received locking result indicates that the locking is successful, performing the interactive operation in the target collaborative scene; determining whether the virtual element corresponding to the interactive operation satisfies a preset change condition; in response to determining that the virtual element corresponding to the interactive operation satisfies the preset change condition, synchronizing the operation result corresponding to the interactive operation to each head-mounted display device included in the collaborative system through the server.

[0012] Optionally, the method also includes: in response to determining that the above-mentioned interactive operation does not meet the above-mentioned preset locking condition, performing the above-mentioned interactive operation in the above-mentioned target collaborative scenario; in response to determining that the received locking result indicates a locking failure, and the above-mentioned interactive operation is detected again, sending the locking request to the above-mentioned server again.

[0013] Optionally, the method also includes: in response to determining that the above-mentioned interactive operation is completed, determining whether the above-mentioned virtual element meets the preset unlocking condition; in response to determining that the above-mentioned virtual element meets the above-mentioned preset unlocking condition, sending an unlocking request to the above-mentioned server, wherein the above-mentioned unlocking request is sent in the form of a message message.

[0014] Optionally, the method further includes: in response to determining that the received locking result indicates a locking failure, displaying operation failure prompt information corresponding to the above-mentioned interactive operation and the above-mentioned virtual element in the above-mentioned target collaborative scene.

[0015] In a second aspect, some embodiments of the present disclosure provide a collaborative system, comprising: at least one head-mounted display device, each head-mounted display device being configured to execute the method described in the first aspect above; a server being configured to send spatial resource information of a target collaborative scene to the head-mounted display device in response to receiving a spatial initialization request for a target collaborative scene sent by the head-mounted display device; and synchronizing operation results corresponding to the interactive operations of the head-mounted display device to each head-mounted display device.

[0016] In a third aspect, some embodiments of the present disclosure provide a head-mounted display device, comprising: a display module for imaging in front of the user's eyes; one or more processors; and a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0017] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0018] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0019] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the data synchronization method in the collaborative scene of some embodiments of the present disclosure, the customization difficulty when processing state synchronization can be reduced, the resource occupation can be reduced, and it can be suitable for application in synchronization scenes with weak real-time and interactivity. Specifically, the reason why it is not suitable for application in some synchronization scenes with weak real-time and interactivity is that the way of introducing the game synchronization framework is more difficult to customize and occupies more resources. It is more suitable for application scenes with high real-time requirements, delay sensitivity and rapid development, and is not suitable for application in some synchronization scenes with weak real-time and interactivity. Based on this, the data synchronization method in the collaborative scene of some embodiments of the present disclosure, first, in response to detecting an entry operation for the target collaborative scene, sends a space initialization request to the server included in the above-mentioned collaborative system, wherein the above-mentioned space initialization request is sent in the form of a message message. Thus, the space initialization request that can be parsed by the server can be sent to the server in the form of a message message. Then, the space resource information corresponding to the above-mentioned target collaborative scene sent by the above-mentioned server is received. Thus, after the server parses the space initialization request, the space resource information of the target collaborative scene can be obtained through the server. Then, the above-mentioned space resource information is loaded in the above-mentioned head-mounted display device. Thus, the loaded spatial resource information can be used by users to view the content in the target collaborative scene. Finally, in response to detecting the interactive operation acting on the virtual elements in the displayed target collaborative scene, the operation results corresponding to the interactive operation are synchronized to each head-mounted display device included in the collaborative system through the server. Thus, after the interaction occurs on the head-mounted display device, the operation results corresponding to the interactive operation can be synchronized to other head-mounted display devices through the server, so that lightweight data synchronization in the target collaborative scene can be achieved through simple message sending between the server. Furthermore, there is no need to introduce a game synchronization framework, which reduces the customization difficulty when processing state synchronization and reduces resource usage, and can be suitable for application in synchronization scenarios with weak real-time and interactivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0021] Figure 1 is an architectural diagram of an exemplary system in which some embodiments of the present disclosure may be applied;

[0022] Figure 2 is a flow chart of some embodiments of a data synchronization method in a collaborative scenario according to the present disclosure;

[0023] Figure 3 are flowcharts of other embodiments of the data synchronization method in a collaborative scenario according to the present disclosure;

[0024] Figure 4 is a schematic diagram of an application scenario of a data synchronization method in a collaborative scenario according to some embodiments of the present disclosure;

[0025] Figure 5 is a schematic diagram of the structure of some embodiments of the collaborative system according to the present disclosure;

[0026] Figure 6 It is a schematic diagram of the structure of a head-mounted display device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] Figure 1 An exemplary system architecture 100 is shown in which a data synchronization method in a collaborative scenario to which some embodiments of the present disclosure can be applied.

[0033] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, collaborative operation applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0035] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various head-mounted display devices with display screens and supporting imaging in front of the user's eyes, including but not limited to AR glasses, VR glasses, MR glasses, etc. When terminal devices 101, 102, 103 are software, they can be installed in the head-mounted display devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0036] The server 105 may be a server that provides various services, such as a background server that provides support for information displayed on the terminal devices 101, 102, and 103. The background server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0037] It should be noted that the data synchronization method in a collaborative scenario provided by the embodiments of the present disclosure can be executed by the terminal devices 101 , 102 , and 103 .

[0038] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules for providing distributed services, or it can be implemented as a single software or software module. No specific limitation is made here.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Continue to refer Figure 2, shows a process 200 of some embodiments of a data synchronization method in a collaborative scenario according to the present disclosure. The data synchronization method in the collaborative scenario includes the following steps:

[0041] Step 201, in response to detecting an entry operation to a target collaborative scene, sending a space initialization request to a server included in the collaborative system.

[0042] In some embodiments, the execution subject of the data synchronization method in the collaborative scenario (for example Figure 1 The terminal device shown in the figure can respond to the detection of the entry operation for the target collaborative scene and send a space initialization request to the server included in the above collaborative system. Among them, the above target collaborative scene can be a three-dimensional space scene for multi-user collaborative interaction. The above target collaborative scene can be a synchronous scene with weak real-time and interactivity, rather than a synchronous scene with strong real-time and interactivity such as online games. For example, the above target collaborative scene can be an AR museum exhibition scene with multi-person collaborative participation. The above entry operation can be an operation for entering the above target collaborative scene. For example, the interaction method of the above entry operation can include but is not limited to at least one of the following: touch interaction, button interaction, voice interaction, gesture interaction, head control interaction, and eye movement interaction. The above space initialization request can be a request for synchronizing the latest spatial resource information of the target collaborative scene. When a user enters the target collaborative scene for the first time or enters the target collaborative scene midway, the latest spatial resource information can be obtained by sending a space initialization request to the server. The latest spatial resource information can represent the latest scene space state. The collaborative system can include a server and each head-mounted display device. In practice, the above execution subject can send a space initialization request to the server included in the above collaborative system in the form of a message message. The communication protocol between the head mounted display device and the server may be the MQTT protocol.

[0043] Step 202: Receive spatial resource information corresponding to the target collaborative scene sent by the server.

[0044] In some embodiments, the execution subject may receive the spatial resource information corresponding to the target collaborative scene sent by the server. The spatial resource information may be various resources required to load the target collaborative scene. The spatial resource information may include but is not limited to at least one of the following: posture information and data status corresponding to the virtual element. The virtual element may include but is not limited to at least one of the following: video, audio, virtual object. For example, the spatial resource information may include click-to-play and progress control of video and audio, movement and event triggering of virtual objects. The user may complete the interaction as needed, and these interactions may cause the state of the above data to change. It is understandable that when the user enters the target collaborative scene for the first time, the spatial resource information may include complete data for loading each virtual element, and when the user enters the target collaborative scene again, the spatial resource information may include state change data of each virtual element with state change. In practice, the server may obtain the spatial resource information corresponding to the target collaborative scene from a database. For example, the database may be MongoDB.

[0045] Step 203: Loading spatial resource information into the head mounted display device.

[0046] In some embodiments, the execution subject may load the spatial resource information in the head mounted display device. In practice, when the user enters the target collaborative scene for the first time, the execution subject may directly load the spatial resource information to display the target collaborative scene. When the user enters the target collaborative scene again, the execution subject may use the spatial resource information to update the target collaborative scene.

[0047] Step 204 , in response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, synchronize the operation result corresponding to the interactive operation to each head mounted display device included in the collaborative system through the server.

[0048] In some embodiments, the execution subject may, in response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, synchronize the operation result corresponding to the interactive operation to each head-mounted display device included in the collaborative system through the server. The interactive mode of the interactive operation may include but is not limited to at least one of the following: touch interaction, button interaction, voice interaction, gesture interaction, head control interaction, and eye movement interaction. The operation result may be the result of the change in the state of the virtual element triggered after the execution of the interactive operation. For example, when the interactive operation is to move the virtual element from an original position to another position, the operation result may include the identifier of the virtual element and the position information of the virtual element after the move. In practice, the execution subject may send the operation result to the server, and after receiving the operation result, the server may synchronize the operation result to each other head-mounted display device included in the collaborative system.

[0049] Optionally, the operation result corresponding to the above-mentioned interactive operation includes the posture information corresponding to the virtual element. The above-mentioned posture information may include but is not limited to at least one of the following: position coordinates, rotation angle coordinates, and scaling coordinates. The position coordinates may be three-dimensional coordinates. The rotation angles include the rotation angles corresponding to the three axes respectively. The scaling coordinates may correspond to the scaling ratios of the three axes respectively.

[0050] In some optional implementations of some embodiments, the execution subject may synchronize the operation results corresponding to the interactive operation to each head mounted display device included in the collaborative system through the server through the following steps:

[0051] The first step is to determine the interaction time corresponding to the above interaction operations.

[0052] The second step is to determine the element identifier of the virtual element corresponding to the above interaction operation. The element identifier can be used to uniquely identify the virtual element.

[0053] The third step is to generate a synchronization request based on the above interaction time, the above element identifier and the above posture information. The above synchronization request includes the above interaction time, the above element identifier and the above posture information. In practice, the above execution entity can fill the interaction time, the above element identifier and the above posture information into the preset message format for synchronization to obtain a synchronization request.

[0054] The fourth step is to synchronize the synchronization request to each head-mounted display device included in the collaborative system through the server. The synchronization request is sent in the form of a message. In practice, the execution entity can send the synchronization request to the server, so that the server synchronizes the posture information to each head-mounted display device included in the collaborative system. Thus, the posture information generated by the interactive operation can be synchronized to other head-mounted display devices by generating a synchronization request.

[0055] Optionally, the execution subject may periodically send the heartbeat request of the head mounted display device corresponding to the target collaborative scene to the server to obtain the heartbeat status information sent by the server. The heartbeat request may be a request for obtaining the heartbeat status information. The heartbeat request may include but is not limited to: message identifier, device identifier, scene identifier, message type. Here, the message type may be a heartbeat request type. The heartbeat status information sent by the server may include but is not limited to: message identifier, device identifier, scene identifier, message type, and timestamp. Here, the message type may be a heartbeat result. The heartbeat status information may represent the online status of the user in the target collaborative scene. After the user enters the target collaborative scene, the heartbeat status information may represent online. After the user exits the target collaborative scene, the heartbeat status information may represent offline. The heartbeat status information of the head mounted display device may be stored in a heartbeat status information queue maintained by the server. The heartbeat status information queue may be stored in a distributed database. For example, the distributed database may be redis. The distributed database may also store the device identifiers of each head mounted display device included in the collaborative system. For example, a distributed database may maintain a device identifier queue. The above-mentioned heartbeat status information queue corresponds to the above-mentioned head-mounted display device and the above-mentioned target collaborative scene. The above-mentioned heartbeat request is sent in the form of a message message. In practice, the above-mentioned execution entity can regularly send the heartbeat request to the above-mentioned server at every preset time interval. For example, the above-mentioned preset time length can be 5 seconds. Thus, heartbeat requests can be sent to the server regularly, so that the server can maintain the online status of each head-mounted display device in the target collaborative scene as a whole.

[0056] Optionally, the execution subject may send a user exit request to the server in response to detecting an exit operation corresponding to the target collaborative scene. The exit operation may be an operation of exiting the target collaborative scene. The interaction mode of the exit operation may include but is not limited to at least one of the following: touch interaction, button interaction, voice interaction, gesture interaction, head control interaction, and eye movement interaction. The user exit request may be a request to characterize the exit from the target collaborative scene. The user exit request may be sent in the form of a message message. The user exit request may include but is not limited to: request time, scene identifier of the target collaborative scene, and device identifier of the head-mounted display device. The user exit request may also include a message identifier and a message type. Here, the message type may be a user exit request type. The server may send a user exit result to the execution subject. The user exit result may include but is not limited to: exit time, scene identifier, device identifier, and message type. Here, the message type may be a user exit result type. Thus, the user exit event can be processed by the server, so that the server can maintain the exit of each head-mounted display device in the target collaborative scene as a whole, and synchronize it to other head-mounted display devices in a timely manner.

[0057] Optionally, the above execution entity may further perform the following steps:

[0058] The first step is to parse the synchronization operation result in response to receiving the synchronization operation result sent by the server to obtain the synchronization operation information. The synchronization operation result may be the operation result of the virtual element synchronized to the server by other head-mounted display devices. The synchronization operation information may include but is not limited to: operation time, element identification, interaction type and posture information. The operation time may be the trigger time for the user corresponding to other head-mounted display devices to perform an interactive operation on the virtual element. The interaction type may be a continuous operation, a one-time operation or an instantaneous operation. The posture information may include position coordinates, rotation angle coordinates, and scaling coordinates. In practice, the execution subject may parse the synchronization operation result by field parsing to obtain the synchronization operation information.

[0059] In the second step, in response to determining that the above interaction type meets the preset continuity condition, the current time is determined as the receiving time. The preset continuity condition may be that the interaction type belongs to a continuous operation.

[0060] The third step is to determine the difference between the receiving time and the operating time as the synchronization time difference.

[0061] The fourth step is to determine the original posture information corresponding to the synchronization operation result in response to determining that the synchronization time difference satisfies the preset time interval condition. The original posture information includes the original position coordinates, the original rotation angle coordinates, and the original scaling coordinates. The original posture information corresponding to the synchronization operation result may be the posture information of the corresponding virtual element before the interactive operation corresponding to the synchronization operation result.

[0062] The fifth step is to generate an interpolated position coordinate sequence corresponding to a preset number of frames based on the position coordinates included in the above-mentioned posture information and the original position coordinates included in the above-mentioned original posture information. In practice, the above-mentioned execution subject can use an interpolation algorithm to interpolate the position coordinates included in the above-mentioned posture information and the original position coordinates included in the above-mentioned original posture information to obtain an interpolated position coordinate sequence corresponding to the preset number of frames. The number of each interpolated position coordinate included in the interpolated position coordinate sequence is equal to the above-mentioned preset number of frames. The interpolation algorithm can be linear interpolation or spline interpolation.

[0063] The sixth step is to generate an interpolated rotation angle coordinate sequence corresponding to a preset number of frames according to the rotation angle coordinates included in the above-mentioned posture information and the original rotation angle coordinates included in the above-mentioned original posture information. In practice, the above-mentioned execution subject can use an interpolation algorithm to interpolate the rotation angle coordinates included in the above-mentioned posture information and the original rotation angle coordinates included in the above-mentioned original posture information to obtain an interpolated rotation angle coordinate sequence corresponding to the preset number of frames. The number of each interpolated rotation angle coordinate included in the interpolated rotation angle coordinate sequence is equal to the above-mentioned preset number of frames. The interpolation algorithm can be linear interpolation or spline interpolation.

[0064] The seventh step is to generate an interpolated scaling coordinate sequence corresponding to a preset number of frames based on the scaling coordinates included in the above-mentioned posture information and the original scaling coordinates included in the above-mentioned original posture information. In practice, the above-mentioned execution entity can use an interpolation algorithm to interpolate the scaling coordinates included in the above-mentioned posture information and the original scaling coordinates included in the above-mentioned original posture information, corresponding to an interpolated scaling coordinate sequence of a preset number of frames. The number of each interpolated scaling coordinate included in the interpolated scaling coordinate sequence is equal to the above-mentioned preset number of frames. The interpolation algorithm can be linear interpolation or spline interpolation.

[0065] In the eighth step, the coordinates corresponding to the same frame in the interpolation position coordinate sequence, the interpolation rotation angle coordinate sequence and the interpolation scaling ratio coordinate sequence are combined to obtain an interpolation pose information sequence. Each interpolation pose information in the interpolation pose information sequence includes the interpolation position coordinates, the interpolation rotation angle coordinates and the interpolation scaling ratio coordinates.

[0066] The ninth step is to perform a posture update process on the virtual element corresponding to the synchronization operation result according to the interpolation posture information sequence. In practice, the execution subject may perform a posture update process on the virtual element according to each interpolation posture information in the interpolation posture information sequence.

[0067] The above-mentioned first step to the ninth step, as an inventive point of an embodiment of the present disclosure, solves the technical problem that "when the time difference between the operation time and the time when the operation is synchronized to other terminals is large, the screen of other terminals will flicker, the smoothness of the change of virtual elements is poor, resulting in poor user experience". The factors that lead to poor user experience are often as follows: when the time difference between the operation time and the time when the operation is synchronized to other terminals is large, the screen of other terminals will flicker, and the smoothness of the change of virtual elements is poor. If the above factors are solved, the effect of improving user experience can be achieved. In order to achieve this effect, after receiving the operation results synchronized by other head-mounted display devices, the present disclosure parses the operation results to obtain the required information, and after determining that the interaction type and the synchronization time difference meet the conditions, the parsed posture information and the original posture information are interpolated for each dimension data, so that the multi-frame dimensional data obtained after interpolation can be used for smooth update, thereby achieving smooth processing of virtual element changes and improving user experience.

[0068] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the data synchronization method in the collaborative scene of some embodiments of the present disclosure, the customization difficulty when processing state synchronization can be reduced, the resource occupation can be reduced, and it can be suitable for application in synchronization scenes with weak real-time and interactivity. Specifically, the reason why it is not suitable for application in some synchronization scenes with weak real-time and interactivity is that the way of introducing the game synchronization framework is more difficult to customize and occupies more resources. It is more suitable for application scenes with high real-time requirements, delay sensitivity and rapid development, and is not suitable for application in some synchronization scenes with weak real-time and interactivity. Based on this, the data synchronization method in the collaborative scene of some embodiments of the present disclosure, first, in response to detecting an entry operation for the target collaborative scene, sends a space initialization request to the server included in the above-mentioned collaborative system, wherein the above-mentioned space initialization request is sent in the form of a message message. Thus, the space initialization request that can be parsed by the server can be sent to the server in the form of a message message. Then, the space resource information corresponding to the above-mentioned target collaborative scene sent by the above-mentioned server is received. Thus, after the server parses the space initialization request, the space resource information of the target collaborative scene can be obtained through the server. Then, the above-mentioned space resource information is loaded in the above-mentioned head-mounted display device. Thus, the loaded spatial resource information can be used by users to view the content in the target collaborative scene. Finally, in response to detecting the interactive operation acting on the virtual elements in the displayed target collaborative scene, the operation results corresponding to the interactive operation are synchronized to each head-mounted display device included in the collaborative system through the server. Thus, after the interaction occurs on the head-mounted display device, the operation results corresponding to the interactive operation can be synchronized to other head-mounted display devices through the server, so that lightweight data synchronization in the target collaborative scene can be achieved through simple message sending between the server. Furthermore, there is no need to introduce a game synchronization framework, which reduces the customization difficulty when processing state synchronization and reduces resource usage, and can be suitable for application in synchronization scenarios with weak real-time and interactivity.

[0069] Further references Figure 3 , which shows a process 300 of another embodiment of a data synchronization method in a collaborative scenario. The process 300 of the data synchronization method in a collaborative scenario includes the following steps:

[0070] Step 301, in response to detecting an entry operation to a target collaborative scene, sending a space initialization request to a server included in the collaborative system.

[0071] Step 302: Receive spatial resource information corresponding to the target collaborative scene sent by the server.

[0072] Step 303: Loading spatial resource information into the head mounted display device.

[0073] In some embodiments, the specific implementation of steps 301-303 and the technical effects brought about can be referred to Figure 2 The corresponding steps 201-203 in the embodiments are not described in detail here.

[0074] Step 304 , in response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, determining whether the interactive operation satisfies a preset locking condition.

[0075] In some embodiments, the execution subject of the data synchronization method in a collaborative scene (such as a head-mounted display device) can determine whether the interactive operation meets the preset locking condition in response to detecting the interactive operation acting on the virtual elements in the displayed target collaborative scene. Among them, the preset locking condition may be that the interaction type corresponding to the interactive operation belongs to a continuous operation. The continuous operation may be a series of operations that the user needs to complete continuously. For example, dragging the video progress bar and moving objects are both continuous operations. In contrast, turning on the video switch and muting the audio are both one-time or instantaneous operations and do not require locking.

[0076] Step 305: In response to determining that the interactive operation meets the preset locking condition, a locking request is sent to the server.

[0077] In some embodiments, in response to determining that the above-mentioned interactive operation satisfies the above-mentioned preset locking condition, a locking request is sent to the above-mentioned server. The above-mentioned locking request is sent in the form of a message message. The above-mentioned locking request may be a request to lock the virtual element to shield the operation of the virtual element by other head-mounted display devices. The above-mentioned locking request may include but is not limited to: device identification, element identification, request time, action type, message type. Here, the action type may be locking. The message type may be a locking request type. Thus, the server may be requested to lock the virtual element by sending a message message to shield the operation of the virtual element by other head-mounted display devices, thereby avoiding multi-terminal operation conflicts.

[0078] Step 306, receiving the locking result sent by the server.

[0079] In some embodiments, the execution entity may receive the locking result sent by the server. The server may send the locking result to each head mounted display device in the collaborative system to synchronize the locking result. The locking result may include but is not limited to: message type, device identifier, scene identifier, element identifier, and result. Here, the message type may be the locking result type. The device identifier may be the identifier of the head mounted display device requesting locking. The result may represent whether the locking is successful or failed.

[0080] Step 307 : In response to determining that the received locking result indicates that the locking is successful, an interactive operation is performed in the target collaborative scenario.

[0081] In some embodiments, the execution subject may perform the interaction operation in the target collaborative scene in response to determining that the received locking result indicates that the locking is successful. Thus, after shielding the operations of other head mounted display devices on the successfully locked virtual element, the interaction may be performed on the virtual element alone.

[0082] Step 308: Determine whether the virtual element corresponding to the interactive operation satisfies a preset change condition.

[0083] In some embodiments, the execution subject may determine whether the virtual element corresponding to the interactive operation satisfies a preset change condition. The preset change condition may be that the state of the virtual element changes. Here, the state change may include but is not limited to at least one of the following changes: position change, rotation angle change, scaling change, style change, and element configuration information change.

[0084] Step 309 , in response to determining that the virtual element corresponding to the interactive operation satisfies the preset change condition, the operation result corresponding to the interactive operation is synchronized to each head mounted display device included in the collaborative system through the server.

[0085] In some embodiments, the execution subject may synchronize the operation result corresponding to the interactive operation to each head-mounted display device included in the collaborative system through the server in response to determining that the virtual element corresponding to the interactive operation satisfies the preset change condition. Thus, when the state of the virtual element changes, the change result may be synchronized to other head-mounted display devices.

[0086] As an example, refer to Figure 4 In a mixed reality scenario, user A and user B can operate different virtual objects at the same time, and user A, user B, and user C can see the entire process of the object's movement at the same time, but the operation applied by user C to the object being operated will be rejected until the operation of user A and user B is terminated.

[0087] Optionally, the execution subject may perform the interactive operation in the target collaborative scenario in response to determining that the interactive operation does not satisfy the preset locking condition. Thus, when the interactive operation does not require locking, the interactive operation can be directly performed locally. Secondly, in response to determining that the received locking result indicates a locking failure, and detecting the interactive operation again, the locking request is sent to the server again. Thus, an attempt can be made to lock the virtual element again after the locking fails.

[0088] Optionally, the execution subject may determine whether the virtual element satisfies a preset unlocking condition in response to determining that the execution of the interactive operation ends. The preset unlocking condition may be that the virtual element is in a locked state. Then, in response to determining that the virtual element satisfies the preset unlocking condition, an unlocking request may be sent to the server. The unlocking request is sent in the form of a message. The unlocking request may include but is not limited to: device identification, element identification, request time, action type, and message type. Here, the action type may be unlocking. The message type may be an unlocking request type. Correspondingly, the server may send an unlocking result to each head-mounted display device in the collaborative system to synchronize the unlocking results. The unlocking result may include but is not limited to: message type, device identification, scene identification, element identification, and result. Here, the message type may be an unlocking result type. The device identification may be an identification of the head-mounted display device requesting unlocking. The result may represent a success or failure of unlocking. Thus, the server may be requested to unlock the virtual element by sending a message to release the lock of the current head-mounted display device on the virtual element.

[0089] Optionally, in response to determining that the received locking result represents a locking failure, operation failure prompt information corresponding to the above-mentioned interactive operation and the above-mentioned virtual element is displayed in the above-mentioned target collaborative scene. The above-mentioned operation failure prompt information can be prompt information in text form, or it can be a contour prompt around a visual virtual element. For example, the operation failure prompt information can be a flashing red contour around the virtual element. There is no limitation on the specific display form of the operation failure prompt information. Thus, when locking fails, the user can be prompted that the operation failed.

[0090] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 The process 300 of the data synchronization method in the collaborative scenario in some corresponding embodiments reflects the extended steps of locking the virtual element. Therefore, the schemes described in these embodiments can request the server to lock the virtual element by sending a message message to shield other head-mounted display devices from operating the virtual element, thereby avoiding multi-terminal operation conflicts.

[0091] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a collaborative system, which includes at least one head-mounted display device 501 and a server 502.

[0092] Each head mounted display device may be configured to perform Figure 1-4The server 502 may be configured to send the spatial resource information of the target collaborative scene to the head-mounted display device in response to receiving a spatial initialization request for the target collaborative scene sent by the head-mounted display device. Then, the operation results corresponding to the interactive operations of the head-mounted display device may be synchronized to each head-mounted display device. Through the collaborative system of some embodiments of the present disclosure, there is no need to introduce a game synchronization framework, which reduces the customization difficulty when processing state synchronization, reduces resource usage, and can be suitable for application in synchronization scenarios with weak real-time and interactivity.

[0093] Reference below Figure 6 , which shows a head mounted display device 600 (eg, Figure 1 Schematic diagram of the structure of the terminal device in the system. Figure 6 The head mounted display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0094] like Figure 6 As shown, the head mounted display device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the head mounted display device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0095] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; and a communication device 609. The communication device 609 may allow the head mounted display device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The head mounted display device 600 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0096] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0097] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0098] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0099] The computer-readable medium may be included in the head-mounted display device; or it may exist independently without being assembled into the head-mounted display device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the head-mounted display device, the head-mounted display device: in response to detecting an entry operation to a target collaborative scene, sends a space initialization request to a server included in the collaborative system, wherein the space initialization request is sent in the form of a message; receives the space resource information corresponding to the target collaborative scene sent by the server; loads the space resource information in the head-mounted display device; in response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, synchronizes the operation result corresponding to the interactive operation to each head-mounted display device included in the collaborative system through the server.

[0100] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0103] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements the data synchronization method in any of the above-mentioned collaborative scenarios when executed by a processor.

[0104] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A data synchronization method in a collaborative scenario, applied to a head mounted display device included in a collaborative system, comprising: In response to detecting an entry operation for a target collaborative scene, sending a space initialization request to a server included in the collaborative system, wherein the space initialization request is sent in the form of a message; Receiving spatial resource information corresponding to the target collaborative scene sent by the server; Loading the spatial resource information in the head mounted display device; In response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, an operation result corresponding to the interactive operation is synchronized to each head-mounted display device included in the collaborative system through the server.

2. The method according to claim 1, wherein: The method further comprises: The heartbeat request of the head-mounted display device corresponding to the target collaborative scene is periodically sent to the server to obtain heartbeat status information sent by the server, wherein the heartbeat status information is stored in a heartbeat status information queue maintained by the server, and the heartbeat status information queue corresponds to the head-mounted display device and the target collaborative scene, and the heartbeat request is sent in the form of a message.

3. The method according to claim 1, wherein: The method further comprises: In response to detecting an exit operation corresponding to the target collaborative scene, a user exit request is sent to the server, wherein the user exit request is sent in the form of a message message, and the user exit request includes the request time, the scene identifier of the target collaborative scene, and the device identifier of the head-mounted display device.

4. The method according to claim 1, wherein: The operation result corresponding to the interactive operation includes the posture information corresponding to the virtual element; And synchronizing the operation result corresponding to the interactive operation to each head mounted display device included in the collaborative system through the server, including: Determining an interaction time corresponding to the interaction operation; Determining an element identifier of a virtual element corresponding to the interactive operation; Generate a synchronization request according to the interaction time, the element identifier and the posture information, wherein the synchronization request includes the interaction time, the element identifier and the posture information; The synchronization request is synchronized to each head mounted display device included in the collaborative system through the server, wherein the synchronization request is sent in the form of a message message.

5. The method according to claim 1, wherein: In response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, synchronizing an operation result corresponding to the interactive operation to each head mounted display device included in the collaborative system through the server, including: In response to detecting an interactive operation acting on a virtual element in the displayed target collaborative scene, determining whether the interactive operation satisfies a preset locking condition; In response to determining that the interactive operation satisfies the preset locking condition, sending a locking request to the server, wherein the locking request is sent in the form of a message; Receiving the locking result sent by the server; In response to determining that the received locking result indicates that the locking is successful, performing the interactive operation in the target collaborative scenario; Determining whether the virtual element corresponding to the interactive operation meets a preset change condition; In response to determining that the virtual element corresponding to the interactive operation satisfies the preset change condition, the operation result corresponding to the interactive operation is synchronized to each head-mounted display device included in the collaborative system through the server.

6. The method according to claim 5, wherein: The method further comprises: In response to determining that the interactive operation does not satisfy the preset locking condition, performing the interactive operation in the target collaborative scenario; In response to determining that the received locking result indicates a locking failure and detecting the interactive operation again, the locking request is sent to the server again.

7. The method according to claim 5, wherein: The method further comprises: In response to determining that the interactive operation is finished, determining whether the virtual element satisfies a preset unlocking condition; In response to determining that the virtual element satisfies the preset unlocking condition, an unlocking request is sent to the server, wherein the unlocking request is sent in the form of a message.

8. The method according to claim 5, wherein: The method further comprises: In response to determining that the received locking result indicates a locking failure, operation failure prompt information corresponding to the interactive operation and the virtual element is displayed in the target collaborative scene.

9. A collaborative system, comprising: At least one head mounted display device, each head mounted display device being configured to perform the method according to any one of claims 1 to 8; The server is configured to send the spatial resource information of the target collaborative scene to the head mounted display device in response to receiving the spatial initialization request of the target collaborative scene sent by the head mounted display device; The operation results corresponding to the interactive operations of the head-mounted display device are synchronized to each head-mounted display device.

10. A head mounted display device, comprising: A display module, used to form an image in front of the user's eyes; one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images