Message transmission processing method, system, device and medium applied to interactive scenario

Abstract

The application discloses a message transmission processing method, system and device applied to an interactive scene, and a medium. The method applied to a server comprises the following steps: identifying a first transmission protocol when a first data packet of a sending end is received; parsing first message content from the first data packet according to a parsing mode corresponding to the first transmission protocol; determining a service processing thread according to a resource type of the first message content, and routing the first message content to the service processing thread, so as to obtain transmission content through the service processing thread; determining a second transmission protocol based on the transmission content, processing second message content according to a processing mode of the second transmission protocol, assembling a second data packet, and sending the second data packet to a receiving end. The application effectively improves the timeliness of message transmission of different data types in a meta universe interactive scene.

Description

Technical Field

[0001] This invention relates to the field of computer and network information processing technology, and specifically to a message transmission processing method, system, device, and medium applied to interactive scenarios. Background Technology

[0002] Metaverse is an open and shared online platform that integrates information technology, communication technology, AR, VR, and other virtual technologies. It is a vast and evolving virtual universe, and more and more people will enter and live within it. Metaverse has diverse applications, such as games, social networking, and virtual offices. One common factor influencing people's experience across these various Metaverse applications is the interactive experience within the scene, and the timeliness of the data generated during interaction is a crucial factor. For example, user A, through an avatar, is in a scene within the Metaverse. When A performs an action towards several other user avatars in the same scene, this action data should be transmitted to the other avatars promptly and accurately, allowing them to see the action of user A's avatar from their respective locations. Similarly, user A should be able to receive timely feedback from the other user avatars regarding this action. However, because interactions generate a large amount of heterogeneous data, such as video, audio, text, and interaction states, not only is the data volume large, but the data structures and types are also different. Therefore, conventional data transmission methods struggle to guarantee the timeliness of data transmission, thus affecting the user experience within the Metaverse. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a message transmission processing method, system, terminal device, server, computer-readable storage medium, and computer program product applicable to interactive scenarios, at least to solve the timeliness problem of message transmission in metaverse interactive scenarios.

[0004] This invention provides a message transmission processing method for interactive scenarios, applied to a server, for transmitting interactive messages between a sender and a receiver. The method includes:

[0005] Upon receiving the first data packet from the sender, the first transmission protocol used to send the first data packet is identified;

[0006] The first message content sent by the sender is parsed from the first data packet according to the parsing method corresponding to the first transmission protocol;

[0007] The corresponding business processing thread is determined based on the resource type of the first message content, and the first message content is routed to the business processing thread. The business processing thread processes the first message content to obtain the transmission content, which includes at least the second message content to be sent to the receiving end.

[0008] Based on the transmission content, a second transmission protocol is determined, and the second message content is processed according to the processing method of the second transmission protocol; and

[0009] Assemble a second data packet, wherein the second data packet includes at least the processed second message content and the second transmission protocol identifier, and send the second data packet to the receiving end;

[0010] The processing methods provided by the first and second transmission protocols correspond to the data types of the scene and the interactive message content, respectively.

[0011] Optionally, the first / second data packet includes a protocol identifier field, and the first transmission protocol is identified based on the content of the protocol identifier field in the first data packet; when assembling into the second data packet, the second transmission protocol identifier is added to the protocol identifier field.

[0012] Optionally, each resource type corresponds to a business processing thread.

[0013] Optionally, the step of determining the corresponding business processing thread based on the content of the first message includes:

[0014] The parsed content of the first message and the user data of the sending end are queried to obtain a route identifier, wherein the route identifier corresponds to one or more resource type identifiers;

[0015] The route hash value is obtained by hashing the route identifier;

[0016] The first value is obtained by performing a modulo operation based on the route hash value and the number of business processing threads; and

[0017] Match the business processing thread identifier that matches the first value.

[0018] Optionally, the routing identifier includes a combination of one or more identifiers such as user identifier, scenario identifier, and service identifier.

[0019] This invention provides a message transmission processing method for interactive scenarios, applied to a client in a user terminal. When the client acts as the sender, the method includes:

[0020] The first transmission protocol is determined based on the data type and scenario of the first message content sent by the user;

[0021] The first message content is processed according to the processing method of the first transmission protocol; and

[0022] The processed first message content and the first transmission protocol identifier are assembled into a first data packet and sent to the server;

[0023] When the client acts as the receiving end, the method includes:

[0024] Upon receiving the second data packet sent by the server, the second transport protocol is identified from it; and

[0025] The second message content sent by the server is parsed from the second data packet according to the parsing method corresponding to the second transmission protocol. The second message content is the message content after the first message content sent by the server to the sender is processed by the business.

[0026] The processing methods provided by the first and second transmission protocols correspond to the data types of the scene and the interactive message content, respectively.

[0027] This invention also provides a message transmission processing system for interactive scenarios. Applied to a server, it transmits interactive messages between a sender and a receiver. The system includes a server-side communication module, a packet parsing module, a routing module, a packet data processing module, and a packet assembly module. The server-side communication module communicates with both the sender and receiver, and is configured to receive a first data packet sent by the sender and send a second data packet to the receiver. The packet parsing module is connected to the server-side communication module and is configured to identify a first transmission protocol from the first data packet sent by the sender, and parse the first message content sent by the sender from the first data packet according to the parsing method corresponding to the first transmission protocol. The routing module is connected to the packet parsing module and is configured to... The system determines the corresponding business processing thread based on the type of message and routes the first message content to that thread. The business processing thread processes the message to obtain transmission content, which includes at least the second message content to be sent to the receiving end. The packet data processing module is configured to determine a second transmission protocol based on the transmission content and processes the second message content according to the processing method of the second transmission protocol. The packet assembly module is connected to the packet data processing module and the server communication module. It is configured to assemble a second data packet, including the processed second message content and the second transmission protocol identifier, and sends the second data packet to the receiving end via the server communication module. The processing methods provided by the first and second transmission protocols correspond to the data types of the scene and the interactive message content, respectively.

[0028] This invention provides a server, including a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the aforementioned message transmission processing method applied to interactive scenarios.

[0029] This invention provides a terminal device, including a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the aforementioned message transmission processing method applied to interactive scenarios.

[0030] This invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the aforementioned message transmission processing method applied to a server or a terminal device in an interactive scenario.

[0031] A computer program product includes computer program instructions that, when executed by a processor, implement the aforementioned message transmission processing method applied to a server or a terminal device in an interactive scenario.

[0032] This invention effectively improves the timeliness of message transmission for different data types in the metaverse interaction scenario by configuring corresponding transmission protocols for heterogeneous data when user avatars interact with each other. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below.

[0034] Figure 1 This is a schematic diagram of an AR system architecture based on a server and a terminal device according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of a virtual-real image fusion method for AR navigation using a mobile app.

[0036] Figure 3 This is a flowchart of a message transmission processing method applied to a client in an embodiment of the present invention.

[0037] Figure 4 This is a flowchart of a message transmission and processing method applied to the server side according to an embodiment of the present invention.

[0038] Figure 5 This is a schematic diagram of the overall structure and principle of the message transmission system according to an embodiment of the present invention.

[0039] Figure 6 This is a block diagram illustrating the principle structure of the routing module in an embodiment of the present invention.

[0040] Figure 7 This is a schematic diagram of the process of forwarding data from different clients to the corresponding business processing threads according to the resource type of the business processing in an embodiment of the present invention.

[0041] Figure 8 This is a schematic diagram of the server-side framework structure according to an embodiment of the present invention.

[0042] Figure 9 This is a schematic diagram of an electronic device used to implement the message transmission processing method of the embodiments of the present invention.

[0043] Figure 10 This is a schematic diagram of the software structure of an exemplary terminal device according to an embodiment of the present invention. Detailed Implementation

[0044] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of the present invention clearer and more thorough, enabling those skilled in the art to better understand and implement the principles and spirit of the present invention. The exemplary embodiments provided herein are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described herein without inventive effort are within the scope of protection of the present invention.

[0045] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, computer-readable storage medium, or computer program product. Therefore, the present invention can be specifically implemented in at least one of the following forms: entirely hardware, entirely software, or a combination of hardware and software. According to specific embodiments of the present invention, the present invention claims protection for a message transmission processing method, system, terminal device, server, computer-readable storage medium, and computer program product applied in interactive scenarios.

[0046] In this document, terms such as first, second, and third are used only to distinguish one entity (or operation) from another, and are not intended to require or imply any order or relationship between these entities (or operations).

[0047] Embodiments of the present invention can be applied to servers and terminal devices. Please refer to... Figure 1 This diagram schematically illustrates an AR system architecture based on servers and terminal devices. The AR system architecture includes a server 1000 and several terminal devices 2000. In some examples, the terminal devices 2000 are AR devices, which can be dedicated AR devices such as head-mounted displays (HMDs), smart gloves, clothing, and other smart wearable electronic devices. In other examples, the terminal devices 2000 can be general-purpose AR devices, such as mobile phones, laptops, tablets, virtual reality (VR) devices, in-vehicle devices, navigation devices, gaming devices, etc.

[0048] Taking AR helmets or AR glasses as an example, a head-mounted display, machine vision system, and mobile computer can be integrated into a wearable device. This device has a display resembling glasses and is worn on the user's head. It transmits augmented reality information to the display or projects it onto the user's eyes, enhancing the user's visual immersion. In some examples, AR devices also include cameras, which can be wide-angle, telephoto, or structured light cameras (also known as point cloud depth cameras, 3D structured light cameras, or depth cameras). Structured light cameras, based on 3D vision technology, can acquire the planar and depth information of objects. A structured light camera projects light with specific structural features onto the object being photographed using a near-infrared laser. The reflected light is then collected by an infrared camera and processed by a processor chip. The calculation principle involves calculating the object's position and depth information based on changes in the light signal caused by the object, presenting a 3D image. Typical terminal devices, such as mobile phones, display two-dimensional images and cannot show the depth of different locations within the image. Structured light cameras can capture and acquire 3D image information, obtaining not only color and other information at different locations but also depth information, which can be used for AR ranging. Of course, ordinary terminal devices can also acquire 2D images using optical cameras and combine this with deep learning algorithms to obtain depth information, ultimately displaying 3D images as well.

[0049] In some examples, the terminal device 2000 has AR-enabled software or an application (APP) installed. The server 1000 can be a management server or application server for this software or APP. The server 1000 can be a single server, a server cluster consisting of multiple servers, or a cloud server, etc. The terminal device 2000 integrates networking modules, such as a Wireless-Fidelity (Wi-Fi) module, a Bluetooth module, and a 2G / 3G / 4G / 5G communication module, to connect to the server 1000 via a network.

[0050] For example, users can log in to their user accounts through an app installed on their mobile phones, or through software installed on AR glasses.

[0051] Taking an AR navigation app as an example, the app can possess capabilities such as high-precision map navigation, environmental understanding, and virtual-real fusion rendering. The app can report its current geographical location information to the server 1000 via the terminal device 2000. The server 1000 then provides AR navigation services to the user based on the real-time geographical location information. For example, if the terminal device 2000 is a mobile phone, in response to the user launching the app, the phone can activate its camera to capture images of the real environment. Then, the system performs AR enhancement on the captured images, integrating or overlaying rendered AR effects (such as navigation route markers, road names, merchant information, and advertising displays) into the real environment images, displaying the virtual-real fusion image on the phone screen.

[0052] Figure 2 The illustration schematically shows a virtual-real fusion image for AR navigation using a mobile app, where the AR navigation pointer arrows are superimposed on the real road surface and space in the image, and the electronic promotional materials of merchants float in the space in the form of parachutes carrying gift boxes at designated locations.

[0053] Embodiments of this invention relate to terminal devices and servers. Users use the terminal device to enter a metaverse as an avatar. To address the timeliness of message transmission when multiple user avatars interact, this invention provides a message transmission processing method and system. It involves a client installed on the user's terminal device and a server installed on the server. Through the cooperation of the client and server, timely transmission of interactive messages between user avatars is achieved. These messages can carry chat messages, actions, or emoticons during interactions between user avatars, such as text, voice, video, or identifiers. The principles and spirit of this invention will be explained in detail below through several exemplary embodiments or representative implementations.

[0054] See Figure 3 , Figure 3 This is a flowchart of a message transmission processing method applied to a client according to an embodiment of the present invention. In this embodiment, to distinguish between messages sent and received by a client, the sent message is called the first message, the client that sends the message is called the sender, the received message is called the second message, and the client that receives the message is called the receiver. The message transmission processing method applied to the client includes the following steps:

[0055] Step S101: Determine the first transmission protocol for transmission based on the data type and scenario of the first message content sent by the user.

[0056] Step S102: Process the first message content according to the processing method of the first transmission protocol.

[0057] Step S103: Assemble the processed first message content and the first transmission protocol identifier into a first data packet and send it to the server.

[0058] Step S104: Monitor whether the second data packet sent by the server is received. If it is received, proceed to step S105. If not, continue monitoring.

[0059] Step S105: Identify the second transmission protocol from the second data packet.

[0060] Step S106: Parse the second message content sent by the server from the second data packet according to the parsing method corresponding to the second transmission protocol.

[0061] In this invention, the data portions of the first and second data packets include multiple fields arranged in sequence, each with a specific meaning. Table 1 shows the fields and their meanings in a data packet from one embodiment. The client and server agree to use the same data packet format to transmit interactive messages.

[0062] Table 1

[0063]

[0064] In the data packet format, the third field indicates the transmission protocol used by the current data packet. In one embodiment, this field contains a transmission protocol identifier, such as a protocol number. Different transmission protocol numbers correspond to different data types and scenarios of interactive message content. The data types include, for example, strings representing text, images, audio, video, action identifiers, and scene identifiers. The scenarios include various scenarios defined by the current application, typically determined by the application's business logic, such as small-scale chat scenarios, game battle scenarios, etc. This invention defines the data types of the transmitted interactive message content and the format used for transmitting the interactive message content, such as whether encoding is required and the encoding format used when encoding is required, thereby adapting to the efficient transmission of various heterogeneous data in 3D scenes. For example, in a small-scale chat scenario within the metaverse, user avatar A sends a voice message. Client A uses a transmission protocol (numbered 001) suitable for "small-scale" scenarios and transmitting audio data to encode the voice data of this interactive message using a low compression ratio. The protocol also defines the message type as a voice message. The processing method further includes determining packet data according to the message type and the order of the encoded chat messages before sending it to the server. The server then forwards it to other users' clients, ensuring that other avatars in the current scenario can hear the voice message promptly. However, if the current scenario is a large-scale performance or lecture, to quickly transmit the speaker's voice to other user avatars in the scenario, client A uses a transmission protocol (numbered 002) suitable for "large-scale" scenarios and transmitting audio data to encode the voice data of this interactive message using a high compression ratio before sending it to the server. The server then forwards it to other user avatars in the current scenario. Although both scenarios involve transmitting voice information, different protocols are used due to the different scenarios. For example, still taking a small-scale chat scenario, when user avatar A performs an action towards other user avatars, client A encodes the interaction message using a transmission protocol 100 suitable for transmitting action identifiers in "small-scale" scenarios and sends it to the server. After processing by the server's business system, the system obtains the action that each other user avatar should see, and then forwards it to the other users' clients, so that each other user avatar sees the action performed by user avatar A at their current location. Although the implementation of transmitting actions and the aforementioned example of transmitting voice information are both small-scale scenarios, the data types of the transmitted messages are different, and therefore different protocols are used.

[0065] In one embodiment, when a user avatar sends a text chat message to another user avatar, the client determines, in step S101, the first transmission protocol for processing text data in a "chat" scenario based on the data type and scenario of the first message content. For example, the text message transmission protocol numbered 1001. Then, in step S102, the first message content is processed according to the processing method of the first transmission protocol. This processing includes the encoding method and the content to be assembled in the data packet and the assembly order. For example, when encoding chat text data, assembling the data packet should include the message data type and the encoded text data. The steps for assembling the first data packet specifically include: first, determining the content corresponding to each field according to the data packet format, and then filling the field content into the corresponding fields. For example, enter "1001" in the "Protocol Number" field; since this message is actively sent by the user avatar, the packet type is request type, so enter "0" in the "Packet Type" field; when this message is the first message sent by the user avatar, enter "0" in the "Message ID" field to identify the sequence number of the message sent by the user avatar; when the user avatar has received 3 messages pushed by the server, enter "2" in the "Push ID" field to identify the sequence number of the last message pushed to it by the server; add the text type and the encoded content of the first message to the "Protocol Body" field; after all fields are assembled, calculate the total packet length and add it to the "Total Packet Length" field to complete the assembly of the data packet.

[0066] When the client detects that it has received data from the server, in step S105, it identifies the second transmission protocol used by the data packet from the "protocol identifier" field. After determining the transmission protocol, in step S106, it parses the second message content sent by the server from the second data packet according to the parsing method corresponding to the second transmission protocol. Then, the second message content is forwarded to the client's business processing module for corresponding processing, such as directly sending it to the interface to display text chat content, playing voice content through the audio output interface, or using the action processing module to calculate and render the action of the user avatar that sent the action, and enhancing the display of the user avatar performing the corresponding action, etc.

[0067] See Figure 4 , Figure 4 This is a flowchart of a message transmission processing method applied to the server side. The server side is used to transmit interactive messages between the sending end and the receiving end. In this embodiment, the message transmission processing method includes the following steps:

[0068] S201, upon receiving the first data packet from the sending end, the first transmission protocol used to send the first data packet is identified.

[0069] S202, the first message content sent by the sending end is parsed from the first data packet according to the parsing method corresponding to the first transmission protocol.

[0070] S203, determine the corresponding business processing thread based on the resource type of the first message content, and route the first message content to the business processing thread.

[0071] S204, Receive the transmission content processed by the service processing thread, wherein the transmission content includes at least the second message content to be sent to the receiving end.

[0072] S205, determine the second transmission protocol based on the transmission content, and process the second message content according to the processing method of the second transmission protocol.

[0073] S206, assemble the second data packet and send it to the receiving end.

[0074] When the server receives the first data packet from the sender, it identifies the transmission protocol used by the first data packet from the "protocol identifier" field. After determining the transmission protocol, in step S202, the server processes the content of the "protocol body" field in the first data packet according to the parsing method corresponding to the first transmission protocol, thereby obtaining the first message content sent by the sender. For example, when the content read from the "protocol identifier" is "001", it can be known that the content read from the "protocol body" field is the message type of the current message and the chat message after audio encoding. At this time, the parameters required for decoding the audio encoded data corresponding to the transmission protocol can be queried, and the audio encoded data is decoded according to the decoding parameters, thereby obtaining the voice content sent by the sender.

[0075] To improve server-side data processing efficiency and address latency and blocking issues caused by high message concurrency, the server-side business processing system employs a single-threaded processing approach when handling client-side interactive messages. This means that resources of the same type are processed by a single thread. The resource type can be determined by one or more factors, such as user identifier, specific scene identifier, and specific business identifier, and these identifiers serve as a resource type identifier. For example, when only the user identifier is used as the resource type, one thread processes all data for one user; when only the specific scene identifier is used as the resource type, one thread processes all data for a specified scene, such as a game battle, live streaming, or an application activity within a metaverse, such as releasing sky lanterns at a tourist attraction or a book club activity in a library. Business functions include user avatar actions, voice messages, text messages, images, and videos sent by the user avatar. Of course, these three factors can also be combined to create more granular resource types, such as a business function within a scene as a resource type processed by a separate thread; or a user avatar within a scene as a resource type processed by a separate thread, and so on. By specifying that a resource can only be processed by a specific thread, it is ensured that there is no multi-threaded contention for a single resource. Therefore, locking and unlocking of resources is no longer required during processing, thereby improving business processing efficiency and reducing latency in message transmission. Each business processing thread in the business processing system has a unique identifier. In one embodiment, the business processing thread identifier is obtained by taking the hash value calculated based on the resource type identifier and the modulo operation of the number of business processing threads, and this value is recorded in the business processing thread identifier list.

[0076] To forward client data to the corresponding business processing thread, the server sets up a data routing strategy. This routing strategy refers to the processing method and related parameters used to determine the matching business processing thread for messages involving different types of resources. These parameters include various preset routing identifiers (or resource type identifiers), hash calculation methods, hash value bit lengths, etc. The processing method, for example, is a calculation method to obtain a first value that matches the business processing thread identifier. The routing identifiers correspond one-to-one with the resource type identifiers. For example, a single user identifier, scene identifier, or business identifier can be used as a routing identifier, or two or three of these identifiers can be combined as a routing identifier. In step S203, when the server determines the corresponding business processing thread, in one embodiment, it queries the protocol body content of the first data packet and the user data that sent the first data packet to obtain routing identifier data, such as the user identifier, the scene identifier representing the scene, and the business identifier representing the business. Then, it performs a hash calculation on the routing identifier according to a hash calculation method to obtain a routing hash value with a predetermined number of bits. The hash calculation method, for example, calculates the routing hash value using a certain hash algorithm, such as MD4, MD5, SHA-1, etc. Then, the number of business processing threads is obtained. A value is calculated by taking the modulo between the route hash value and the number of business processing threads. In this invention, this value is called the first value. The identifier of the business processing thread that matches the first value is then matched from the list of business processing thread identifiers.

[0077] Once the business processing thread is determined, a suitable forwarding method is determined based on the resource type and scenario contained in the interaction message, ensuring that at least the interaction message content is forwarded to the business processing thread. Forwarding methods include synchronous calls, asynchronous calls, and event notifications.

[0078] After the business processing thread processes the interaction message according to the corresponding business requirements and business logic, it obtains the transmission content. The transmission content includes at least the second message content to be sent to the receiving end. In other embodiments, parameters such as receiving end identifier, packet type, or message type are also included. In this embodiment, when the server receives a message such as... Figure 3In the illustrated embodiment, when one user avatar sends a data packet containing a text chat message to other user avatars, the business processing thread determines, according to business logic, to push the chat message content to multiple user avatars within the group and obtains the identifiers of the user avatars that need to receive the message. The user avatar identifier (or receiver identifier) ​​and the chat message content are used as the transmission content. This can also include a packet type of "push" or a "text" message type. The server determines the second transmission protocol (numbered 1002) based on the transmission content processed by the business processing thread, and encodes the chat message content. The message type, the encoded chat message content, and the serialized data of the sender identifier are used as the protocol body field content. Simultaneously, other fields of the second data packet are assembled; for example, "1002" is added to the "protocol number" field; "2" is added to the "packet type" field; and the total packet length is calculated and added to the "total packet length" field. The "packet type" can be provided by the business processing thread, determined by the content defined by the transmission protocol, or determined as a push type based on multiple receiver identifiers. Then, in step S206, the second data packet is sent to the user who will receive the message, i.e., the receiving end, as determined by the business processing thread.

[0079] To efficiently transmit various heterogeneous data involved in interactive messages, this invention customizes multiple transmission protocols based on factors such as the data type and scenario of the interactive messages. Each transmission protocol uses an encoding / decoding format corresponding to the data type and scenario to encode / decode the transmitted interactive messages, ensuring that the encoding format or compression rate is applied to the scenario, thereby adapting to the timeliness of interactive messages in different scenarios.

[0080] It should be noted that, for clarity, each embodiment of the present invention is described as a combination of a series of actions or processes. Those skilled in the art should understand that the implementation process is not limited by the order of the described actions or processes, and some steps in the embodiments of the present invention may be processed in other orders or simultaneously.

[0081] The present invention also provides a message transmission and processing system applicable to interactive scenarios. Figure 5This diagram illustrates the structural principle of a message transmission system according to an embodiment of the present invention. It includes a user terminal and a server terminal. The user terminal includes multiple clients 100, each located on a user terminal. The clients communicate with the server via a network. The server terminal includes a message transmission processing system 200 located at the network layer and a service processing system 300 located at the data control layer. The message transmission processing system 200 transmits interactive messages between the sending and receiving ends. The sending and receiving ends refer to the clients that send and receive messages, respectively. Since interactive messages sent between clients typically include business data, the message transmission processing system 200 also requires business processing by the service processing system 300 when transmitting interactive messages. The service processing system 300 is connected to the message transmission processing system 200 in this embodiment. It performs corresponding business processing on messages received from clients by the message transmission processing system 200, and then sends the processed data content back to the message transmission processing system 200, which then sends it to the clients. Specifically, see [link to documentation]. Figure 5 The message transmission processing system 200 includes a server communication module 201, a packet parsing module 202, a routing module 203, a packet data processing module 204, and a packet assembly module 205. The server communication module 201 communicates with the client 100 via a network to receive a first data packet sent by the client 100 as the sender, and to send a second data packet to one or more other clients 100 as receivers. The packet parsing module 202 is connected to the server communication module 201, identifies a first transmission protocol from the first data packet, and parses the first message content sent by the sender from the first data packet according to the parsing method corresponding to the first transmission protocol. The routing module 203 determines the corresponding business processing thread based on the resource type of the first message content and routes the first message content to the business processing thread in the business processing system 300; the business processing thread processes the first message content to obtain transmission content, which includes at least the second message content to be sent to the receiver. The packet data processing module 204 receives the transmission content obtained after processing by the business processing thread in the business processing system 300, determines the second transmission protocol based on the transmission content, and processes the second message content in the transmission content according to the second transmission protocol, such as encoding the second message content according to the corresponding encoding algorithm and encoding parameters, and determining the message type. The packet assembly module 205 is connected to the packet data processing module 204 and the server communication module 201, and is used to assemble the second data packet, for example, determining the content of each field in the second data packet, such as the content of each field in Table 1, and sending it to multiple other clients 100 as receiving ends via push through the server communication module 201.

[0082] To efficiently process various business data, the business processing system 300 of this invention allocates a thread for processing each type of resource. The hash value obtained by hashing the resource type identifier is used as the business processing thread identifier and recorded in the business processing thread identifier list. Correspondingly, the schematic diagram of the routing module 203 in the message transmission processing system 200 is as follows: Figure 6 As shown. The routing module 203 includes a hash calculation unit 2031, a matching unit 2032, and a forwarding unit 2033. The hash calculation unit 2031 queries the parsed first message content and the user data of the sending end to obtain a routing identifier. The routing identifier corresponds to one or more resource type identifiers, and a hash calculation is performed on the routing identifier to obtain a routing hash value. The matching unit 2032 is connected to the hash calculation unit 2031. Based on the routing hash value and the number of business processing threads, a modulo operation is performed to obtain a first value. Based on the calculated first value, a list of business processing thread identifiers is queried, and a matching business processing thread identifier is obtained. The forwarding unit 2033 is connected to the matching unit 2032 and forwards the parsed first message content to the corresponding business processing thread via synchronous call, asynchronous adjustment, or event notification. The number of business processing threads and the list of business processing thread identifiers can be obtained from the business processing system 300.

[0083] like Figure 7 As shown, Figure 7 This is a flowchart illustrating how data from different clients is forwarded to the corresponding business processing threads based on the type of resource being processed. The messages sent by the clients involve certain types of resources, which are obtained after being parsed by the packet parsing module 202. For example... Figure 7 As shown, the messages sent by the three clients include resources A and B. According to the routing strategy, the corresponding business processing flow is determined. Then, the message containing resource A is forwarded to the corresponding resource A processing thread, and the message containing resource B is forwarded to the corresponding resource B processing thread. These requests are then stored as resource requests in the message box of the corresponding business processing thread. The business processing thread processes each resource request in the message box sequentially based on the corresponding resource.

[0084] In data processing, there are often situations where multiple threads perform read and write operations on the same resource. To ensure data consistency, conventional methods use locking (including optimistic and pessimistic locking) to ensure that a resource can only be processed by one thread at a time. However, this approach can significantly impact system performance. In this embodiment, each resource is processed by a single thread, meaning that a specific resource can only be processed by a specific thread. This ensures that there is no multi-threaded contention for each resource, eliminating the need for locking and unlocking, thereby improving data processing efficiency.

[0085] Figure 8 This is a schematic diagram of a server-side framework structure according to an embodiment of the present invention. In this embodiment, the client-side Session Manager in the server-side framework structure is responsible for maintaining communication with the client and storing user data of the current session, such as user ID, received message ID, pushed message ID, etc., thus acting as a communication module. After receiving a client data packet, the client-side Session Manager sends it to the encoding / decoding module. The encoding / decoding module includes two parts: one part is used to parse the client data packet, identify the transmission protocol and transmission parameters (such as packet type, message ID, etc.) used by the client, parse the interactive message content sent by the sender in the protocol body field of the first data packet according to the transmission protocol, and send the user ID, message ID, and other information that may be included in the protocol body content to the client-side Session Manager. At the same time, the parsed interactive message content and user data are sent to the consistent hash routing unit. The consistent hash routing unit obtains the number and identifier list of business processing threads from the registry center for the instance Session Manager, performs hash calculation and modulo calculation based on the routing identifier corresponding to the resource type identifier in the interactive message content, matches the corresponding business processing thread, and notifies the forwarding unit. The forwarding unit determines the appropriate forwarding method based on the resource type and scenario contained in the interaction message content, such as synchronous call, asynchronous call, or event notification, and forwards at least the interaction message content and the identified business processing thread identifier to the instance Session Manager. The business processing system includes multiple business processing threads, each corresponding to an instance and used to process one type of resource, such as... Figure 8 Instances A, B, and C handle three different types of resources. Each instance registers with the registry center after creation, including registration address information and instance name. The registry center then sends the registration information of each instance to the instance session manager. Upon receiving the interaction message content forwarded by the forwarding unit and the identified business processing thread identifier, the instance session manager sends the interaction message content to the corresponding instance in the business cluster, such as instance A or instance B, based on the instance address information corresponding to the business processing flow. After processing by the instance, the processed data is returned to the instance session manager, which then returns it to the encoding / decoding module. The encoding / decoding module determines the appropriate transmission protocol based on the returned data, encodes the message content to be sent to the receiving end according to the transmission protocol, determines other protocol body content, and adds these data sequences to the protocol body fields. Simultaneously, it assembles other fields of the data packet. After the data packet is assembled, it is sent to the receiving end via the client session manager.

[0086] In this embodiment, the user's interactive messages are routed from the gateway layer to the processing instances, and the processing results from the processing instances are also forwarded back to the client by the gateway layer. The server provides multiple business processing instances that process different resources, and the processing instances are directly connected without going through a gateway, thus saving network overhead and further enhancing the transmission of interactive messages.

[0087] Those skilled in the art will understand that the embodiments described herein are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] Figure 9 This is a schematic diagram of the structure of an electronic device used to implement the message transmission processing method of the embodiments of the present invention. The electronic device 10 includes a processor 11, a memory 12, and a communication bus for connecting the processor 11 and the memory 12. The memory 12 stores a computer program that can run on the processor 11. When the processor 11 runs the computer program, it can execute or implement the steps of the methods in the various embodiments of the present invention. The electronic device 10 can be a server in the embodiments of the present invention, or it can be a cloud server. The electronic device 10 can also be a terminal device or an AR device in the embodiments of the present invention. Where appropriate, the electronic device can also be called a computing device. The electronic device 10 can also be a cloud server. The electronic device 10 also includes a communication interface for receiving and sending data.

[0089] In some embodiments, processor 11 may be a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), modem processor, image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, neural network processing unit (NPU), etc. Processor 11 may also be other general-purpose processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors may be microprocessors or any conventional processor. Among them, the neural network processor (NPU), by drawing inspiration from biological neural network structures, can quickly process input information and continuously learn itself. NPU electronic devices can realize applications such as intelligent cognition, image recognition, face recognition, semantic recognition, speech recognition, and text understanding.

[0090] In some embodiments, memory 12 may be an internal storage unit of electronic device 10, such as a hard disk or memory of electronic device 10; memory 12 may also be an external storage device of electronic device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device 10. Memory 12 may include both internal storage units and external storage devices of electronic device 10. Memory 12 can be used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 12 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 12 is used to store program code executed by electronic device 10 and data transmitted. Memory 12 can also be used to temporarily store data that has been output or will be output.

[0091] Those skilled in the art will understand that Figure 9 This is merely an example of electronic device 10 and does not constitute a limitation on electronic device 10. Electronic device 10 may include more or fewer components than shown, or combine certain components, or include different components, such as input / output devices, network access devices, etc.

[0092] Figure 10 This is a schematic diagram of the software structure of a terminal device according to an embodiment of the present invention. Taking the Android system as the mobile phone operating system as an example, in some embodiments, the Android system is divided into four layers: application layer, application framework layer (Framework, FWK), system layer, and hardware abstraction layer. The layers communicate with each other through software interfaces.

[0093] First, the application layer can include multiple application packages, which can be various application apps such as calling, camera, video, navigation, weather, instant messaging, education, etc., or AR-based application apps.

[0094] Second, the Application Framework Layer (FWK) provides Application Programming Interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer can include predefined functions, such as functions for receiving events sent by the application framework layer.

[0095] The application framework layer may include a window manager, a resource manager, and a notification manager, among others.

[0096] The window manager manages the windowed applications. It can determine the screen size, the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, and phonebook entries.

[0097] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0098] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0099] In addition, the application framework layer may include a view system, which includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application. The display interface can consist of one or more views; for example, the display interface of a text notification icon may include a view for displaying text and a view for displaying images.

[0100] Third, the system layer can include multiple functional modules, such as sensor service modules, physical state recognition modules, 3D graphics processing libraries (e.g., OpenGLES), and so on.

[0101] The sensor service module monitors sensor data uploaded by various sensors at the hardware layer to determine the physical state of the phone; the physical state recognition module analyzes and recognizes user gestures, faces, etc.; and the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0102] In addition, the system layer may include a surface manager and a media library. The surface manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files.

[0103] Finally, the hardware abstraction layer is the layer between hardware and software. The hardware abstraction layer can include display drivers, camera drivers, sensor drivers, etc., used to drive the relevant hardware in the hardware layer, such as displays, cameras, and sensors.

[0104] This invention also provides a computer-readable storage medium storing a computer program or instructions, which, when executed, implement the message transmission processing method for interactive scenarios applied to a server or a terminal device as described in the above embodiments.

[0105] This invention also provides a computer program product, including a computer program or instructions, which, when executed, implement the message transmission processing method for interactive scenarios applied to a server or a terminal device as described in the above embodiments. For example, the computer program product may be a software installation package.

[0106] Those skilled in the art should understand that the functions of the methods, steps, or related modules / units described in the embodiments of the present invention can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product, or by a processor executing computer program instructions. The computer program product includes at least one computer program instruction, which can be composed of corresponding software modules. These software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, read-only optical disk (CD-ROM), or any other form of storage medium known in the art. The computer program instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., SSD).

[0107] Regarding the various devices / products described in the above embodiments, the modules / units included can be software modules / units, hardware modules / units, or a combination of both. For example, for devices / products applied to or integrated into a chip, all of its modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining modules / units can be implemented using hardware methods such as circuits. Similarly, for devices / products applied to or integrated into a terminal, all of its modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the terminal, while the remaining modules / units can be implemented using hardware methods such as circuits.

[0108] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A message transmission processing method applied to a metaverse interaction scenario, characterized in that, The method is applied to a server and is used to transmit interactive messages between a sender and a receiver. The method includes: Upon receiving the first data packet from the sender, the first transmission protocol used to send the first data packet is identified; The first message content sent by the sending end is parsed from the first data packet according to the parsing method corresponding to the first transmission protocol. The first transmission protocol is determined based on the data type of the first message content sent by the user avatar and the metaverse interaction scene in which the user avatar is located. The data type includes text, image, audio, video, action identifier representing an action, and scene identifier. The scene identifier is used to characterize the metaverse interaction scene as a small-scale scene, a large-scale scene, or a game battle scene. When the data type of the first message content is a voice message, the compression ratio used when encoding the voice message in a small-scale scene is less than the compression ratio used when encoding the voice message in a large-scale scene. The parsed content of the first message and the user data of the sending end are queried to obtain the user identifier, scene identifier and business identifier, and the user identifier, scene identifier and business identifier are combined as the routing identifier; The route identifier is hashed according to the hash calculation method to obtain a route hash value with a predetermined number of bits. A first value is obtained by performing a modulo operation based on the route hash value and the number of service processing threads. Then, a service processing thread identifier that matches the first value is matched from the list of service processing thread identifiers. Once the business processing thread is determined, the forwarding method is determined based on the resource type and scenario contained in the first message content, and the first message content is forwarded to the business processing thread at least. The forwarding method includes synchronous call, asynchronous call, and event notification. After the first message content is processed by the business processing thread, the transmission content is obtained. The transmission content includes at least the second message content that should be sent to the receiving end. Based on the transmission content, a second transmission protocol is determined, and the second message content is processed according to the processing method of the second transmission protocol; and Assemble a second data packet, wherein the second data packet includes at least the processed second message content and the second transmission protocol identifier, and send the second data packet to the receiving end; The first message content and the second message content are the interactive message content between user avatars in the metaverse interaction scenario. The processing methods provided by the first transmission protocol and the second transmission protocol correspond to the data types of the metaverse interaction scenario and the interactive message content, respectively. Different transmission protocols are used to define the data types of the interactive message content they transmit, as well as whether encoding is required when transmitting the interactive message content, and the encoding format used when encoding is required. Different transmission protocols are used to adapt to the transmission of various heterogeneous data in the metaverse interaction scenario.

2. The method according to claim 1, characterized in that, The first / second data packet includes a protocol identifier field, and the first transmission protocol is identified based on the content of the protocol identifier field in the first data packet; when assembling the second data packet, the second transmission protocol identifier is added to the protocol identifier field.

3. A message transmission processing method applied to a metaverse interaction scenario, characterized in that, The method is applied to a client in a user terminal. When the client acts as a sender, the method includes: The first transmission protocol is determined based on the data type of the first message content sent by the user avatar and the metaverse interaction scenario in which the user avatar is located. The data type includes text, image, audio, video, action identifier representing an action, and scene identifier. The scene identifier is used to characterize whether the metaverse interaction scenario is a small-scale scenario, a large-scale scenario, or a game battle scenario. When the data type of the first message content is a voice message, the compression ratio used when encoding the voice message in a small-scale scenario is lower than the compression ratio used when encoding the voice message in a large-scale scenario. The first message content is processed according to the processing method of the first transmission protocol; and The processed first message content and the first transmission protocol identifier are assembled into a first data packet and sent to the server. The server queries the parsed first message content and the user data of the sender to obtain the user identifier, scene identifier, and service identifier. The user identifier, scene identifier, and service identifier are combined as a routing identifier. The routing identifier is hashed according to a hash calculation method to obtain a routing hash value with a predetermined number of bits. A first value is obtained by performing a modulo operation based on the routing hash value and the number of service processing threads. A service processing thread identifier that matches the first value is matched from the list of service processing thread identifiers. After the service processing thread is determined, the forwarding method is determined based on the resource type and scene contained in the first message content, and the first message content is forwarded to at least the service processing thread. The forwarding method includes synchronous call, asynchronous call, and event notification. When the client acts as the receiving end, the method includes: Upon receiving the second data packet sent by the server, the second transport protocol is identified from it; and The second message content sent by the server is parsed from the second data packet according to the parsing method corresponding to the second transmission protocol. The second message content is the message content after the server processes the first message content sent by the sender. The processing methods provided by the first and second transmission protocols correspond to the data types of the metaverse interaction scenario and the interaction message content, respectively. The first message content and the second message content are the interaction message content between user avatars in the metaverse interaction scenario. Different transmission protocols are used to define the data type of the interaction message content they transmit, as well as whether encoding is required when transmitting the interaction message content and the encoding format used when encoding is required. Different transmission protocols are used to adapt to the transmission of various heterogeneous data in the metaverse interaction scenario.

4. A message transmission and processing system applied to a metaverse interaction scenario, characterized in that, An application on the server side for transmitting interactive messages between a sender and a receiver, the system includes: The server-side communication module communicates with both the sending and receiving ends. It is configured to receive the first data packet sent by the sending end and send the second data packet to the receiving end. The packet parsing module, connected to the server-side communication module, is configured to identify the first transmission protocol from the first data packet sent by the sender, and parse the first message content sent by the sender from the first data packet according to the parsing method corresponding to the first transmission protocol. The first transmission protocol is determined based on the data type of the first message content sent by the user avatar and the metaverse interaction scene in which the user avatar is located. The data type includes text, images, audio, video, action identifiers representing actions, and scene identifiers. The scene identifiers are used to characterize whether the metaverse interaction scene is a small-scale scene, a large-scale scene, or a game battle scene. When the data type of the first message content is a voice message, the compression ratio used when encoding the voice message in a small-scale scene is less than the compression ratio used when encoding the voice message in a large-scale scene. A routing module, connected to the packet parsing module, is configured to query the parsed first message content and the sender's user data to obtain a user identifier, a scene identifier, and a service identifier. The user identifier, scene identifier, and service identifier are combined as a routing identifier. A hash calculation is performed on the routing identifier to obtain a routing hash value with a predetermined number of bits. A first value is obtained by performing a modulo operation based on the routing hash value and the number of service processing threads. A service processing thread identifier matching the first value is then matched from the service processing thread identifier list. Once a service processing thread is determined, a forwarding method is determined based on the resource type and scene contained in the first message content. At least the first message content is forwarded to the service processing thread. The forwarding method includes synchronous calls, asynchronous calls, and event notifications. The service processing thread processes the content to obtain transmission content, which includes at least the second message content to be sent to the receiving end. The packet data processing module is configured to determine a second transmission protocol based on the transmission content, and processes the second message content according to the processing method of the second transmission protocol; and A packet assembly module, which is connected to the packet data processing module and the server communication module, is configured to assemble a second data packet, wherein the second data packet includes at least the processed second message content and the second transmission protocol identifier, and sends the second data packet to the receiving end via the server communication module; The processing methods provided by the first and second transmission protocols correspond to the data types of the metaverse interaction scenario and the interaction message content, respectively. The first message content and the second message content are the interaction message content between user avatars in the metaverse interaction scenario. Different transmission protocols are used to define the data type of the interaction message content they transmit, as well as whether encoding is required when transmitting the interaction message content and the encoding format used when encoding is required. Different transmission protocols are used to adapt to the transmission of various heterogeneous data in the metaverse interaction scenario.

5. The system according to claim 4, characterized in that, The routing module includes: The hash calculation unit is configured to query the parsed first message content and the user data of the sending end to obtain a route identifier, and to perform a hash calculation on the route identifier to obtain a route hash value, wherein the route identifier corresponds to one or more resource type identifiers; A matching unit, connected to the hash calculation unit, is configured to perform a modulo operation based on the route hash value and the number of service processing threads to obtain a first value; match a service processing thread identifier that matches the first value; and A forwarding unit, which is connected to the matching unit, is configured to route the first message content to the service processing thread.

6. A server, characterized in that, It includes a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the message transmission processing method applied to the metaverse interaction scenario as described in any one of claims 1-2.

7. A terminal device, characterized in that, It includes a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the message transmission processing method for applying a metaverse interaction scenario as described in claim 3.

8. A computer-readable storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the message transmission processing method for a metaverse interaction scenario as described in any one of claims 1-3.

9. A computer program product, characterized in that, It includes computer program instructions, which, when executed by a processor, implement the message transmission processing method for a metaverse interaction scenario as described in any one of claims 1-3.

Images