Method and system for multi-player online multi-player screen sharing by different software digital humans

Abstract

This invention provides a method and system for enabling multi-person online interaction and simultaneous video feeds using digital humans from different software, relating to the field of data processing technology. The method includes: adjusting a virtual camera in the second digital human software to the same viewpoint as the host's end, moving the second digital human model within the virtual camera's field of view, rendering a guest performance scene with a background that can be removed, and uploading the guest performance scene to a server in real time; on the server, extracting the foreground image of the second digital human based on the received guest performance scene; superimposing the foreground image of the second digital human onto the background image to generate a composite multi-person video feed; and sending the composite multi-person video feed to the host's end and each guest's end in real time. This invention enables all participants to obtain low-latency multi-person online video feeds in real time, meeting the needs of interactive performances and live streaming.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for enabling multiple people to connect online and view the same screen simultaneously using digital humans with different software. Background Technology

[0002] In existing technologies, to enable multiple digital humans from different companies or developed using different software to perform together (e.g., joint live streams of virtual idols, remote multi-person virtual press conferences), the common approach is as follows: each party needs to package and submit their respective digital human assets (including 3D models, skeletal rigging information, textures, animation blueprints, driving logic, and other core data) to one party, which then imports all assets into a single 3D rendering engine (such as Unity or Unreal Engine). After importation, due to the differences in skeletal architecture, coordinate systems, driving interfaces, and material definitions between different software, they are often not directly compatible. Technicians must perform a series of complex rebinding and adaptation tasks for each digital human, including skeletal relocation, animation state machine reconstruction, facial expression remapping, and rendering pipeline adaptation.

[0003] This process has the following drawbacks: First, the entire process is extremely time-consuming, from asset collection and environment setup to individual adaptation testing, which usually takes several days or even weeks, making it difficult to meet the rapid response requirements of online real-time collaboration; second, and more importantly, the solution requires each participating party to directly hand over its core digital assets, which creates the risk of intellectual property leakage and greatly reduces the willingness of different companies, especially those with business competition or those that value asset confidentiality, to participate in digital human collaboration. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for realizing multi-person online connection and multi-person simultaneous screen display using digital humans with different software, so that all participants can obtain low-latency multi-person online simultaneous screen display in real time, which meets the needs of interactive performances and live broadcasts.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a method for enabling multiple digital humans to connect online and appear on the same screen simultaneously using different software, the method comprising: Install a camera synchronization plugin in the first digital human software on the homeowner's side, and install a camera synchronization plugin in at least one second digital human software on the guest's side. The first digital human software and the second digital human software are digital human driving software provided by different developers. The virtual camera parameters are obtained from the camera synchronization plugin on the host's side and sent to the camera synchronization plugins on each guest's side in real time via the network, so as to obtain virtual camera data that is synchronized and consistent on each side. On the homeowner's end, based on the synchronized virtual camera data, the virtual camera in the first digital human software is adjusted, and the first digital human model is moved into the shooting range of the virtual camera, rendering and generating the base image, and uploading the base image to the server in real time; On each guest's end, based on synchronized virtual camera data, the virtual camera in the second digital human software is adjusted to the same perspective as the host's end, and the second digital human model is moved into the shooting range of the virtual camera to render and generate guest performance footage with background removal capabilities, and the guest performance footage is uploaded to the server in real time. On the server, the foreground image of the second digital human is extracted based on the received footage of the guest's performance; Based on the foreground image of the second digital human and the background image, the foreground image of the second digital human is superimposed and composited onto the background image to generate a composite image of multiple people on the same stage. The composite image of multiple people on the same stage is sent to the host's and each guest's terminals in real time.

[0006] Secondly, a system that enables multiple users to connect online and view the same screen simultaneously using digital humans from different software includes: On the homeowner's end, the First Digital Human software and camera synchronization plugin are deployed; At least one guest-side device is equipped with a second digital human software and a camera synchronization plugin. The first and second digital human software are digital human driver software provided by different developers. The server communicates and connects with the landlord and each tenant via the network. The camera synchronization plugin on the host's side is used to obtain virtual camera parameters and send them to the camera synchronization plugins on each guest's side in real time, so that each side obtains synchronized and consistent virtual camera data. The homeowner's side is used to adjust the virtual camera in the first digital human software based on synchronized virtual camera data, move the first digital human model into the shooting range, render and generate the base image, and upload the base image to the server in real time; Each guest's end is used to adjust the virtual camera in the second digital human software to the same perspective as the host's end based on synchronized virtual camera data, move the second digital human model into the shooting range, render and generate guest performance footage with background removal capability, and upload the guest performance footage to the server in real time. The server is used to extract the foreground image of the second digital human from the performance footage of the guests, synthesize the foreground image of the second digital human with the background image to generate a composite image of multiple people on stage, and distribute the composite image of multiple people on stage to the host and each guest in real time.

[0007] The above-described solution of the present invention has at least the following beneficial effects: Significantly improves the efficiency of simultaneous performances, eliminating the cumbersome steps of asset migration, engine switching, and rebinding, and quickly completing the preparation for multiple digital humans to perform together; it can avoid the direct exchange of digital human assets, effectively protecting the digital assets of various developers and increasing the willingness of different companies to participate in simultaneous performances; with the help of unified virtual camera data, it ensures that the perspective relationship and field of view of digital humans in different software are reasonably consistent, presenting a natural and harmonious simultaneous performance; the server side performs real-time keying and compositing, and sends the composited image back to each terminal or directly pushes it to the live broadcast, so that all participants can obtain low-latency multi-person simultaneous performance images in real time, meeting the needs of interactive performances and live broadcasts. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a method for enabling multiple people to connect online and view the same screen using digital humans from different software, as provided in an embodiment of the present invention.

[0009] Figure 2 This is a schematic diagram of a system provided by an embodiment of the present invention, which enables multiple people to connect online and view the same screen simultaneously using digital humans with different software. Detailed Implementation

[0010] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough understanding of the present disclosure and to fully convey its scope to those skilled in the art.

[0011] To enable those skilled in the art to better understand the technical solutions of the present invention, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. Where there is no conflict, the following embodiments and their technical features can be combined with each other.

[0012] Example 1 This embodiment provides a method for enabling multiple users to connect online and view the same screen simultaneously using digital humans from different software. Figure 1 This is a flowchart illustrating a method for enabling multiple users to connect online and view the same screen simultaneously using digital humans from different software, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps.

[0013] Step S101: Install a camera synchronization plugin in the first digital human software on the homeowner's end, and install a camera synchronization plugin in at least one second digital human software on the guest's end. The first digital human software and the second digital human software are digital human driver software provided by different developers.

[0014] Specifically, the host's end refers to the user terminal device that initiates a multi-person performance and is responsible for uniformly controlling the performance's visual perspective, while the guest's end refers to the user terminal device that is invited to join the performance. Both the host's and guest's ends can be personal computers, workstations, or other computing devices with graphics rendering capabilities that have digital human driving software installed. Digital human driving software refers to the software used to load locally stored digital human asset files, drive the digital human model to complete actions and expressions, and render and output the digital human's visuals. The first and second digital human software are provided by different developers, and their digital human skeletal architecture, coordinate systems, driving interfaces, and material definitions are different, making them incompatible with the same rendering engine.

[0015] A camera synchronization plugin is a functional module embedded in the runtime environment of digital human software. On the host's end, it reads the camera parameters of the virtual camera within the digital human software and sends these parameters externally. On the guest's end, it receives the camera parameters and adjusts the virtual camera within the guest's digital human software accordingly. Because the camera synchronization plugin only reads and writes the virtual camera parameters and does not access the digital human model's mesh, texture maps, or skeletal binding data, it can be adapted to digital human software from different developers without affecting their core digital assets. In this embodiment, the host's end and at least one guest's end establish communication connections with the server via a network, collectively forming a multi-user online system.

[0016] In step S102, the virtual camera parameters are obtained by the camera synchronization plugin on the host's end, and the virtual camera parameters are sent to the camera synchronization plugins on each guest's end in real time via the network to obtain virtual camera data that is synchronized and consistent on each end.

[0017] Specifically, virtual camera parameters include the virtual camera's position coordinates in three-dimensional space, rotation quaternions, and field of view. Position coordinates refer to the coordinate values ​​of the virtual camera along three mutually perpendicular coordinate axes in the three-dimensional virtual space coordinate system. Rotation quaternions are mathematical representations of the virtual camera's orientation in three-dimensional space, consisting of one real part and three imaginary parts. The combination of these four values ​​uniquely determines the virtual camera's orientation after rotating around any axis at any angle. The field of view is the angle of view subtended by the virtual camera in the shooting direction; a larger field of view results in a wider field of view.

[0018] The specific implementation process of obtaining virtual camera parameters through the camera synchronization plugin on the host's end is as follows: The camera synchronization plugin on the host's end captures the panning, rotation, and zooming controls applied by the user to the virtual camera via input devices. These input devices can be mice, keyboards, or touchpads, etc. Panning refers to the control that moves the virtual camera to a different position in three-dimensional space; rotation refers to the control that changes the virtual camera's orientation; and zooming refers to the control that moves the virtual camera closer to or further away from the viewpoint.

[0019] For translation control, the displacement components corresponding to the translation control in the three coordinate axes are added one by one to the components of the virtual camera's current position coordinates along the corresponding coordinate axis to obtain the updated position coordinates.

[0020] For rotational manipulation, the rotation angle corresponding to the rotational manipulation is converted into an incremental quaternion according to the rotation axis direction in three-dimensional space. The conversion rule is as follows: take the cosine value of half of the rotation angle as the real part of the incremental quaternion, take the sine value of half of the rotation angle, and multiply it by the unit component of the rotation axis in the three coordinate axis directions respectively to obtain the three imaginary parts of the incremental quaternion. Then, according to the quaternion multiplication rule, cross-multiply the four components of the incremental quaternion with the four components of the current rotational quaternion, and accumulate and combine each product according to the real part and the three imaginary parts, to obtain the composite quaternion. Finally, divide the four components of the composite quaternion by the square root of the sum of the squares of the four components to restore the modulus of the quaternion to one, and obtain the updated rotational quaternion. Through the above normalization process, the rotational attitude distortion caused by the accumulation of numerical errors after multiple rotational composites can be avoided.

[0021] For zoom control, the scroll amount corresponding to the zoom control is multiplied by a preset field of view adjustment coefficient to obtain the field of view increment. This increment is then subtracted from the current field of view of the virtual camera to obtain the intermediate field of view. When the intermediate field of view is less than the preset minimum field of view, the minimum field of view is used as the updated field of view. When the intermediate field of view is greater than the preset maximum field of view, the maximum field of view is used as the updated field of view. Otherwise, the intermediate field of view is used as the updated field of view. By setting upper and lower limits for the field of view, image distortion caused by excessive magnification or reduction of the field of view can be avoided.

[0022] The updated position coordinates, rotation quaternions, and field of view are combined to form virtual camera parameters. The host's camera synchronization plugin packages these virtual camera parameters into data messages according to a preset synchronization cycle and sends them in real-time to the camera synchronization plugins on each guest's end via the network. This can be done either by the host sending the parameters directly to each guest individually, or by the host sending them to the server first, which then forwards them to each guest. Upon receiving the virtual camera parameters, each guest's camera synchronization plugin writes them into the corresponding attributes of the virtual camera in their local second-hand digital human software, ensuring that the virtual cameras on the host's end and each guest's end maintain the same pose and framing, thus obtaining consistent virtual camera data across all ends. Therefore, every adjustment the host makes to the performance perspective is reflected in real-time on all participating ends, guaranteeing consistency in the viewing angle across all ends.

[0023] Step S103: On the homeowner's end, based on the synchronized virtual camera data, adjust the virtual camera in the first digital human software, move the first digital human model to the shooting range of the virtual camera, render and generate a base image, and upload the base image to the server in real time.

[0024] Specifically, in the first digital human software on the homeowner's end, a preset 3D virtual scene is loaded. A 3D virtual scene refers to a virtual environment composed of 3D models, lighting, materials, etc., used as a stage background for performances, such as a virtual studio or virtual stage. Based on synchronized virtual camera data, the position coordinates, rotation quaternions, and field of view are written into the corresponding position, rotation, and field of view attributes of the virtual camera in the first digital human software, thus completing the adjustment of the virtual camera.

[0025] Subsequently, a view frustum corresponding to the virtual camera is constructed. The view frustum is a cone-shaped visible space formed by the virtual camera's location as its vertex, its orientation as its axis, and its field of view as its angle, extending outwards. Only objects falling within this visible space are rendered in the image. Based on the view frustum, visibility clipping is performed on the 3D virtual scene: each 3D object in the 3D virtual scene is checked to see if it intersects with the view frustum. 3D objects completely outside the view frustum are removed from the rendering list for this frame, and only 3D objects falling within or partially within the view frustum are retained for rendering. This reduces unnecessary rendering calculations and ensures a high frame rate for real-time rendering.

[0026] Simultaneously, the first digital human model is moved to a predetermined position within the 3D virtual scene. This predetermined position refers to a pre-set location within the 3D virtual scene for the digital human model to stand and perform, ensuring the model falls within the shooting range of the virtual camera. The 3D virtual scene and the first digital human model within the viewfinder are rendered, resulting in a background image containing the first digital human model and the 3D virtual background. This background image serves as the background for the composite image of multiple people on stage and as the carrier of the content performed by the host. The host's client compresses and encodes the background image frame by frame and uploads it to the server in real-time via the network.

[0027] Step S104: On each guest's end, based on synchronized virtual camera data, adjust the virtual camera in the second digital human software to the same perspective as the host's end, move the second digital human model into the shooting range of the virtual camera, render and generate a guest performance scene with a background that can be removed, and upload the guest performance scene to the server in real time.

[0028] Specifically, the camera synchronization plugin on each guest end writes the position coordinates, rotation quaternions, and field of view from the received virtual camera data into the corresponding attributes of the virtual camera in the local second digital human software, so that the virtual camera on the guest end and the virtual camera on the host end have the same viewpoint; and moves the second digital human model into the shooting range of the virtual camera.

[0029] Because the 3D virtual scenes of digital human software from different developers are different and incompatible with each other, directly rendering the 3D virtual scene on the guest's side will not be able to blend with the base image on the host's side. Therefore, this embodiment generates an image with a background that can be removed when rendering the guest's performance. The background that can be removed is a solid color background of a predetermined color, such as a pure green background or a pure blue background. The specific implementation process is as follows: In the second digital human software, a command to start the same-stage mode is received. This command can be triggered by the user on the guest's side through interface operation, or it can be automatically triggered by the camera synchronization plugin when joining the same-stage activity. According to the command, the 3D virtual scene rendering stage in the rendering pipeline of the second digital human software is disabled. The rendering pipeline refers to the process flow of the digital human software to process the 3D scene and digital human model in stages and finally render it into a frame. Disabling the 3D virtual scene rendering stage means skipping the rendering of background objects in the 3D virtual scene during the rendering of this frame. The color clearing value of the frame buffer is set to a predetermined color. The frame buffer is a storage area in the graphics card used to temporarily store all pixel color data of a frame. The color clearing value is the initial color value used to fill the frame buffer before each frame is rendered.

[0030] According to the configured rendering pipeline, at the start of each frame's rendering, all pixels in the frame buffer are first filled with a predetermined color using a color clearing value. Then, only the second digital human model is rendered to the frame buffer. The pixel area covered by the second digital human model retains the model's original rendered color, while the remaining areas are automatically filled with the predetermined color using the clearing value. This results in a guest performance scene where the model area retains its original rendered color, and the background area is the predetermined color. The guest client compresses and encodes the guest performance scene frame by frame and uploads it to the server in real time via the network. Because the background of the guest performance scene is a single predetermined color, the server can complete background removal at low cost and high speed, meeting the low latency requirements of real-time performances.

[0031] Step S105: On the server, extract the foreground image of the second digital human based on the received performance footage of the tenant.

[0032] Specifically, after the server decodes the received performance footage of the guests, it performs the following processing on each pixel in the image: the color value of the current pixel is split into red, green and blue components, and the predetermined color is also split into the corresponding three components. The difference between each component of the current pixel and the corresponding component of the predetermined color is calculated. The squares of the differences of the three components are calculated and then added together. The square root of the sum is then taken to obtain the distance between the color value of the current pixel and the predetermined color in the color space.

[0033] Based on the aforementioned distances, a corresponding transparency coefficient is generated for each pixel. The transparency coefficient characterizes the degree to which a pixel should be removed; a higher transparency coefficient indicates a more transparent pixel. The generation rules for the transparency coefficient are as follows: a first distance threshold and a second distance threshold are preset, with the first distance threshold being less than the second distance threshold. When the pixel distance does not exceed the first distance threshold, the transparency coefficient of that pixel is set to its maximum value, meaning the pixel is considered fully transparent. When the pixel distance is not less than the second distance threshold, the transparency coefficient of that pixel is set to zero, meaning the pixel is considered completely opaque. When the pixel distance is between the first and second distance thresholds, the first distance threshold is subtracted from the pixel distance to obtain a first difference; the first distance threshold is subtracted from the second distance threshold to obtain a second difference. The ratio of the first difference to the second difference is used as the transparency coefficient of that pixel, allowing the transparency coefficient to smoothly transition from its maximum value to zero as the distance increases. In general, the smaller the distance, the higher the transparency coefficient; that is, pixels whose color is closer to the predetermined color are more transparent.

[0034] Based on the transparency coefficient, pixel areas in the guest's performance image that are close to the predetermined color are set to full transparency, while the remaining pixel areas retain their original color and opacity. A corresponding transparency coefficient is added to each pixel, resulting in a second digital human foreground image containing a transparency channel. The transparency channel refers to an additional data component added to the pixel's red, green, and blue color components to record the pixel's transparency level. By setting a first distance threshold and a second distance threshold to create a smooth transition band, harsh jagged edges or residual background color edges on the digital human model can be avoided, improving the keying quality.

[0035] Step S106: Based on the foreground image of the second digital human and the background image, the foreground image of the second digital human is superimposed and synthesized into the background image to generate a composite image of multiple people on the same stage.

[0036] Specifically, on the server, based on synchronized virtual camera data and the preset position information corresponding to each guest's camera, the three-dimensional spatial coordinates of each second digital human model in the virtual scene of the base image are calculated. The preset position information refers to the coordinate information of the positions assigned to each guest's digital human model in the host's three-dimensional virtual scene beforehand. Since all terminals use synchronized virtual camera data, the virtual cameras on each terminal and the host's three-dimensional virtual scene are under the same spatial coordinate description; therefore, the position coordinates recorded in the preset position information are directly used as the three-dimensional spatial coordinates of the corresponding second digital human model in the virtual scene of the base image.

[0037] Based on the virtual camera positions in the 3D spatial coordinates and virtual lens data, the vertical distance from each second digital human model to the virtual camera's imaging plane is calculated, yielding the depth value corresponding to the foreground image of each second digital human. The imaging plane is an imaginary plane perpendicular to the virtual camera's orientation, used to present the captured image. The specific calculation rule for the depth value is as follows: Subtract the components of the virtual camera's position coordinates along the corresponding coordinate axes from the components of the second digital human model's 3D spatial coordinates along the three coordinate axes, obtaining the three components of the relative vector pointing from the virtual camera to the second digital human model. Multiply these three components of the relative vector one-to-one with the three components of the unit direction vector of the virtual camera's orientation, and then add the three products together. The sum is the vertical distance from the second digital human model along the virtual camera's orientation to the imaging plane, which is used as the depth value corresponding to the foreground image of the second digital human. A larger depth value indicates that the second digital human model is farther from the virtual camera.

[0038] Based on the depth values, multiple foreground images of the second digital human are sorted in descending order of depth value to obtain a stacking order from far to near. Following this stacking order, starting with the farthest foreground image, each image is sequentially overlaid onto the base image using a transparency blending method. The specific rules for transparency blending are as follows: for each pixel in the foreground image to be overlaid, subtract the pixel's transparency coefficient from 1 to obtain the pixel's opacity; multiply the pixel's color value by the opacity to obtain the first color component; multiply the color value of the corresponding pixel in the base image by the pixel's transparency coefficient to obtain the second color component; add the first color component and the second color component to obtain the color value of the pixel after overlay; perform the above processing pixel by pixel until all pixels of the foreground image are overlaid. The currently overlaid foreground image and the base image together serve as the background image for the next overlay. The remaining foreground images of the second digital human are then overlaid according to the stacking order until all foreground images of the second digital human are overlaid, resulting in a composite image of multiple people on stage with correct foreground and background occlusion relationships. Because digital figures farther from the virtual camera are superimposed first and those closer are superimposed later, when two digital figures overlap in the image, the closer digital figure can correctly occlude the farther digital figure, making the occlusion relationship of the composite image conform to the real perspective law.

[0039] Step S107: The composite image of multiple people on the same stage is sent to the host's terminal and each guest's terminal in real time.

[0040] Specifically, the server acquires network status information periodically reported by the host and each guest, including available bandwidth, packet loss rate, and round-trip latency. Based on this information, the target bitrate and resolution for transcoding are determined for each client. Bitrate refers to the amount of video data per unit time, and resolution refers to the number of pixels horizontally and vertically. The determination rule is as follows: the available bandwidth of a client is compared with several preset bandwidth tiers to determine its appropriate tier. The bitrate and resolution corresponding to that tier are then selected as the target bitrate and resolution for that client. The higher the available bandwidth, the higher the selected target bitrate and resolution. For clients with packet loss rates or round-trip latency exceeding preset thresholds, the target bitrate is lowered accordingly to ensure smooth video transmission.

[0041] Based on the composite image of multiple participants and the target bitrate and resolution for each device, real-time transcoding is performed. This involves compressing and encoding the video data of the composite image into streaming media data with different bitrates and resolutions, generating multiple adapted streaming media data streams. These multiple streaming media data streams are then pushed to the respective host and guest devices via transmission channels established by the server with the host and guests respectively. Each device decodes the received streaming media data and displays the composite image of multiple participants. As a result, all participants can watch the composite image of multiple participants in real time with picture quality matching their own network conditions, and even participants in weak network environments can enjoy a smooth viewing experience.

[0042] Step S108: On the server, based on the composite image of multiple people on the same stage, a hardware or software video encoder is invoked to perform real-time compression to obtain a video stream conforming to the network transmission format. According to the preset target live streaming platform push address, the video stream is pushed to the content distribution entry of the target live streaming platform through the push module deployed on the server. Pushing the stream refers to the process of continuously sending the video stream to the live streaming platform server according to the streaming media transmission protocol agreed upon by the live streaming platform. After receiving the video stream, the content distribution entry of the target live streaming platform distributes it to the viewers watching the live stream via the live streaming platform's content distribution network, thereby realizing real-time live streaming of the composite image of multiple people on the same stage.

[0043] It should be noted that throughout the entire process of achieving a joint performance, the data exchanged between the host's and each guest's devices via the network only includes virtual camera parameters and composite image data. The first and second digital human software independently access their locally stored digital human asset files to drive the rendering of the digital human models. These digital human asset files include the mesh data, texture maps, and skeletal rigging data of the digital human models. During the entire joint performance, the digital human model mesh, texture maps, and skeletal rigging data are not uploaded or exchanged. The server only receives and processes the two-dimensional pixel images rendered from each device, and performs keying and compositing based on these two-dimensional pixel images to obtain the composite image of multiple people performing together. Therefore, each developer's digital human assets remain locally on their respective devices, fundamentally avoiding the risk of leakage of digital human assets during the collaboration process.

[0044] Example 2 This embodiment provides a system that enables multiple users to connect online and view the same screen simultaneously using digital humans from different software. Figure 2 This is a schematic diagram of a system provided by an embodiment of the present invention, illustrating how digital humans using different software can achieve multi-user online interaction and simultaneous viewing. For example... Figure 2 As shown, the system includes a host terminal, at least one guest terminal, and a server. The host terminal and each guest terminal are connected to the server via a network.

[0045] The homeowner's end is equipped with the first digital human software provided by the first developer, which includes a camera synchronization plugin; each tenant's end is equipped with a second digital human software provided by a different developer, which also includes a camera synchronization plugin.

[0046] The camera synchronization plugin on the host's end is used to obtain virtual camera parameters and send them in real time to the camera synchronization plugins on each guest's end via the network, so that each end obtains synchronized virtual camera data. The host's end is also used to adjust the virtual camera in the first digital human software based on the synchronized virtual camera data, move the first digital human model into the shooting range of the virtual camera, render and generate the base image, and upload the base image to the server in real time.

[0047] Each guest's end is used to adjust the virtual camera in the second digital human software to the same perspective as the host's end based on synchronized virtual camera data, move the second digital human model into the shooting range of the virtual camera, render and generate a guest performance scene with a background that can be removed, and upload the guest performance scene to the server in real time.

[0048] The server is used to extract the foreground image of the second digital human based on the received performance footage of the guests; based on the foreground image of the second digital human and the background image, the foreground image of the second digital human is superimposed and synthesized into the background image to generate a composite image of multiple people on stage; the composite image of multiple people on stage is sent to the host's end and each guest's end in real time; the server is also used to compress and encode the composite image of multiple people on stage in real time, and push the video stream to the content distribution entry of the target live streaming platform through the deployed push streaming module.

[0049] In this embodiment, the specific working process of the host, each guest and the server is the same as the implementation process described in steps S101 to S108 in embodiment one, and will not be repeated here.

[0050] As can be seen from the above embodiments, this invention, by installing a camera synchronization plugin in the digital human software on each end, replaces the migration and exchange of digital human assets with real-time synchronization of virtual camera parameters. This allows digital human software from different developers to achieve multi-person online and multi-person simultaneous screens without uniformly importing into the same rendering engine. It eliminates the tedious rebinding work such as skeletal redirection, animation state machine reconstruction, expression-driven remapping, and rendering pipeline adaptation, significantly improving the efficiency of simultaneous performances. The digital human asset files on each end are always kept locally, without being uploaded or exchanged, effectively protecting the digital assets of each developer and increasing the willingness of different companies to participate in simultaneous performances. With synchronized and consistent virtual camera data, the perspective relationship and viewing angle of digital humans in different software are ensured to be reasonably consistent in the composite image, presenting a natural and harmonious simultaneous image. The server performs real-time keying and compositing, and sends the composite image back to each end or directly pushes it to the live streaming platform, enabling all participants to obtain low-latency multi-person online simultaneous images in real time, meeting the needs of interactive performances and live streaming.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0052] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for enabling multiple users to connect online and view the same screen simultaneously using digital humans from different software, characterized in that... include: Install a camera synchronization plugin in the first digital human software on the homeowner's side, and install a camera synchronization plugin in at least one second digital human software on the guest's side. The first digital human software and the second digital human software are digital human driving software provided by different developers. The virtual camera parameters are obtained from the camera synchronization plugin on the host's side and sent to the camera synchronization plugins on each guest's side in real time via the network, so as to obtain virtual camera data that is synchronized and consistent on each side. On the homeowner's end, based on the synchronized virtual camera data, the virtual camera in the first digital human software is adjusted, and the first digital human model is moved into the shooting range of the virtual camera, rendering and generating the base image, and uploading the base image to the server in real time; On each guest's end, based on synchronized virtual camera data, the virtual camera in the second digital human software is adjusted to the same perspective as the host's end, and the second digital human model is moved into the shooting range of the virtual camera to render and generate guest performance footage with background removal capabilities, and the guest performance footage is uploaded to the server in real time. On the server, the foreground image of the second digital human is extracted based on the received footage of the guest's performance; Based on the foreground image of the second digital human and the background image, the foreground image of the second digital human is superimposed and composited onto the background image to generate a composite image of multiple people on the same stage. The composite image of multiple people on the same stage is sent to the host's and each guest's terminals in real time.

2. The method according to claim 1, characterized in that, The background that can be removed is a solid color background of a predetermined color; the extraction of the foreground image of the second digital human includes: In the server, based on the received performance footage of the guests, the distance between the color value of each pixel and the predetermined color in the color space is calculated; Based on the distance, a corresponding transparency coefficient is generated for each pixel; the smaller the distance, the higher the transparency coefficient. Based on the transparency coefficient, the pixel areas in the guest performance image that are close to the predetermined color are set to full transparency, while the remaining pixel areas retain their original color and opacity, thus obtaining a second digital human foreground image containing a transparency channel.

3. The method according to claim 2, characterized in that, The rendering generates a scene of guests performing with a background that can be removed, including: In the second digital human software, a command to start the same-platform mode is received; According to the instruction, the 3D virtual scene rendering stage in the second digital human software rendering pipeline is disabled, and the color clear value of the frame buffer is set to the predetermined color; According to the configured rendering pipeline, only the second digital human model is rendered to the frame buffer, and the remaining areas are automatically filled by the clear value, resulting in the guest performance screen where the model area retains the original rendering color and the background area is the predetermined color.

4. The method according to claim 3, characterized in that, The virtual camera parameters include the virtual camera's position coordinates in three-dimensional space, rotation quaternion, and field of view. The virtual camera parameters are obtained from the camera synchronization plugin on the homeowner's end, including: The camera synchronization plugin on the homeowner's end captures the panning, rotation, and zoom operations applied by the user to the virtual camera via an input device. Based on the translation control and the current state of the virtual camera, the updated position coordinates are calculated; based on the rotation control and the current state, the updated rotation quaternion is calculated; based on the zoom control and the current state, the updated field of view is calculated. The updated position coordinates, rotation quaternions, and field of view are combined to form the virtual camera parameters.

5. The method according to claim 4, characterized in that, The rendered base image includes: In the first digital human software on the homeowner's end, a preset three-dimensional virtual scene is loaded; Based on the synchronized virtual camera data, a viewing frustum corresponding to the virtual camera is constructed. Based on the viewing frustum, visibility clipping and rendering are performed on the three-dimensional virtual scene, and the first digital human model is placed at a predetermined position in the three-dimensional virtual scene to render the base image containing the first digital human model and the three-dimensional virtual background.

6. The method according to claim 5, characterized in that, The step of overlaying and synthesizing the second digital human foreground image onto the base image includes: In the server, based on the synchronized virtual camera data and the preset position information corresponding to each guest's terminal, the three-dimensional spatial coordinates of each second digital human model in the virtual scene of the base image are calculated. Based on the three-dimensional spatial coordinates and the virtual camera position in the virtual lens data, the vertical distance from each second digital human model to the virtual camera imaging plane is calculated, and the depth value corresponding to the foreground image of each second digital human is obtained. Based on the depth values, the multiple foreground images of the second digital human are sorted to obtain a superposition order from far to near; Following the stacking order, starting from the farthest second digital human foreground image, the images are sequentially overlaid onto the base image using a transparency blending method to obtain the composite image of multiple people on the same stage with correct front and back occlusion relationships.

7. The method according to claim 6, characterized in that, Also includes: In the server, based on the composite image of multiple people on the same stage, a hardware or software video encoder is invoked to perform real-time compression to obtain a video stream that conforms to the network transmission format; Based on the preset target live streaming platform push address, the video stream is pushed to the content distribution entry of the target live streaming platform through the push module deployed on the server.

8. The method according to claim 7, characterized in that, Sending the composite video of multiple people on the same stage to the homeowner's terminal and each of the guests' terminals in real time includes: The server acquires network status information periodically reported by the homeowner and each of the tenants. Based on the network status information of each end, the target bitrate and resolution for transcoding of each end are determined respectively; Based on the multi-person simultaneous composite image and the target bitrate and resolution corresponding to each terminal, real-time transcoding is performed to generate multi-channel adapted streaming media data; The multi-channel streaming data is pushed to the corresponding homeowner and guest terminals through separately established transmission channels, and each terminal decodes and displays a composite image of multiple people on the same stage.

9. The method according to claim 8, characterized in that, During the process of achieving simultaneous display, the data exchanged between the homeowner's terminal and each of the guests' terminals via the network includes virtual camera parameters and composite image data; The first digital human software and the second digital human software each independently call the digital human asset files stored locally to drive the rendering of the digital human model, without uploading or exchanging digital human model mesh, texture mapping, or skeleton binding data; The server only receives and processes the two-dimensional pixel images rendered from each end, and completes the compositing based on the two-dimensional pixel images to obtain a composite image of multiple people on the same stage.

10. A system for enabling multiple users to connect online and view the same screen simultaneously using digital humans from different software, wherein the system implements the method as described in any one of claims 1 to 9, characterized in that... include: On the homeowner's end, the First Digital Human software and camera synchronization plugin are deployed; At least one guest-side device is equipped with a second digital human software and a camera synchronization plugin. The first and second digital human software are digital human driver software provided by different developers. The server communicates and connects with the landlord and each tenant via the network. The camera synchronization plugin on the host's side is used to obtain virtual camera parameters and send them to the camera synchronization plugins on each guest's side in real time, so that each side obtains synchronized and consistent virtual camera data. The homeowner's side is used to adjust the virtual camera in the first digital human software based on synchronized virtual camera data, move the first digital human model into the shooting range, render and generate the base image, and upload the base image to the server in real time; Each guest's end is used to adjust the virtual camera in the second digital human software to the same perspective as the host's end based on synchronized virtual camera data, move the second digital human model into the shooting range, render and generate guest performance footage with background removal capability, and upload the guest performance footage to the server in real time. The server is used to extract the foreground image of the second digital human from the performance footage of the guests, synthesize the foreground image of the second digital human with the background image to generate a composite image of multiple people on stage, and distribute the composite image of multiple people on stage to the host and each guest in real time.

Images