Multi-pixel stream rendering method and electronic equipment

Abstract

The invention provides a multi-pixel stream rendering method and electronic device.The method comprises the steps that client equipment determines instances bound to viewports on the client equipment according to a configuration instruction of a user, the client equipment generates a pixel stream obtaining request according to a monitored viewport operation event, and the pixel stream obtaining request is sent to the client equipment; the pixel stream acquisition request is sent to the back-end device, the back-end device sends the parameter information to the target instance corresponding to the target viewport according to the identifier of the target viewport in the pixel stream acquisition request, and the target instance renders the target image corresponding to the target viewport according to the parameter information. And the target instance sends the target image to the target viewport for display through the pixel stream. The rendering efficiency of the multi-instance pixel stream is improved, and the stability and maintainability of the rendering system can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multi-pixel stream rendering method and an electronic device. Background Art

[0002] The digital twin system can be presented through three-dimensional visualization scenes and data statistical charts. The user's interaction with the three-dimensional scene is usually achieved through mouse and keyboard control to move and observe within the scene. This method only supports detection and observation from one perspective at the same time.

[0003] In order to achieve multi-perspective observation or multi-viewport display, the current Unreal Engine local client multi-viewport rendering creates multiple instance objects, renders multiple viewports or camera perspectives through multiple instance objects in one application, and then renders each viewport separately inside the client device, thereby achieving multi-viewport rendering.

[0004] However, when rendering a more complex scene, there is a problem of low rendering efficiency when the client device renders each viewport separately. Summary of the invention

[0005] The purpose of the present application is to provide a multi-pixel stream rendering method and an electronic device to address the deficiencies in the above-mentioned prior art, so as to solve the problem of low rendering efficiency when a client device renders each viewport separately in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0007] In a first aspect, the present application provides a multi-pixel stream rendering method, the method comprising:

[0008] Applied to a multi-pixel stream rendering system, the multi-pixel stream rendering system includes: at least one client device and a backend device, the client device includes multiple viewports, and the backend device is deployed with multiple instances; the method includes:

[0009] The client device determines, according to a configuration instruction of a user, an instance of binding each viewport on the client device;

[0010] The client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the backend device, wherein the pixel stream acquisition request includes: an identifier of a target viewport and parameter information, wherein the parameter information includes: an identifier of the viewport operation event or rendering parameters;

[0011] The backend device sends the parameter information to a target instance corresponding to the target viewport according to the identifier of the target viewport in the pixel stream acquisition request;

[0012] The target instance renders a target image corresponding to the target viewport according to the parameter information;

[0013] The target instance sends the target image to the target viewport via a pixel stream for display.

[0014] Optionally, the rendering, by the target instance according to the parameter information, of a target image corresponding to the target viewport includes:

[0015] If the parameter information is an identifier of a viewport operation event, the target instance determines the rendering parameter according to the identifier of the viewport operation event;

[0016] The target instance renders a target image corresponding to the target viewport according to the rendering parameters.

[0017] Optionally, the rendering, by the target instance according to the parameter information, of a target image corresponding to the target viewport includes:

[0018] If the parameter information is a rendering parameter, the target instance renders the target image corresponding to the target viewport according to the rendering parameter.

[0019] Optionally, the target instance sending the target image to the target viewport through a pixel stream for display includes:

[0020] If there are multiple viewports bound to the target instance, the target instance converts the target image into multiple target pixel streams according to the number of bound viewports, and sends each target pixel stream to each viewport bound to the target instance for display.

[0021] Optionally, after sending the pixel stream acquisition request to the backend device, the method further includes:

[0022] The backend device determines the associated instance according to the identifier of the target viewport and the parameter information, and sends the parameter information to each associated instance;

[0023] The associated instances render the image to be displayed according to the parameter information;

[0024] The associated instance sends the image to be displayed to the associated viewport bound to the associated instance through a pixel stream for display.

[0025] Optionally, the backend device determines the associated instance according to the identifier of the target viewport and the parameter information, including:

[0026] The backend device determines, according to the identifier of the target viewport, an associated viewport associated with the target viewport;

[0027] The backend device determines whether the associated viewport needs to be rendered synchronously according to the parameter information;

[0028] If so, the instance bound to the associated viewport is used as the associated instance.

[0029] Optionally, the multi-pixel stream rendering system further includes: an intermediate proxy server;

[0030] The client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the backend device, including:

[0031] The client device generates a pixel stream acquisition request according to the monitored event, and sends the pixel stream acquisition request to the intermediate proxy server;

[0032] The intermediate proxy server determines whether the instance corresponding to the pixel stream acquisition request is online;

[0033] If the instance is online and the client device has permission to access the instance, the intermediate proxy server sends the pixel stream acquisition request to the backend device.

[0034] Optionally, the intermediate proxy server determines whether the instance corresponding to the pixel stream acquisition request is online, including:

[0035] The intermediate proxy server determines, based on the user information in the pixel stream acquisition request, an instance to establish a connection with the user indicated by the user information, and obtains the running status of the instance. If the running status of the instance is normal operation, it is determined that the instance is online.

[0036] Optionally, the target instance sending the target image to the target viewport through a pixel stream for display includes:

[0037] The backend device sends the pixel stream of each target image to the intermediate proxy server;

[0038] The intermediate proxy server determines a client device corresponding to the target image, and sends a pixel stream of the target image to the client device.

[0039] In a second aspect, the present application provides a multi-pixel stream rendering system, the system comprising: at least one client device and a backend device, the client device comprising multiple viewports, and the backend device having multiple instances deployed thereon;

[0040] The client device is used to determine the instance of each viewport binding on the client device according to the user's configuration instruction;

[0041] The client device is used to generate a pixel stream acquisition request according to the monitored viewport operation event, and send the pixel stream acquisition request to the back-end device, wherein the pixel stream acquisition request includes: an identifier of a target viewport and parameter information, and the parameter information includes: an identifier or rendering parameters of the viewport operation event;

[0042] The back-end device is used to send the parameter information to the target instance corresponding to the target viewport according to the identifier of the target viewport in the pixel stream acquisition request; the target instance renders the target image corresponding to the target viewport according to the parameter information; and the target instance sends the target image to the target viewport via the pixel stream for display.

[0043] Optionally, the target instance is also used for:

[0044] If the parameter information is an identifier of a viewport operation event, the target instance determines the rendering parameter according to the identifier of the viewport operation event;

[0045] The target instance renders a target image corresponding to the target viewport according to the rendering parameters.

[0046] Optionally, the target instance is also used for:

[0047] If the parameter information is a rendering parameter, the target instance renders the target image corresponding to the target viewport according to the rendering parameter.

[0048] Optionally, the target instance is also used to:

[0049] If there are multiple viewports bound to the target instance, the target instance converts the target image into multiple target pixel streams according to the number of bound viewports, and sends each target pixel stream to each viewport bound to the target instance for display.

[0050] Optionally, the backend device is further used for:

[0051] According to the identifier of the target viewport and the parameter information, an associated instance is determined, and the parameter information is sent to each of the associated instances; each of the associated instances renders an image to be displayed according to the parameter information; and the associated instance sends the image to be displayed to the associated viewport bound to the associated instance via a pixel stream for display.

[0052] Optionally, the backend device is further used for:

[0053] Determining, according to the identifier of the target viewport, an associated viewport associated with the target viewport;

[0054] Determining whether the associated viewport needs to be rendered synchronously according to the parameter information;

[0055] If so, the instance bound to the associated viewport is used as the associated instance.

[0056] Optionally, the multi-pixel stream rendering system further includes: an intermediate proxy server;

[0057] The client device is used to generate a pixel stream acquisition request according to the monitored viewport operation event, and send the pixel stream acquisition request to the back-end device, including:

[0058] The client device is used to generate a pixel stream acquisition request according to the monitored event, and send the pixel stream acquisition request to the intermediate proxy server;

[0059] The intermediate proxy server is used to determine whether the instance corresponding to the pixel stream acquisition request is online;

[0060] If the instance is online and the client device has permission to access the instance, the intermediate proxy server is further configured to send the pixel stream acquisition request to the backend device.

[0061] Optionally, the intermediate proxy server is used to:

[0062] According to the user information in the pixel stream acquisition request, an instance connected to the user indicated by the user information is determined, and the running status of the instance is obtained. If the running status of the instance is normal operation, it is determined that the instance is online.

[0063] Optionally, the backend device is used to send the pixel stream of each target image to the intermediate proxy server;

[0064] The intermediate proxy server is used to determine the client device corresponding to the target image, and send the pixel stream of the target image to the client device.

[0065] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the multi-pixel stream rendering method as described in any one of the first aspects.

[0066] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-pixel stream rendering method as described in any one of the first aspects are executed.

[0067] The beneficial effects of the present application are: by implementing rendering, data processing, and front-end display on the front-end client device and the back-end device respectively, it is possible to achieve flexible deployment and efficient management of multi-instance pixel streams, make full use of distributed computing resources, and reduce the load on a single server. The front-end client device can realize the control and display of multiple video streams, the back-end device can uniformly process the data from each instance, and realize split-screen interaction through event monitoring, thereby realizing effective management and synchronization of multi-instance data, and ensuring the real-time and accuracy of user interaction. In addition, for multi-perspective presentation and interaction in complex scenes, the present application can be rendered by the back-end device and displayed by the front-end client device, reducing the risk of single point failure and improving the stability and maintainability of the system.

[0068] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 A schematic diagram of the architecture of a multi-pixel stream rendering system provided by an embodiment of the present application is shown;

[0071] Figure 2 A flowchart of a multi-pixel stream rendering method provided by an embodiment of the present application is shown;

[0072] Figure 3 An example diagram of an interface for multi-view viewport control provided by an embodiment of the present application is shown;

[0073] Figure 4 An example diagram of an interface for comparing multiple viewport effects provided by an embodiment of the present application is shown;

[0074] Figure 5 A flow chart for determining a target image provided by an embodiment of the present application is shown;

[0075] Figure 6A flowchart of rendering and displaying an associated instance provided by an embodiment of the present application is shown;

[0076] Figure 7 A flowchart of determining an associated instance provided by an embodiment of the present application is shown;

[0077] Figure 8 A flowchart of request forwarding by an intermediate proxy server provided in an embodiment of the present application is shown;

[0078] Fig. 9 A flowchart of pixel stream forwarding by an intermediate proxy server provided in an embodiment of the present application is shown;

[0079] Fig.10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0080] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0081] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0082] Digital twins enable feedback from real physical systems to digital models in space, greatly reducing the cost and risk of testing in the real physical world. Testing and optimization in virtual space can significantly reduce the cost of actual operations and avoid losses caused by trial and error.

[0083] In the digital twin scene, in order to achieve multi-perspective observation or multi-viewport display, the current Unreal Engine local client multi-viewport rendering creates multiple instance objects, renders multiple viewports or camera perspectives through multiple instance objects in one application, and then renders each viewport separately inside the client device, thereby achieving multi-viewport rendering.

[0084] However, when the scene is more complex, rendering the viewport by each client device individually has the problem of low rendering efficiency.

[0085] Based on this, the present application proposes a multi-pixel stream rendering method, which can be applied to a multi-pixel stream rendering system, such as Figure 1 , which is a schematic diagram of the architecture of a multi-pixel stream rendering system provided by the present application. Figure 1 The system includes a front-end display layer, an intermediate layer, and a service layer. The front-end display layer includes at least one client device with a front-end browser deployed. The intermediate layer can be a Web server for pixel stream reverse proxy and load balancing. The service layer includes back-end devices. The back-end device can be a server, cloud, or electronic device with a back-end data processing service deployed. The back-end device can also be a distributed server cluster, including multiple servers for processing rendering.

[0086] The client device includes multiple viewports. Taking the digital twin scene as an example, the viewport can be used to display the rendered image from a certain perspective in the virtual scene. Multiple instances are deployed on the backend device. The instance can be a UE5 instance, which is used to calculate and render the image from a certain perspective in the virtual scene. Each instance can be bound to at least one viewport, and the viewport bound to the instance can display the image rendered by the instance.

[0087] Next, combine Figure 2 , the multi-pixel stream rendering method of the present application is described, and the method can be applied to Figure 1 In the multi-pixel stream rendering system shown in FIG. Figure 2 , the method comprising:

[0088] S201. The client device determines the instance to which each viewport on the client device is bound according to the configuration instruction of the user.

[0089] The user's configuration instruction may be an instruction for binding an instance to a viewport. The client device may provide a configuration interface through a front-end browser, and the configuration interface includes a plurality of selectable instances, each of which is used to render an image under a viewing angle in a virtual scene. The user may select an instance to be bound for each viewport in the configuration interface, thereby achieving binding of the viewport of the client device to the instance of the back-end device.

[0090] Among them, multiple viewports can be displayed in split screen on the client device, and each viewport is used to display a rendered image of the virtual scene at a certain viewing angle, or to display a rendered image of the virtual scene at the same viewing angle in different rendering modes.

[0091] In the first example, Figure 3As shown, some digital twin scenarios require real-time monitoring of virtual scenes from multiple angles. At this time, the virtual scene can be used as a project to be rendered. The same project can be rendered using the same rendering parameters. The project includes multiple instances, each of which is used to describe the rendered image of the virtual scene at a certain perspective.

[0092] In the second example, Figure 4 As shown, different rendering methods are required for rendering virtual scenes. For example, when comparing different planning schemes for the same area from the same perspective, two instances can be used to render the same virtual scene. The two instances correspond to different rendering methods. When comparing the rendering results of the two rendering methods, the rendering results of the two instances can be displayed on the display interface of the client device in split screens. Figure 4 Viewport 5 and viewport 6 are the rendering results of the same viewpoint and the same area under different lighting effects.

[0093] S202. The client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the backend device. The pixel stream acquisition request includes: the identifier of the target viewport and parameter information. The parameter information includes: the identifier of the viewport operation event or rendering parameters.

[0094] The client device includes an event listener for listening to viewport operation events initiated by the user on the client device. The viewport operation event can be a trigger event for the viewport, which can trigger the image in the viewport displayed on the client device to be updated, such as camera positioning events, weather events, scene switching events, traffic simulation events, and building vertical layering events.

[0095] Taking weather events as an example, the user can change the weather displayed in viewport 1 on the client device, such as changing the current weather from sunny to rainy. The event listener can generate a pixel stream acquisition request based on the monitored weather event and send the pixel stream acquisition request to the backend device. The pixel stream acquisition request is used to instruct the instance of the relevant viewport in the backend device to re-render the content.

[0096] The viewport and the instance establish a binding relationship in the above step S201 , so after receiving the pixel stream acquisition request, the backend device can determine the instance bound to the target viewport based on the identifier of the target viewport.

[0097] The parameter information may be an identifier of the viewport operation event or a rendering parameter. In the first implementation, when the parameter information is an identifier of the viewport operation event, the backend device may determine the event type of the viewport operation event based on the identifier of the viewport operation event, and determine an instance and rendering parameters related to the viewport operation event.

[0098] In the second implementation method, when the client device monitors the viewport operation event, it can generate rendering parameters of the viewport operation event. The rendering parameters can be the parameters used by the instance when rendering content, including lighting parameters, material parameters, texture parameters, rendering resolution, etc.

[0099] S203: The backend device sends parameter information to a target instance corresponding to the target viewport according to the identifier of the target viewport in the pixel stream acquisition request.

[0100] Optionally, the pixel stream acquisition request may further include indication information for indicating that the viewport needs to be updated to the target viewport, or the target viewport and a viewport associated with the target viewport. The target instance corresponding to the target viewport may be an instance bound to the target viewport, or an instance bound to the target viewport and an instance bound to a viewport associated with the target viewport.

[0101] Optionally, when multiple viewports are displayed in split screen on the client device, the multiple viewports describe the rendered images of the same virtual scene from different perspectives. When the user views the virtual scene from multiple perspectives, the update of one of the viewports can be controlled to thereby control the synchronous update of related viewports.

[0102] For example, the user Figure 3 If the weather in viewport 1 shown in the figure is changed, the weather in viewports 2, 3 and 4 can be changed at the same time. At this time, the identifier of the target viewport is the identifier of viewport 1, and the instances bound to the target viewport may include not only the instances bound to viewport 1, but also the instances bound to viewports 2, 3 and 4 associated with viewport 1.

[0103] In a multi-view split-screen viewing scenario, instances of the same project have the same project ID, for example Figure 3 The viewport 1, viewport 2, viewport 3 and viewport 4 shown belong to the same project and have the same project identifier. The backend device can save the identifier of the viewport and the identifier of the project corresponding to the viewport. After receiving the pixel stream acquisition request, the backend device can determine the project to which the target viewport belongs according to the identifier of the target viewport, and send the parameter information to all instances corresponding to the project, thereby updating the image under all viewing angles.

[0104] In another possible implementation, the pixel stream acquisition request received by the backend device may be for updating only one viewport. In this case, the backend device may send parameter information to a target instance bound to the target viewport according to the identifier of the target viewport.

[0105] For example, users Figure 3A separate zoom operation is performed on viewport 1 in the image. At this time, the indication information of the pixel stream acquisition request can indicate that only the image of viewport 1 is updated, and the back-end device can send the parameter information to the instance bound to viewport 1 according to the identifier of the target viewport.

[0106] S204: The target instance renders a target image corresponding to the target viewport according to the parameter information.

[0107] The parameter information may indicate rendering parameters required for rendering the target instance, and the target instance may run based on the rendering parameters indicated by the parameter information to render a target image corresponding to the target viewport.

[0108] The target image corresponding to the target viewport includes an image rendered by an instance bound to the target viewport, or an image rendered by an instance bound to the target viewport and an image rendered by an instance bound to a viewport associated with the target viewport.

[0109] S205: The target instance sends the target image to the target viewport via a pixel stream for display.

[0110] After the target instance completes rendering, the rendered target image can be sent to the corresponding target viewport in the form of a pixel stream for display. Pixel stream is a viewport output data transmission method. The result of viewport rendering, that is, the target image, can be transmitted to the client device in the form of a pixel stream, thereby providing low-latency rendering image transmission.

[0111] It should be noted that the target instance may be bound to multiple viewports, for example, multiple users view the content rendered by the target instance through different client devices. When the application scenario is multi-person synchronous collaboration, in order to ensure the consistency of the content rendered and displayed by multiple viewports, the target instance can send the target image in the form of a pixel stream to the viewports of all client devices bound to the target instance after the rendering is completed. When the application scenario is that each viewport runs independently and does not affect each other, the target instance can send the target image in the form of a pixel stream only to the target viewport of the client device that initiated the pixel stream acquisition request after the rendering is completed.

[0112] In an embodiment of the present application, the client device determines the instances bound to each viewport on the client device according to the user's configuration instructions, the client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the back-end device, the pixel stream acquisition request includes: the identification of the target viewport and parameter information, the parameter information includes: the identification of the viewport operation event or rendering parameters, the back-end device sends the parameter information to the target instance corresponding to the target viewport according to the identification of the target viewport in the pixel stream acquisition request, the target instance renders the target image corresponding to the target viewport according to the parameter information, and the target instance sends the target image to the target viewport via the pixel stream for display.

[0113] By implementing rendering, data processing, and front-end display on the front-end device and the back-end device respectively, it is possible to achieve flexible deployment and efficient management of multi-instance pixel streams, make full use of distributed computing resources, and reduce the load on a single server. The front-end client device can control and display multiple video streams, the back-end device can uniformly process data from each instance, and realize split-screen interaction through event monitoring, thereby realizing effective management and synchronization of multi-instance data and ensuring the real-time and accuracy of user interaction. In addition, for multi-perspective presentation and interaction in complex scenes, this application can be rendered by the back-end device and displayed by the front-end client device, reducing the risk of single point failure and improving the stability and maintainability of the system.

[0114] The following is a further description of the target image corresponding to the target viewport rendered by the target instance according to the parameter information. Figure 5 As shown, the above step S204 includes:

[0115] S501: If the parameter information is an identifier of a viewport operation event, the target instance determines rendering parameters according to the identifier of the viewport operation event.

[0116] As a possible implementation, the correspondence between the viewport operation event and the rendering parameters can be pre-stored in the backend device. It should be noted that for some events, the rendering parameters are fixed, such as weather events. The rendering parameters of different weather conditions can be pre-configured, so that after the weather event is triggered, the rendering parameters corresponding to the event can be determined based on the identifier of the viewport operation event.

[0117] S502: The target instance renders a target image corresponding to the target viewport according to the rendering parameters.

[0118] The target instance can render the virtual scene according to the lighting parameters, material parameters, texture parameters, and rendering resolution indicated by the rendering parameters to obtain a target image corresponding to the target viewport.

[0119] In the second implementation, the step of rendering the target image corresponding to the target viewport by the target instance according to the parameter information includes:

[0120] If the parameter information is a rendering parameter, the target instance renders the target image corresponding to the target viewport according to the rendering parameter.

[0121] The rendering parameters of some viewport operation events need to be determined based on the user's current operation. For example, if the user zooms in, rotates, or shrinks the viewport image, or if the user changes the lighting conditions of the current viewport image, the viewport operation event can include rendering parameters, and the rendering parameters can be generated based on the parameters adjusted by the user.

[0122] For example, suppose the user Figure 3 The viewport 1 shown has been zoomed in, and the viewport operation event may include the rendering resolution after the user zooms in, and the target instance may re-render according to the rendering resolution in the rendering parameters to obtain the target image corresponding to the target viewport.

[0123] In the scenario of multi-device collaborative rendering, the present application can realize synchronous update rendering of multiple client devices. The following is a further description of the target instance sending the target image to the target viewport through the pixel stream for display, and the above S205 step includes:

[0124] If there are multiple viewports bound to the target instance, the target instance converts the target image into multiple target pixel streams according to the number of bound viewports, and sends each target pixel stream to each viewport bound to the target instance for display.

[0125] After the target instance renders the target image, if the target instance is bound to N viewports, the target instance can copy the target image into N target pixel streams, and send the target pixel streams to the N viewports bound to the target instance for display.

[0126] It should be noted that, in another implementation, if images on multiple client devices do not need to be synchronized, the target instance can send the pixel stream of the target image to the client device that initiates the pixel stream acquisition request, so that the client device can receive the target pixel stream and display it in the viewport on the client device.

[0127] The following is an explanation of the steps after sending the pixel stream acquisition request to the backend device. Figure 2 , multiple viewports displayed in split screen on the client device can be controlled synchronously, such as Figure 6 As shown, the specific steps include:

[0128] S601: The backend device determines the associated instance according to the identifier of the target viewport and the parameter information, and sends the parameter information to each associated instance.

[0129] The backend device stores the identifier of the target viewport and the association relationship of the viewports. The backend device also stores the identifier of the viewport operation event that can cause the association relationship.

[0130] The backend device can determine the associated viewports that are associated with the target viewport based on the identifier of the target viewport, and determine whether the viewport operation event triggers a synchronous update of multiple viewports based on the identifier of the viewport operation event in the parameter information. If so, the parameter information is sent to each associated instance.

[0131] For example, Figure 3For viewport 1 shown, when the back-end device receives a pixel stream acquisition request generated by the user after performing a traffic simulation event on viewport 1, it can determine whether the update is for a single viewport or multiple viewports based on the indication information in the pixel stream acquisition request, or determine that the traffic simulation event is an event that will cause multiple viewports to be updated synchronously based on the viewport operation event identifier in the pixel stream acquisition request. The back-end device can then determine viewport 2, viewport 3, and viewport 4 associated with viewport 1 based on the viewport identifier of viewport 1, take the instances of viewport 2, viewport 3, and viewport 4 as associated instances, and send parameter information to each associated instance.

[0132] S602: Each associated instance renders an image to be displayed according to the parameter information.

[0133] Each associated instance can be rendered according to the rendering parameters in the parameter information to obtain the rendering results of each instance. The image to be displayed can be the rendering result obtained after the instance is run.

[0134] S603: The associated instance sends the image to be displayed to the associated viewport bound to the associated instance through a pixel stream for display.

[0135] Each associated instance is bound to an associated viewport, and the associated instance can transmit the image to be displayed in a pixel stream to the client device where the associated viewport bound to the associated instance is located, and the client device displays the image on the associated viewport.

[0136] The following is a further description of how the above backend device determines the associated instance based on the target viewport identifier and parameter information. Figure 7 As shown, the above step S601 includes:

[0137] S701: The backend device determines an associated viewport associated with the target viewport according to an identifier of the target viewport.

[0138] The backend device stores the association relationship of the target viewport, and an association relationship can be established for the viewports that need to be controlled synchronously. After the backend device receives the pixel stream acquisition request, it can determine at least one associated viewport that has an association relationship with the target viewport based on the identifier of the target viewport in the pixel stream acquisition request.

[0139] S702: The backend device determines whether the associated viewport needs to be rendered synchronously based on the parameter information.

[0140] The parameter information includes the identifier of the viewport operation event. Some viewport operation events can realize synchronous control of multiple viewports. The backend device can determine whether the associated viewport needs to be rendered synchronously based on the identifier of the viewport operation event.

[0141] S703: If yes, the instance bound to the associated viewport is used as the associated instance.

[0142] If the viewport operation event is an event that triggers the synchronous update of multiple viewports, all instances bound to the associated viewports can be used as associated instances, and each associated instance performs rendering according to the parameter information of the pixel stream acquisition request to obtain an image to be displayed rendered by each associated instance.

[0143] Reference Figure 1 ,The multi-pixel stream rendering system also includes an intermediate server, which can be a Web server, which is used to reverse proxy all pixel streams, implement multi-stream routing, integrate and schedule user requests and pixel streams, and allocate corresponding pixel streams or viewports according to user requests.

[0144] The client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the back-end device, such as Figure 8 As shown, including:

[0145] S801. The client device generates a pixel stream acquisition request according to a monitored event, and sends the pixel stream acquisition request to an intermediate proxy server.

[0146] After the event listener in the client device listens to the trigger event, the client device can generate a pixel stream acquisition request based on the listened event, the client device identifier, the viewport identifier and the user's account information, and send the pixel stream acquisition request to the intermediate proxy server.

[0147] S802: The intermediate proxy server determines whether the instance corresponding to the pixel stream acquisition request is online.

[0148] The instance corresponding to the pixel stream acquisition request may be an instance bound to a viewport pre-configured by a user logged in to the client device. The intermediate proxy server may determine the instance bound to the viewport that the user triggers to be updated in the client device according to the pixel stream acquisition request, and obtain the running status of the instance.

[0149] In a possible implementation, the backend device may periodically update the status of each instance to the intermediate proxy server, or the intermediate proxy server may obtain the running status of the instance from the backend device.

[0150] If the running status of the instance is running normally, it can be considered that the instance is online. If the running status of the instance is running abnormally, it can be considered that the instance is offline.

[0151] S803: If the instance is online and the client device has permission to access the instance, the intermediate proxy server sends a pixel stream acquisition request to the backend device.

[0152] If the instance runs normally, and the user account information of the client device indicates that the client device of the current request has established a binding relationship with the instance and has the authority to access the instance, the intermediate proxy server can send the pixel stream acquisition request to the backend device.

[0153] Furthermore, the step of the intermediate proxy server determining whether the instance corresponding to the pixel stream acquisition request is online includes:

[0154] The intermediate proxy server obtains the user information in the pixel stream request, determines the instance to establish a connection with the user indicated by the user information, and obtains the running status of the instance. If the running status of the instance is normal operation, it is determined that the instance is online.

[0155] Referring to the above step S201, the user can bind the viewport and the instance on the client device, and when binding the viewport and the instance, the user information of the bound viewport and the instance can also be recorded. The intermediate proxy server can save the user information and the bound viewport and the instance corresponding to the user information. When receiving the pixel stream acquisition request, the intermediate proxy server can determine the instance of the bound viewport corresponding to the user information according to the user information and the viewport identifier in the pixel stream acquisition request, and obtain the running status of the instance.

[0156] The following is a further description of the target instance sending the target image to the target viewport through the pixel stream for display. Fig. 9 As shown, including:

[0157] S901. The backend device sends the pixel stream of each target image to the intermediate proxy server.

[0158] S902: The intermediate proxy server determines a client device corresponding to the target image, and sends a pixel stream of the target image to the client device.

[0159] The backend device may send the pixel stream of the target image to the intermediate proxy server, and the intermediate proxy server may save the correspondence between the request, the pixel stream, and the viewport, and determine the client device requesting the target image based on the correspondence.

[0160] Based on the same inventive concept, the embodiment of the present application also provides a multi-pixel stream rendering system corresponding to the multi-pixel stream rendering method. Since the principle of solving the problem by the system in the embodiment of the present application is similar to the multi-pixel stream rendering method in the embodiment of the present application, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0161] Reference Figure 1 As shown in the system architecture diagram, the system includes: at least one client device and a backend device, the client device includes multiple viewports, and multiple instances are deployed on the backend device.

[0162] The client device is used to determine the instance bound to each viewport on the client device according to the user's configuration instruction;

[0163] The client device is used to generate a pixel stream acquisition request according to the monitored viewport operation event, and send the pixel stream acquisition request to the backend device, wherein the pixel stream acquisition request includes: the identifier of the target viewport and parameter information, and the parameter information includes: the identifier of the viewport operation event or the rendering parameter;

[0164] The backend device is used to send parameter information to the target instance corresponding to the target viewport according to the identifier of the target viewport in the pixel stream acquisition request; the target instance renders the target image corresponding to the target viewport according to the parameter information; and the target instance sends the target image to the target viewport via the pixel stream for display.

[0165] Optionally, the target instance is also used to:

[0166] If the parameter information is an identifier of a viewport operation event, the target instance determines the rendering parameters according to the identifier of the viewport operation event;

[0167] The target instance renders the target image corresponding to the target viewport according to the rendering parameters.

[0168] Optionally, the target instance is also used to:

[0169] If the parameter information is a rendering parameter, the target instance renders the target image corresponding to the target viewport according to the rendering parameter.

[0170] Optionally, the target instance is also used to:

[0171] If there are multiple viewports bound to the target instance, the target instance converts the target image into multiple target pixel streams according to the number of bound viewports, and sends each target pixel stream to each viewport bound to the target instance for display.

[0172] Optionally, the backend device is also used to:

[0173] According to the identification of the target viewport and the parameter information, the associated instance is determined, and the parameter information is sent to each associated instance; each associated instance renders the image to be displayed according to the parameter information; the associated instance sends the image to be displayed to the associated viewport bound to the associated instance through a pixel stream for display.

[0174] Optionally, the backend device is also used to:

[0175] According to the identifier of the target viewport, determining the associated viewport associated with the target viewport;

[0176] According to the parameter information, determine whether the associated viewport needs to be rendered synchronously;

[0177] If so, the instance bound to the associated viewport is used as the associated instance.

[0178] Optionally, the multi-pixel stream rendering system further includes: an intermediate proxy server;

[0179] The client device is used to generate a pixel stream acquisition request according to the monitored viewport operation event, and send the pixel stream acquisition request to the backend device, including:

[0180] The client device is used to generate a pixel stream acquisition request according to the monitored event, and send the pixel stream acquisition request to the intermediate proxy server;

[0181] The intermediate proxy server is used to determine whether the instance corresponding to the pixel stream acquisition request is online;

[0182] If the instance is online and the client device has permission to access the instance, the intermediate proxy server is also used to send the pixel stream acquisition request to the backend device.

[0183] Optionally, an intermediate proxy server is used to:

[0184] According to the user information in the pixel stream acquisition request, the instance connected to the user indicated by the user information is determined, and the running status of the instance is obtained. If the running status of the instance is normal operation, it is determined that the instance is online.

[0185] Optionally, the backend device is used to send the pixel stream of each target image to the intermediate proxy server;

[0186] The intermediate proxy server is used to determine the client device corresponding to the target image and send the pixel stream of the target image to the client device.

[0187] The embodiments of the present application implement rendering, data processing, and front-end display on the front-end device and the back-end device respectively, so as to realize flexible deployment and efficient management of multi-instance pixel streams, make full use of distributed computing resources, and reduce the load of a single server. The front-end client device can realize the control and display of multiple video streams, the back-end device can uniformly process the data from each instance, and realize split-screen interaction through event monitoring, thereby realizing effective management and synchronization of multi-instance data and ensuring the real-time and accuracy of user interaction. In addition, for multi-perspective presentation and interaction in complex scenes, the present application can be rendered by the back-end device and displayed by the front-end client device, reducing the risk of single point failure and improving the stability and maintainability of the system.

[0188] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. The electronic device may be Figure 1The client device, intermediate proxy server or back-end device in the embodiment, the electronic device includes: a processor 1001, a storage medium 1002 and a bus 1003, the storage medium 1002 stores machine-readable instructions executable by the processor 1001, when the electronic device runs a multi-pixel stream rendering method as in the embodiment, the processor 1001 communicates with the storage medium 1002 through the bus 1003, the processor 1001 executes the machine-readable instructions, the processor 1001 method item preamble part, to perform the steps of the above-mentioned multi-pixel stream rendering method.

[0189] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor executes the steps of the above-mentioned multi-pixel stream rendering method.

[0190] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0191] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0194] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0196] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multi-pixel stream rendering method, characterized in that: Applied to a multi-pixel stream rendering system, the multi-pixel stream rendering system comprises: at least one client device and a backend device, the client device comprises a plurality of viewports, and the backend device has a plurality of instances deployed; The method comprises: The client device determines, according to a configuration instruction of a user, an instance of binding each viewport on the client device; The client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the backend device, wherein the pixel stream acquisition request includes: an identifier of a target viewport and parameter information, wherein the parameter information includes: an identifier of the viewport operation event or rendering parameters; The backend device sends the parameter information to a target instance corresponding to the target viewport according to the identifier of the target viewport in the pixel stream acquisition request; The target instance renders a target image corresponding to the target viewport according to the parameter information; The target instance sends the target image to the target viewport via a pixel stream for display.

2. The method according to claim 1, characterized in that The step of rendering, by the target instance according to the parameter information, a target image corresponding to the target viewport includes: If the parameter information is an identifier of a viewport operation event, the target instance determines the rendering parameter according to the identifier of the viewport operation event; The target instance renders a target image corresponding to the target viewport according to the rendering parameters.

3. The method according to claim 1, characterized in that The step of rendering, by the target instance according to the parameter information, a target image corresponding to the target viewport includes: If the parameter information is a rendering parameter, the target instance renders the target image corresponding to the target viewport according to the rendering parameter.

4. The method according to claim 1, characterized in that: The target instance sending the target image to the target viewport through a pixel stream for display includes: If there are multiple viewports bound to the target instance, the target instance converts the target image into multiple target pixel streams according to the number of bound viewports, and sends each target pixel stream to each viewport bound to the target instance for display.

5. The method according to claim 1, characterized in that After sending the pixel stream acquisition request to the backend device, the method further includes: The backend device determines the associated instance according to the identifier of the target viewport and the parameter information, and sends the parameter information to each associated instance; The associated instances render the image to be displayed according to the parameter information; The associated instance sends the image to be displayed to the associated viewport bound to the associated instance through a pixel stream for display.

6. The method according to claim 5, characterized in that The backend device determines the associated instance according to the identifier of the target viewport and the parameter information, including: The backend device determines, according to the identifier of the target viewport, an associated viewport associated with the target viewport; The backend device determines whether the associated viewport needs to be rendered synchronously according to the parameter information; If so, the instance bound to the associated viewport is used as the associated instance.

7. The method according to claim 1, characterized in that The multi-pixel stream rendering system also includes: an intermediate proxy server; The client device generates a pixel stream acquisition request according to the monitored viewport operation event, and sends the pixel stream acquisition request to the backend device, including: The client device generates a pixel stream acquisition request according to the monitored event, and sends the pixel stream acquisition request to the intermediate proxy server; The intermediate proxy server determines whether the instance corresponding to the pixel stream acquisition request is online; If the instance is online and the client device has permission to access the instance, the intermediate proxy server sends the pixel stream acquisition request to the backend device.

8. The method according to claim 7, characterized in that The intermediate proxy server determines whether the instance corresponding to the pixel stream acquisition request is online, including: The intermediate proxy server determines, based on the user information in the pixel stream acquisition request, an instance to establish a connection with the user indicated by the user information, and obtains the running status of the instance. If the running status of the instance is normal operation, it is determined that the instance is online.

9. The method according to claim 7, characterized in that: The target instance sending the target image to the target viewport through a pixel stream for display includes: The backend device sends the pixel stream of each target image to the intermediate proxy server; The intermediate proxy server determines a client device corresponding to the target image, and sends a pixel stream of the target image to the client device.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the multi-pixel stream rendering method as described in any one of claims 1 to 9.

Images