Adaptive coding control method, device and equipment for multi-modal picture of remote surgery

Abstract

The application provides a remote surgery multi-modal picture adaptive encoding control method, device and equipment, real-time acquisition of a surgery video frame sequence in a picture-in-picture mode; wherein the surgery video frame sequence is generated by superimposition of a main picture and at least one type of auxiliary modal picture; real-time picture perception is performed on a current frame in the surgery video frame sequence, each independent picture area in the current frame is identified, and picture attribute information of each picture area is determined; wherein the picture attribute information includes a boundary box coordinate and a modal type; based on a current network available bandwidth and the picture attribute information, encoding control parameters corresponding to each picture area are determined; based on the encoding control parameters of each picture area, the current frame is differentially encoded, and an encoded video code stream is output; by differentially encoding and controlling each area in a multi-modal composite picture, the problem of inadaptation of an existing encoding mode to a composite picture is solved, and transmission fidelity of key clinical information and surgery safety are improved.

Description

Technical Field

[0001] This application relates to the field of remote surgical communication technology, and in particular to an adaptive coding control method, apparatus, and device for remote surgical multimodal images. Background Technology

[0002] Today, remote surgical robots perform minimally invasive surgeries by remotely controlling robotic arms, breaking geographical limitations and enabling high-quality medical resources to reach remote areas. In remote surgery, the stability and low-latency transmission of surgical images are directly related to the safety and accuracy of the surgical procedure.

[0003] Currently, due to limitations in network bandwidth and system deployment costs, existing remote surgical image transmissions mostly employ Region of Interest (ROI) adaptive coding schemes. These schemes perform localized differential coding only on a single endoscopic image to preserve the clarity of the core surgical area within limited bandwidth. However, this type of coding scheme cannot adapt to multimodal composite images in picture-in-picture mode, making it difficult to implement graded quality assurance for different modal regions within the image. This can easily lead to insufficient fidelity in the transmission of critical clinical information, posing a potential risk to the safe implementation of remote surgery. Summary of the Invention

[0004] This application provides an adaptive coding control method, apparatus, and device for multimodal images in remote surgery. By differentially coding and controlling each region in the multimodal composite image, it solves the problem of incompatibility between existing coding methods and composite images, thereby improving the transmission fidelity of key clinical information and surgical safety.

[0005] In a first aspect, embodiments of this application provide an adaptive encoding control method for multimodal images in remote surgery, the method comprising: Real-time acquisition of surgical video frame sequences in picture-in-picture mode; wherein, the surgical video frame sequence is generated by superimposing the main screen with at least one type of auxiliary modal screen; Real-time image perception is performed on the current frame in the surgical video frame sequence to identify each independent image region in the current frame and determine the image attribute information of each image region; wherein, the image attribute information includes bounding box coordinates and modal type; Based on the current available network bandwidth and the image attribute information, determine the encoding control parameters corresponding to each image region; Based on the encoding control parameters of each of the aforementioned screen regions, the current frame is differentially encoded, and the encoded video stream is output.

[0006] Secondly, embodiments of this application provide an adaptive encoding control device for multimodal images in remote surgery, the device comprising: The data acquisition module is used to acquire the surgical video frame sequence in picture-in-picture mode in real time; wherein, the surgical video frame sequence is generated by superimposing the main screen with at least one type of auxiliary modal screen; A multimodal region recognition module is used to perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region; wherein, the image attribute information includes bounding box coordinates and modal type; The encoding parameter determination module is used to determine the encoding control parameters corresponding to each of the screen regions based on the current available network bandwidth and the screen attribute information; The video differential encoding module is used to perform differential encoding on the current frame based on the encoding control parameters of each of the aforementioned screen regions, and output the encoded video bitstream.

[0007] Thirdly, embodiments of this application also provide an electronic device, which includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement an adaptive encoding control method for remote surgical multimodal images as described in any embodiment of this application.

[0008] This application provides an adaptive encoding control method, apparatus, and device for multimodal images in remote surgery. The method includes: acquiring a sequence of surgical video frames in picture-in-picture mode in real time, wherein the surgical video frame sequence is generated by superimposing a main image with at least one type of auxiliary modal image; further, performing real-time image perception on the current frame in the surgical video frame sequence, identifying each independent image region in the current frame, and determining the image attribute information of each image region, wherein the image attribute information includes bounding box coordinates and modal type; determining the encoding control parameters corresponding to each image region based on the current available network bandwidth and the image attribute information; and then, performing differentiated encoding on the current frame based on the encoding control parameters of each image region, and outputting the encoded video stream. The technical solution provided in this embodiment achieves independent encoding control of each region in the multimodal composite image by real-time sensing of the modal type and bounding box coordinates of each independent image region after picture-in-picture synthesis, and by determining differentiated encoding control parameters for each region based on the available network bandwidth. This solves the problem of mismatch between existing encoding methods and multimodal composite images, improves the transmission fidelity of key clinical information under limited bandwidth and the targeted allocation of encoding resources, thereby ensuring the operational safety and stability of remote surgery. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in describing the embodiments are briefly introduced below. Obviously, the accompanying drawings described are only a portion of the embodiments to be described in this application, and not all of them. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0010] Figure 1 This is a schematic diagram of the system architecture of the remote surgical robot provided in the embodiments of this application; Figure 2 This is a flowchart illustrating an adaptive encoding control method for multimodal images in remote surgery provided in an embodiment of this application. Figure 3 This is a flowchart illustrating another adaptive coding control method for remote surgical multimodal images provided in an embodiment of this application. Figure 4 This is a flowchart illustrating another adaptive coding control method for multimodal remote surgical images provided in an embodiment of this application. Figure 5 This is a flowchart illustrating another adaptive coding control method for multimodal remote surgical images provided in an embodiment of this application. Figure 6 This is a flowchart illustrating another adaptive encoding control method for multimodal remote surgical images provided in an embodiment of this application. Figure 7 This is a flowchart illustrating another adaptive coding control method for multimodal remote surgical images provided in an embodiment of this application. Figure 8 A schematic diagram of an adaptive encoding control device for remote surgical multimodal images provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0012] In this specification, numerous specific technical details are described in certain places to enable those skilled in the art to understand the complete technical solution. However, it should be understood that embodiments of this application can be implemented without these specific technical details. Such detailed descriptions of technical details should not be considered as limitations on this application, and the scope of protection of this application is defined only by the claims. Elsewhere, well-known structures, connections / positional relationships, circuits, and / or other details may not be shown in detail to avoid misunderstanding of the inventive points of this application by the general public. In this specification, accompanying drawings illustrate schematic diagrams of several embodiments of this application. However, the drawings are merely illustrative, and it should be understood that changes in mechanical structure, connections / positional relationships, physical composition, electrical aspects, and steps can be made without departing from the spirit and scope of this application. Such changes can be either substitutions or combinations of elements from several embodiments of this application, or substitutions or combinations of well-known content. The terminology used herein is only for describing particular embodiments and is not intended to limit this application. Spatial relative terms, such as “below,” “lower,” “above,” “upper,” “middle,” “center,” “inner,” “outer,” “central,” “edge,” etc., are used for ease of explanation to describe the relationship between one component or feature and another shown in the figure. It should be understood that spatial relative terms are used only under the condition of the orientation of the device in use or operation (other than the orientation specifically defined in the figure), and are not necessarily unique or constant. For example, if the device in the figure is rotated 180° up and down along the paper, then an element described as “below” other components or features will become “above” other components or features. Therefore, the exemplary term “below” can encompass both above and below directions, depending on how the device is positioned. The device can also be positioned in other directions (e.g., rotated 90° or positioned in other directions), and the spatial relative descriptive terms used herein will be interpreted accordingly.

[0013] As used herein, “several,” “one,” and “the” are intended to include the plural form as well, unless the context otherwise indicates. It should be further understood that the terms “comprising” and / or “including” specify the presence of the said feature, step, operation, element, and / or component, without excluding the presence of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0014] Finally, the terms “or” and “and / or” as used herein should be interpreted inclusively, meaning either one or any combination thereof. Therefore, “A, B, or C” or “A, B, and / or C” means any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C. Exceptions to this definition will only occur when the combination of elements, functions, steps, or actions is inherently mutually exclusive in some way.

[0015] Before introducing the technical solution provided in the embodiments of this application, the application scenario of the solution will be explained first. This embodiment is applicable to the scenario of real-time encoding and transmission of multimodal picture-in-picture composite images during remote surgery.

[0016] Remote surgical robots are core equipment in the field of medical robots. They can perform minimally invasive surgery by remotely controlling robotic arms, breaking geographical limitations and allowing high-quality medical resources to reach remote areas. They are especially suitable for minimally invasive surgical scenarios with extremely high requirements for surgical precision and real-time performance, such as laparoscopy, orthopedics, and neurosurgery. Figure 1 This is a schematic diagram of the system architecture of the remote surgical robot provided in an embodiment of this application. Figure 1 As shown, the remote surgical robot mainly consists of two parts: a local end (patient end) 11 and a remote end (doctor end) 12. The local end 11 further includes a patient carriage 112 and an image carriage 111, while the remote end 12 has a corresponding doctor carriage 121. To enable data interaction between the two ends, the local end 11 and the remote end 12 are respectively equipped with remote communication modules 113 and 122, which are connected through a communication network to form the physical support architecture for remote surgery.

[0017] In remote surgery, stable and low-latency transmission of surgical field images constitutes one of the core technical bottlenecks. Surgeons rely on real-time transmitted surgical field images to obtain intraoperative reference information and control the robotic arm. Therefore, the encoding parameters involved in image transmission (such as bit rate, quantization parameters, frame rate, resolution, buffer size, and forward error correction redundancy rate) directly affect the safety and accuracy of the surgical operation.

[0018] Due to limitations in system equipment compatibility, hardware deployment costs, and synchronization requirements between multiple devices, current remote surgical systems generally adopt a mainstream architecture of "first image synthesis, then single-stream encoding and transmission." This involves pre-combining the endoscopic main view and auxiliary modal views into a single, complete surgical field image, and then encoding and transmitting the synthesized single video stream over the network, rather than encoding each independent video stream separately and then recombining them at the surgeon's end. This architecture eliminates the need for multi-stream synchronization and image overlay operations at the surgeon's end, significantly reducing end-to-end latency and device adaptation complexity. It is the mainstream engineering solution for current remote surgical scenarios and one of the core applicable scenarios of this application.

[0019] In complex minimally invasive surgeries, surgeons need to simultaneously refer to other modalities of medical imaging information in addition to observing the endoscopic field of view. For example, they may use laparoscopic ultrasound images to identify deep blood vessels and lesion boundaries, monitor the patient's real-time heart rate and blood pressure, or use preoperative 3D reconstruction guidance to locate the spatial position of the lesion. Due to limitations in the physical display area of ​​the surgeon's console and the intraoperative interaction methods, this auxiliary information is usually superimposed on the corner area of ​​the main endoscopic view in a picture-in-picture (PiP) format, forming a composite surgical field of view containing multimodal content. For example, the ultrasound auxiliary image is displayed as a fixed rectangular window in the upper right corner of the screen, the monitor waveforms and values ​​are fixedly displayed as vertical bars at the edge of the screen, and the main endoscopic view occupies the remaining display area. The images from these different sources differ fundamentally in their imaging principles, statistical characteristics, and clinical value weighting, as analyzed in detail below: First, the essential differences in the statistical characteristics of multimodal images are reflected in the following: The main view of the endoscope is captured by an optical camera and is a full-color, high-texture natural dynamic image. Due to the frequent movement of surgical instruments and the continuous exposure and removal of tissue structures, the endoscope image contains rich edge information and complex motion vector fields, which are represented by high spatial complexity and high temporal complexity in video encoding. The encoder needs to consume a large amount of bitrate to characterize high-frequency components and motion compensation residuals.

[0020] Ultrasound-assisted images are reconstructed from ultrasound echo signals and are grayscale images. Their content mainly consists of smooth tissue areas and inherent speckle noise. The edge sharpness and color richness are far lower than those of endoscopic images. In video coding, the spatial complexity is low and the motion changes are slow. However, because clinical information is highly concentrated on tissue boundaries and lesion outlines, its overall tolerance to coding distortion is extremely low. Even slight quality degradation may lead to the loss of key diagnostic information.

[0021] The monitor screen is the device's data display interface, which contains high-contrast text and dynamic waveforms against a single dark background. The information density is highly concentrated in the pixels at the edges of the numbers and waveforms. Changes in content only occur in local areas where values ​​are refreshed or waveforms are advanced. The overall screen can be considered quasi-static on a second-level time scale. However, the vital signs parameters (such as heart rate and blood pressure) displayed by the monitor require extremely high recognition accuracy. Any blurring or ghosting introduced by the encoding may directly affect the immediate judgment of the patient's condition.

[0022] In summary, the three main types of images—endoscope, ultrasound, and monitor—differ significantly in color space, texture complexity, motion characteristics, and information density. If a uniform video encoding parameter is used for the synthesized frame, the encoder will be unable to adapt and optimize for the statistical characteristics of different image regions, inevitably leading to a structural mismatch where some regions have excessive bitrate and other regions have insufficient bitrate protection.

[0023] Secondly, considering the safety of remote surgery and the needs of clinical decision-making, the clinical criticality of the information conveyed by different screens varies: The real-time vital signs parameters displayed by the monitor reflect changes in the patient's condition and have the highest response priority during surgery. The clear identification of the numerical range is a basic prerequisite for surgical safety. The information on lesion boundaries, blood vessel course, and tissue layers in the ultrasound image directly determines the surgical resection range and operation path, and its clinical value is secondary. Although the endoscopic main image is the main visual feedback channel for surgical operation, not all areas within the image are equally important for real-time decision-making. The three-dimensional navigation image has high value in the initial positioning stage of surgery, but its importance decreases relatively in the fine operation stage.

[0024] Existing remote surgical image transmission systems typically employ uniform quantization parameters or bitrate allocation strategies for the entire frame of synthesized images during the encoding process, without distinguishing the aforementioned differences in clinical value weights. This results in the encoder allocating bitrate budgets based on the overall image complexity, leading to lower bitrates being assigned to low-texture ultrasound and monitoring areas due to their lower high-frequency components. Consequently, critical information with higher clinical priority faces the risk of over-compression.

[0025] Furthermore, in picture-in-picture overlay scenarios, there is also the problem of artifact accumulation in static areas: ultrasound and monitor images are usually in fixed positions and their content changes slowly. In inter-frame predictive coding mode, the encoder tends to frequently use skip mode (SKIP mode) for these areas, that is, directly reuse the reconstructed pixels at the corresponding positions of the previous frame without performing residual coding and transmission. This strategy can significantly improve efficiency in conventional video coding, but in the cross-network transmission environment of remote surgery, network packet loss and transmission jitter may cause inconsistencies between the encoder and decoder reference frame states. For endoscope areas that are frequently refreshed within frames, such errors can be quickly corrected by subsequent I-frames or forced refresh macroblocks; however, for static overlay image areas that are in SKIP mode for a long time, the decoder lacks an effective error correction mechanism, and quantization and transmission errors will accumulate over time, manifesting as intermittent flickering of ultrasound images, digital ghosting or color shift of monitors, and ringing effects at the boundaries of overlay windows, severely reducing the clinical usability of auxiliary images.

[0026] Furthermore, there is the issue of bitrate spikes caused by abrupt changes in screen layout: during surgery, surgeons may temporarily activate or deactivate the ultrasound picture-in-picture (PIP) or switch auxiliary screen display states as needed. Such abrupt changes in screen layout will cause a step change in the pixel content of the synthesized image at the boundaries of the PIP window. Existing scene transition detection algorithms in video encoders typically trigger I-frame forced encoding based on inter-frame pixel difference statistics. The data volume of an I-frame is usually 5 to 10 times that of a P-frame. Under unstable remote transmission network conditions, sudden bitrate spikes will instantaneously impact the bandwidth link, causing the encoded output buffer to overflow, end-to-end latency to rise sharply, and in severe cases, even triggering the system's forced degradation mechanism, resulting in a sharp drop in overall image quality.

[0027] Finally, existing video coding schemes also have limitations in multimodal composite image scenarios. Previous research has introduced the concept of region-of-interest (ROI) coding into surgical image transmission, using intelligent models to identify key anatomical objects (such as blood vessels, surgical instruments, and vital organs) in endoscopic images and assigning lower quantization parameter values ​​to corresponding regions to improve coding quality. While this approach embodies the idea of ​​differentiated coding, its identification objects are limited to the single endoscopic image and do not address the identification of different source images in multimodal composite images. Furthermore, it does not establish specific coding protection strategies for the grayscale, low-texture, quasi-static, and high clinical value attributes unique to auxiliary images such as ultrasound and monitors, nor does it address the artifact accumulation problem caused by long-term inter-frame prediction in static overlay images. In addition, this type of scheme typically processes images at the level of independent video streams before image synthesis, making it difficult to handle mainstream engineering scenarios where multiple images have been synthesized into a single video stream for transmission. The existing technical solutions closest to this application fall into two main categories.

[0028] Scheme 1 is a differentiated coding scheme based on intelligent surgical object recognition. The core objective of this scheme is to utilize an intelligent model to identify important objects in the surgical image and assign differentiated coding parameters based on the object's surgical importance. Specifically, scalpels, blood vessels, and vital organs are considered high-importance objects and assigned low quantization parameters, while background tissue is considered low-importance and assigned high quantization parameters. The specific workflow is as follows: The image acquisition module acquires surgical video frames; the deep learning recognition module processes the video frames, identifying surgical-related objects such as blood, blood vessels, bones, scalpels, vital organs, and the surgeon's hands, and outputs the category labels and bounding box coordinates for each object; the CU partitioning module divides the video frames into several coding tree units and coding units; the quantization parameter adjustment module, based on the deep learning recognition results, assigns lower quantization parameter values ​​to coding units containing important objects and higher quantization parameter values ​​to coding units in the background area; the video encoding module uses a common video encoding standard to encode and compress the video frames after adjusting the quantization parameters, outputting the encoded video stream.

[0029] Scheme 2 is a region of interest (ROI) extraction scheme based on bimodal medical video. The core objective of this scheme is to jointly determine the ROI of the target tissue using medical videos from two different modalities. The specific workflow is as follows: acquire the first and second modal medical videos of the target tissue; simultaneously input the first modal medical image and the corresponding second modal medical image into a neural network model to obtain the target tissue mask and confidence coefficient; determine the first and second keyframes based on the confidence coefficient; and determine the ROI of the target tissue based on the target tissue mask of the keyframes.

[0030] The aforementioned existing technical solutions all have the following drawbacks: First, there is a misalignment in the hierarchy of the identified objects. Scheme 1 identifies anatomical objects within a single endoscopic view, and its classification system is based on surgical anatomy. However, in a remote surgical picture-in-picture scenario, the composite image is stitched together from multiple sub-images of different sources and modalities. These sub-images should be considered independent semantic units. Existing technologies lack a mapping logic from multimodal image types to encoding strategies, causing the encoder to be unable to distinguish between endoscopic and ultrasound regions, and even less able to implement specialized encoding protection for the grayscale and low-texture characteristics of ultrasound images. While Scheme 2 involves bimodal video, its goal is to utilize bimodal information to jointly locate the region of interest, rather than implementing differentiated encoding parameter configurations for different modalities.

[0031] Secondly, the coding strategy is too narrow. The differentiated coding in Scheme 1 is only reflected in the spatial differentiation of quantization parameter values. It focuses on the bitrate allocation optimization within a single frame and does not address the error accumulation problem at the inter-frame coding level. In picture-in-picture scenarios, static superimposed images such as ultrasound and monitors are in the skip mode of inter-frame prediction for a long time. The quantization error and transmission error at the decoding end will accumulate frame by frame in these areas. Simply reducing the quantization parameter value cannot stop this error accumulation process. Existing technologies do not provide a forced intra-frame refresh mechanism for static key areas.

[0032] Third, existing solutions do not consider the differences in image statistical characteristics between different modalities. The quantization parameter adjustment strategy of Solution 1 is based on the single dimension of the surgical importance of the object, without considering the essential differences in image statistical characteristics between different image types. For example, ultrasound images are grayscale, low-texture images, and a moderate negative vectorization parameter offset can significantly improve edge sharpness. Monitor images have information highly concentrated on text edges, requiring stronger protection. Endoscopic images are full-color, high-texture images, which already have a high bitrate in unified encoding. These differentiated configurations based on image statistical characteristics have not been considered in existing technologies.

[0033] Fourth, existing solutions lack priority protection strategies for bandwidth degradation. When network conditions deteriorate, there is a lack of a step-by-step bitrate recovery mechanism based on image type priority. In remote surgery scenarios with limited bandwidth, indiscriminately degrading the encoding quality of all areas will result in simultaneous loss of image quality for the most critical clinical information, such as monitor readings and ultrasound lesions. Fifth, existing solutions do not consider encoding impacts caused by sudden changes in image layout, and do not perform dedicated encoding smoothing for picture-in-picture layout changes. When a sudden change in image layout occurs, the encoder's scene switching detection mechanism will forcibly insert an I-frame, causing bitrate spikes and global stuttering during network jitter.

[0034] Therefore, there is an urgent need for a method that can perceive the region type of the synthesized multimodal picture-in-picture and implement differentiated coding control based on the modality type and network conditions, so as to prioritize the allocation of limited coding resources to key clinical areas while ensuring low-latency transmission, and at the same time eliminate artifact accumulation in static areas and bitrate fluctuations during layout switching.

[0035] Example 1: Figure 2 This is a flowchart illustrating an adaptive encoding control method for multimodal images in remote surgery, provided in an embodiment of this application. This embodiment is applicable to scenarios involving real-time encoding and transmission of multimodal picture-in-picture composite images during remote surgery. The method can be executed by an adaptive encoding control device for multimodal images in remote surgery. This device can be implemented in software and / or hardware, where the hardware can be an electronic device such as a mobile terminal, PC, or server. Figure 2 As shown, the method specifically includes the following steps: S110. Real-time acquisition of surgical video frame sequences in picture-in-picture mode.

[0036] Picture-in-picture mode refers to a display mode in which multiple independent video signal sources are arranged and presented in a master-slave overlay manner on a single display screen. One signal source occupies the main visual content and the main area of ​​the screen, while the other one or more signal sources are superimposed on the main screen as smaller sub-windows to form a complete display screen.

[0037] The surgical video frame sequence refers to the data stream formed by continuously acquiring and transmitting each frame of digital images of the surgical field of view in chronological order during endoscopic surgery. Each frame contains complete visual information synthesized from all the signal sources that have been accessed at the current moment. The surgical video frame sequence is generated by overlaying the main image with at least one type of auxiliary modal image.

[0038] The main view refers to the screen that occupies the largest display area in the picture-in-picture layout and carries the core surgical field of view. It is usually composed of images of intracavitary tissues captured by the endoscopic camera and serves as the primary visual basis for remote doctors to make judgments during surgery. The auxiliary modal view refers to the supplementary information screen that is superimposed on the main view in the form of a sub-window in the picture-in-picture layout. Its signal source comes from other medical devices that work in conjunction with the endoscope, including but not limited to ultrasound imaging equipment, vital sign monitoring equipment, and 3D surgical navigation systems.

[0039] During remote surgery, the main endoscopic view contains the most important tissue imaging information of the surgical area. Simultaneously, auxiliary medical devices such as ultrasound equipment, vital sign monitors, or 3D navigation systems continuously generate diagnostic data that is crucial for surgical decision-making; this data exists as independent video signals. To facilitate the remote operator acquiring this multidimensional information simultaneously within a single view, the video signals from each auxiliary device are superimposed on the main endoscopic view in a picture-in-picture format, forming a single, fused video frame sequence. This constitutes the surgical video frame sequence in a multimodal picture-in-picture mode.

[0040] S120. Perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region.

[0041] Here, the current frame refers to the frame being processed at the current moment in the surgical video frame sequence. Each independent frame region refers to a single frame sub-region that can be detected as separate from each other in a picture-in-picture layout within the current frame, including the main frame region and each auxiliary modal frame sub-window region, each region having independent geometric boundaries and content sources.

[0042] Among them, image attribute information refers to the structured descriptive data extracted for each independent image region, used to characterize the geometric position and content category of that region within the image. Image attribute information includes bounding box coordinates and modal type.

[0043] Bounding box coordinates refer to the geometric positioning data used to describe the position and size of an independent image region in the current frame within the image plane. This can be determined by the region's top-left corner position, width, and height. Modal type refers to the category label assigned to each independent image region based on its content source and visual characteristics. This includes endoscope main image, ultrasound-assisted image, monitor image, 3D navigation image, and non-clinical information areas, used to distinguish the functional attributes and coding protection priorities of different image regions in clinical surgery. Specifically, non-clinical information areas can refer to areas in the composite image that do not carry clinical surgical information, such as black borders or decorative UI borders.

[0044] Specifically, the process begins by accurately identifying the individual picture-in-picture overlay regions in each frame of the surgical video sequence. This involves distinguishing the location and extent of the main endoscopic view and the various auxiliary modal view sub-windows, which are then defined as bounding box coordinates. After determining the spatial location of each individual view region, the source of the content carried by each region can be further identified. This involves determining whether the region belongs to the main endoscopic view, ultrasound-assisted view, monitor view, 3D navigation view, or a non-clinical information region, thereby obtaining the modality type of each view region.

[0045] In this embodiment, real-time image perception aims to accurately separate the main image from each auxiliary modal image from the superimposed mixed video frames. This can be achieved by relying on a deep learning model to perform end-to-end instance segmentation and classification on each frame, directly outputting the bounding box coordinates and modal type labels of each independent region. Alternatively, traditional computer vision methods can be used, such as region localization based on image border features, texture statistics, or preset template matching, to obtain the bounding box coordinates of each independent region, and then determining the modal type through color histograms or frequency domain features.

[0046] S130. Based on the current available network bandwidth and screen attribute information, determine the encoding control parameters corresponding to each screen area.

[0047] The currently available network bandwidth refers to the actual available network capacity for data transmission on the communication link used for transmitting surgical video streams at the current moment. It is typically obtained through the real-time bandwidth monitoring interface of the communication module, but can also be estimated using network metrics such as packet loss rate and round-trip latency.

[0048] The encoding control parameters refer to a set of encoding constraints independently configured for each frame region. These constraints are used to control the compression intensity and encoding method of each region during differentiated encoding. They include encoding quantization parameters and a forced intra-frame refresh rate. Encoding quantization parameters determine the degree of detail retention in that region; lower values ​​result in higher image quality but require a higher bitrate. The forced intra-frame refresh rate enforces a fixed frame interval for a given region, using an encoding method independent of the reference frame to prevent error accumulation.

[0049] Specifically, the current available network bandwidth reflects the upper limit of the data rate that the transmission link can currently carry. This value determines the total bitrate budget available for the entire frame and is the fundamental constraint for determining the encoding control parameters. The modal type in the image attribute information reflects the importance of each image region in clinical surgery. Different modal types correspond to different encoding quality protection requirements. For example, monitor images require extremely high accuracy in identifying numerical values ​​and waveforms, while background or non-clinical information areas can tolerate a greater degree of image quality sacrifice. Combining the above network constraints and image attribute information, the baseline encoding quantization parameters for the entire frame are first determined based on the current available network bandwidth. Then, the quantization parameter offset of each region relative to the baseline value is determined based on the modal type of each image region. This allows modal types with high clinical value to receive more refined quantization protection, while non-clinical information areas bear greater compression sacrifices. For regions with monitor and ultrasound-assisted modal types, a forced intra-frame refresh cycle is also set, forcing these static or quasi-static critical regions to adopt a reference frame-independent encoding method at fixed frame intervals, thereby preventing the continuous degradation of image quality caused by network transmission packet loss or quantization errors accumulating in the reference frame. The aforementioned quantization parameter offset, and / or forced intra-frame refresh period, can be used as encoding control parameters for each screen region.

[0050] S140. Based on the encoding control parameters of each screen region, perform differential encoding on the current frame and output the encoded video stream.

[0051] Specifically, after determining the encoding control parameters for each region of the current frame, these parameters can be applied to the encoding process of the current frame, thereby achieving differentiated image quality and bitrate allocation for different regions. Specifically: First, the bounding box coordinates and encoding quantization parameters of each region are sorted according to clinical priority. A limited number of regions with the highest priority are selected, and their bounding box coordinates and encoding quantization parameters are converted into configuration instructions that the encoder can recognize. This is because there is a hardware limit to the number of differentiated regions that the encoder can process in the same frame. Then, these configuration instructions are written into the encoder, so that when the encoder encodes the current frame, it compresses the macroblocks within each selected region using the corresponding encoding quantization parameters. That is, regions with higher clinical value receive more refined quantization preservation, while non-clinically important regions undergo greater compression to save bitrate.

[0052] This application provides an adaptive encoding control method for multimodal images in remote surgery. It acquires a sequence of surgical video frames in picture-in-picture mode in real time, where the sequence is generated by overlaying a main image with at least one type of auxiliary modal image. Then, it performs real-time image perception on the current frame in the surgical video frame sequence, identifies each independent image region within the current frame, and determines the image attribute information for each region, including bounding box coordinates and modal type. Based on the available network bandwidth and image attribute information, it determines the encoding control parameters corresponding to each image region. Finally, based on the encoding control parameters for each image region, it performs differentiated encoding on the current frame and outputs the encoded video stream. The technical solution provided in this embodiment, by real-time perception of the modal type and bounding box coordinates of each independent image region after picture-in-picture synthesis, and by determining differentiated encoding control parameters for each region based on the available network bandwidth, achieves independent encoding control of each region in the multimodal composite image. This solves the problem of incompatibility between existing encoding methods and multimodal composite images, improves the transmission fidelity of critical clinical information under limited bandwidth, and enhances the targeted allocation of encoding resources, thereby ensuring the operational safety and stability of remote surgery.

[0053] The following technical biases or cognitive limitations exist in this field, which makes this application not obvious: the research and development direction of existing region of interest coding technology has always focused on the identification of anatomical objects in a single modal image. Its underlying logic is to allocate the bit rate according to the importance of the object under the same image statistical characteristics. It has never paid attention to the essential statistical characteristics differences of different modal sub-images in the synthesized image, and has no motivation to carry out modality-type level semantic recognition and differential coding for a single synthesized video stream.

[0054] This solution can be implemented using a three-layer hybrid control architecture: image perception layer, strategy decision layer, and encoding execution layer. The core input is the sequence of surgical field video frames after multimodal picture-in-picture overlay synthesis. Auxiliary inputs include real-time measurements of available network bandwidth and encoder status feedback signals. The image perception layer receives the synthesized surgical field video frames, processes them in real-time using a lightweight visual model, and outputs a list of regions containing unique codes for each detection region, bounding box coordinates, modal type labels, region priorities, and coordinates of high-priority protected sub-regions. The strategy decision layer receives this region list, current available network bandwidth measurements, and feedback signals from the encoding execution layer. Based on preset two-dimensional mapping rules, it generates differentiated encoding parameter configurations for each region, including quantization parameter offsets for each region, forced intra-frame refresh cycles for high-priority regions, and transition control parameters during layout changes. Finally, it outputs a set of region of interest (ROI) configuration instructions recognizable by the encoder. The encoding execution layer receives this instruction set and applies the differentiated parameter configurations to the actual encoding process through the video encoder's ROI programming interface. Simultaneously, it monitors the actual bitrate output by the encoder, image layout change events, and encoding anomalies in real-time, performs smooth transition control, and feeds back relevant signals to the strategy decision layer, forming a closed-loop bitrate control. The inter-layer signal transmission relationship is as follows: the image perception layer transmits the region list to the strategy decision layer, the strategy decision layer transmits the region of interest configuration instruction set to the encoding execution layer, the encoding execution layer feeds back the actual encoding bitrate, layout change events and encoding abnormal status to the strategy decision layer, and the strategy decision layer dynamically adjusts the encoding parameter configuration based on the feedback to form a closed-loop control.

[0055] Example 2: Figure 3 This illustration shows an adaptive encoding control method for multimodal images in remote surgery, provided in an embodiment of this application. Based on the aforementioned embodiments, this embodiment details the specific implementation of determining the image attribute information of each image region. For specific implementation methods, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 3 As shown, the method specifically includes the following steps: S210. Real-time acquisition of surgical video frame sequences in picture-in-picture mode.

[0056] The surgical video frame sequence is generated by overlaying the main image with at least one type of auxiliary modal image.

[0057] S220. Input the current frame into the shared feature extraction network, and determine the bounding box coordinates of each independent image region in the current frame based on the region detection output head of the shared feature extraction network.

[0058] The shared feature extraction network refers to a neural network structure that performs multi-level visual feature extraction on the input current frame. The region detection output head is a task branch attached to the end of the shared feature extraction network, dedicated to predicting the location of independent regions in the image. It outputs the bounding box coordinates of each independent image region by performing position-by-position regression and classification on the feature map.

[0059] Specifically, the current frame is first fed into a shared feature extraction network, which extracts features from the entire frame to generate a feature map. This feature map retains both spatial location information and visual semantic information of different regions within the current frame. The region detection output head, as a task branch attached to the end of the shared feature extraction network, further performs target location regression and classification prediction based on this feature map, outputting the bounding box coordinates of each independent image region in the current frame. This distinguishes the various sub-image regions superimposed in a picture-in-picture layout within the composite image from one another.

[0060] To further reduce the computational latency of the image perception process while ensuring detection accuracy, in this embodiment, optionally, the current frame can be downsampled to a preset resolution before being input into the shared feature extraction network; based on the detection results output by the region detection output head, the bounding box coordinates of each independent image region are determined. Through downsampling, the spatial resolution of the input image is significantly reduced, and the computational load of the shared feature extraction network and the region detection output head decreases accordingly, thereby effectively shortening the inference time for single-frame image perception. Furthermore, since subsequent image synthesis and encoding processes have a high tolerance for the accuracy of region boundaries, moderate downsampling will not substantially affect the final encoding control parameter decisions, achieving a good balance between perception speed and detection accuracy.

[0061] For example, a lightweight shared feature extraction network can be used to process each current frame of the input. This shared feature extraction network uses a lightweight architecture with no more than 1M parameters, and the input resolution is the downsampled result of the original video frame (e.g., downsampling from 1920×1080 to 512×288). While ensuring detection accuracy, the single-frame inference latency is controlled to within 2ms. The network outputs the bounding box coordinates of each detected independent picture-in-picture sub-region, for example: Region 1 – top left corner coordinates (0, 0), width 1920, height 1080 (endoscope main screen); Region 2 – top left corner coordinates (1520, 60), width 320, height 240 (ultrasound picture-in-picture window).

[0062] S230. Capture the corresponding region map of each screen region in the current frame, and input each region image into the modality classification network to determine the confidence level of each region image belonging to each modality type included in the current picture-in-picture mode.

[0063] The region map refers to the corresponding sub-image patch cropped from the original current frame based on the bounding box coordinates. This sub-image patch retains the complete pixel information of the region.

[0064] The modality classification network is a classification task branch attached to the end of the shared feature extraction network or running independently on the region map. It is used to determine the content category of the input region map, and its output is the confidence score of the current region map belonging to each modality type included in the current picture-in-picture mode. The confidence score refers to the reliability of the modality classification network in predicting that the input region map belongs to a specific modality type, and is quantified in the form of probability values.

[0065] Specifically, after obtaining the bounding box coordinates of each independent frame region, a corresponding region map is cropped from the original current frame based on these coordinates. This region map fully preserves the original pixel information of the region and removes interference from surrounding frame regions. These region maps are then fed one by one into a modality classification network. This network classifies each region map into its content category and outputs a confidence score for each modality type included in the current picture-in-picture mode. Finally, by comparing these confidence scores, the most likely modality type corresponding to each independent frame region can be determined.

[0066] S240. The category with the highest confidence level is determined as the modal type of the corresponding screen area.

[0067] Specifically, for any region map, the modal classification network will output a confidence score for each modal type included in the current picture-in-picture mode, and take the category with the highest confidence score as the modal type of the picture region.

[0068] S250. Based on the coordinates of each bounding box and each modal type, generate the image attribute information of each image region.

[0069] Specifically, after obtaining the bounding box coordinates and modal type of each screen region, these two types of information can be organized into screen attribute information for each screen region according to a preset data structure. For each independent screen region, its corresponding bounding box coordinates and modal type are integrated together to form a structured data unit describing the location and content attributes of that region, which is the screen attribute information of each region.

[0070] S260. Based on the current available network bandwidth and screen attribute information, determine the encoding control parameters corresponding to each screen area.

[0071] S270: Based on the encoding control parameters of each screen region, perform differential encoding on the current frame and output the encoded video stream.

[0072] S280. Obtain the classification confidence of each frame region. When the classification confidence of each region is lower than the preset threshold, clear all differential encoding configurations of the current frame and reset each encoding control parameter to the parameter value of the global unified encoding mode. Control the encoder to switch to the global unified encoding mode to output the video bitstream. And / or, obtain the manually specified protection region and its encoding parameters, merge the manually specified protection region with each independent frame region, and when there is overlap, overwrite the corresponding encoding control parameters with the manually specified encoding parameters, and perform differential encoding on the current frame based on the overwritten encoding control parameters.

[0073] The preset threshold refers to a pre-set judgment threshold value for classification confidence, used to determine whether the image perception result is credible. Differential coding configuration refers to a set of coding control parameters set for several selected image regions in the current frame, including the coding quantization parameters and forced intra-frame refresh period for each region.

[0074] The global uniform coding mode refers to a coding method that does not perform differentiated processing on any region in the current frame. The entire picture is encoded using the same reference quantization parameter, and no quantization parameter offset or forced intra-frame refresh period is enabled in any region. All macroblocks are predicted between frames or within frames according to the encoder's default strategy.

[0075] During the image perception and encoding decision execution process, there are two independent safety fallback mechanisms to deal with different abnormal scenarios. On the one hand, when the classification confidence of each image region is lower than the preset threshold, it indicates that the overall image perception result of the current frame is unreliable. At this time, all differential encoding configurations of the current frame can be cleared, and each encoding control parameter can be reset to the parameter values ​​of the globally unified encoding mode. The encoder switches to the globally unified encoding mode to output the video bitstream, thus avoiding damage to the diagnostic quality of the surgical image due to differential encoding configurations generated based on erroneous perception results.

[0076] On the other hand, when remote operators believe that the automatic sensing results cannot meet the needs of a specific surgical scenario, they can manually designate a protected area and configure corresponding encoding parameters on the surgical field of view. The manually designated protected area is then merged with each of the automatically identified independent screen areas. If the manually designated protected area overlaps spatially with any independent screen area, the manually designated encoding parameters overwrite the original encoding control parameters of that screen area. The current frame is then differentially encoded based on the overwritten encoding control parameters, ensuring that the encoder's actual behavior prioritizes the manual instructions. Both mechanisms can be selectively enabled or configured simultaneously as needed, ensuring the availability of screen transmission even when automatic sensing fails, and providing operators with direct control over encoding behavior in special demand scenarios.

[0077] For example, when the confidence scores of all regions output by the image perception layer are lower than a preset threshold (e.g., lower than 0.7), or when there are drastic changes in the region detection results for 10 consecutive frames (e.g., the detection box position shifts by more than 10% of the image size between adjacent frames), it is determined that the perception module has temporarily failed. At this time, all differential encoding configurations for the current frame are cleared, and all encoding control parameters are reset to the parameter values ​​of the globally unified encoding mode. The encoder is then controlled to switch to the globally unified encoding mode to output the video stream. At the same time, the operator is prompted through the human-computer interaction interface that "Picture-in-Picture adaptive encoding has been paused." When the confidence score of the perception module recovers to above the threshold and remains stable for 3 seconds, it automatically switches back to the differential encoding mode.

[0078] And / or, obtain the manually specified protected area and its encoding parameters, merge the manually specified protected area with each independent screen area, when the manually specified area overlaps with the intelligent recognition area, overwrite the corresponding encoding control parameters with the manually specified encoding parameters, and perform differential encoding on the current frame based on the overwritten encoding control parameters; or disable all intelligent driver differential encoding with one click through the interface button, and force the use of the global unified encoding mode.

[0079] The technical solution of this application embodiment, when determining the image attribute information of each image region, achieves spatial localization of each independent picture-in-picture sub-image region in the synthesized image through the cooperation of a shared feature extraction network and a region detection output head, and outputs the bounding box coordinates of each region. Based on this, the corresponding region map is extracted and input into a modality classification network to determine the type of content source for each region, ultimately generating image attribute information containing bounding box coordinates and modality type. The above process reduces redundant computation through a shared backbone network mechanism, effectively reducing computational overhead and processing latency while ensuring perceptual accuracy, and providing accurate spatial location and semantic category basis for subsequent configuration of modality-based differential coding parameters.

[0080] In the technical solution of this application embodiment, during the encoding process, the classification confidence level output by the image perception module is an important indicator for measuring the reliability of the current frame recognition result. When all classification confidence levels are lower than a preset threshold, it means that the perception layer's reliability in judging the content of each region of the current frame is insufficient. At this time, it is necessary to clear all differentiated encoding configurations of the current frame and reset each encoding control parameter to the parameter values ​​of the globally unified encoding mode. The encoder is then controlled to switch to the globally unified encoding mode to output the video bitstream, so as to avoid allocating encoding resources based on unreliable perception results, which would damage the image quality of key areas. At the same time, in order to take into account the need for correction of automatic perception results in special surgical scenarios, a manually specified protection region and its encoding parameters are obtained. The manually specified protection region is merged with each independent image region. When there is overlap, the manually specified encoding parameters are used to cover the corresponding encoding control parameters, and the current frame is differentiatedly encoded based on the covered encoding control parameters. The above mechanism ensures that when perception fails, it can automatically fall back to a safe and reliable baseline encoding method, ensuring the availability of surgical image transmission. At the same time, it gives operators the control right to directly intervene in encoding behavior in abnormal or special scenarios, so that automatic perception and manual judgment complement each other and jointly constitute a complete security guarantee for encoding control.

[0081] Example 3: Figure 4 This illustration shows an adaptive encoding control method for multimodal remote surgical images provided in this application. Based on the aforementioned embodiments, this embodiment elaborates on another method for determining the image attribute information of each image region. For specific implementation details, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 4 As shown, the method specifically includes the following steps: S310: Real-time acquisition of surgical video frame sequences in picture-in-picture mode.

[0082] The surgical video frame sequence is generated by overlaying the main image with at least one type of auxiliary modal image.

[0083] S320. Obtain the prior region configuration information corresponding to the current frame.

[0084] Among them, the prior area configuration information refers to the pre-set structured data that describes the expected layout of each candidate screen area in the composite image. Its content includes the preset position of each candidate screen area and the corresponding preset modal type.

[0085] Here, candidate frame regions refer to independent sub-regions of the frame that are expected to exist in the current frame and require perceptual processing, as predetermined based on prior region configuration information. Each candidate frame region corresponds to a predicted location and a predicted modality type. The preset location refers to the predicted spatial position and size of a candidate frame region in the composite frame, as specified in the prior region configuration information. The preset modality type refers to the predicted content category corresponding to a candidate frame region, as specified in the prior region configuration information.

[0086] Specifically, in order to quickly determine the possible locations of each independent image region in the current frame and their corresponding content types, the prior region configuration information corresponding to the current frame can be obtained first. This information includes the preset positions of each candidate image region in the current frame and the preset modal types corresponding to each candidate image region. These preset data are derived from the fixed layout information of the picture-in-picture sub-window during the operation.

[0087] S330. Based on the preset position, determine the candidate bounding box coordinates of each candidate image area.

[0088] Specifically, the prior region configuration information provides the preset positions of each candidate image region in the current frame. Based on these preset positions, the candidate bounding box coordinates of each candidate image region within the current frame can be derived, that is, the preliminary location of each candidate image region in the image coordinate system is determined according to the spatial range information described by the preset positions. These candidate bounding box coordinates serve as initial estimates for subsequent processing steps.

[0089] S340. Perform edge detection on the current frame, correct the coordinates of the candidate bounding boxes based on the edge detection results, and determine the bounding box coordinates of each frame region.

[0090] Specifically, since the actual screen layout may deviate slightly from the preset position due to factors such as window scaling, position fine-tuning, or signal switching, the candidate bounding box coordinates can be corrected to obtain more accurate screen region boundaries. To this end, edge detection is performed on the current frame to extract the boundary contour information between different regions in the image. Based on the edge detection results, using the candidate bounding box coordinates as an initial reference, the actual region boundary positions are found near the candidate boundaries. Based on this, the position and size of the candidate bounding box coordinates are finely adjusted so that the adjusted bounding box accurately matches the actual sub-screen region boundaries in the current frame. After the above corrections, the bounding box coordinates of each screen region are determined.

[0091] S350. Based on the coordinates of each bounding box, crop the region image, extract the image features of the region image and match them with the preset feature templates of each modality type to determine the modality type of each screen region.

[0092] Specifically, after determining the bounding box coordinates of each image region, the modality type corresponding to each region can be further determined, i.e., whether the region belongs to the endoscope main image, ultrasound-assisted image, monitor image, 3D navigation image, or non-clinical information region. This determination process first crops the corresponding region image from the current frame based on the bounding box coordinates. This region image only contains the pixel content of that region, excluding interference from surrounding regions. Subsequently, image features that characterize the visual content of the region image are extracted from the region image, including color distribution features and texture structure features. These features describe the visual appearance of the region image from different dimensions. At the same time, for each preset modality type, a corresponding preset feature template is pre-stored. This template represents the typical distribution pattern of that modality content in the feature space. For example, it is constructed by pre-collecting typical image samples of various modalities such as endoscopy, ultrasound, and monitors, and extracting statistical features such as grayscale histogram, texture complexity, and edge density. The image features extracted from the region image are matched one by one with the preset feature templates of each modality type to calculate the similarity between the two. The modality type corresponding to the preset feature template with the highest similarity in the matching results is determined as the modality type of the image region. This modality discrimination method based on feature template matching does not rely on a large amount of labeled data for model training; it can achieve rapid classification based on preset prior templates.

[0093] S360 generates image attribute information for each image region based on the coordinates of each bounding box and each modal type.

[0094] S370. Based on the current available network bandwidth and screen attribute information, determine the encoding control parameters corresponding to each screen area.

[0095] S380: Based on the encoding control parameters of each screen region, perform differential encoding on the current frame and output the encoded video stream.

[0096] The technical solution in this embodiment, as another implementation method, when determining the image attribute information of each image area, fully utilizes the prior knowledge that the picture-in-picture layout in the surgical scene is relatively fixed, and uses the preset position provided by the prior area configuration information as the initial positioning basis, which greatly reduces the spatial search range of the perception link and reduces the computational complexity and processing latency. On this basis, the preset position is locally corrected by edge detection, so that the bounding box coordinates are accurately aligned with the actual image edge, taking into account both the speed of prior information and the adaptability of the actual image. Finally, modality type discrimination is achieved by matching image features with preset feature templates, avoiding the high computational overhead brought about by building and running deep neural networks. In this way, while ensuring perception accuracy, the consumption of computing resources and the single-frame processing latency are reduced, making it suitable for remote surgical communication scenarios with limited computing power.

[0097] Example 4: Figure 5 This diagram illustrates an adaptive encoding control method for multimodal images in remote surgery, provided in an embodiment of this application. Based on the aforementioned embodiments, and when the encoding control parameters include encoding quantization parameters, the implementation method for determining the encoding control parameters corresponding to each image region can be further detailed. Specific implementation methods can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 5 As shown, the method specifically includes the following steps: S410: Real-time acquisition of surgical video frame sequences in picture-in-picture mode.

[0098] The surgical video frame sequence is generated by overlaying the main image with at least one type of auxiliary modal image.

[0099] S420. Perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region.

[0100] The image attribute information includes bounding box coordinates and modal type.

[0101] After determining the image attribute information of each image region, suitable encoding parameters can be configured for each image region based on the available network bandwidth to achieve image quality protection for clinically critical areas under limited bandwidth conditions. Encoding control parameters include encoding quantization parameters, which determine the degree of image detail retention in each region during encoding and are the core adjustment means to control the image quality and bitrate allocation of each region.

[0102] S430. Based on the available network bandwidth, determine the baseline quantization parameters for the entire frame.

[0103] Among them, the baseline quantization parameter refers to the baseline value of basic compression intensity uniformly determined for the entire frame based on the available bandwidth of the current network. This value determines the overall image quality level of the entire frame before differential adjustments are applied.

[0104] Specifically, before determining the differentiated encoding parameters for each frame region, a baseline for the encoding quality of the entire frame can be established. The current available network bandwidth reflects the upper limit of the data rate that the transmission link can currently carry, determining the total available bitrate budget for the entire frame. The baseline quantization parameter is the reference value used by the encoder when compressing the entire frame. Its value is inversely proportional to the current available network bandwidth; that is, when bandwidth is ample, the baseline quantization parameter value is lower, and the overall image quality is higher; when bandwidth is limited, the baseline quantization parameter value is higher, and the overall image quality is correspondingly lower. This baseline quantization parameter serves as the reference zero point for the subsequent quantization parameter offsets of each region. The actual encoding quantization parameters of each region are adjusted differentially based on this baseline value, thereby achieving a redistribution of image quality among different regions while the overall bitrate is limited by the current available network bandwidth.

[0105] For example, when determining the encoding quality benchmark for the entire frame, the benchmark bitrate of the entire frame is first determined based on the measurement of the available bandwidth of the current network. The benchmark bitrate is set according to a preset proportion of the available bandwidth of the current network, and a portion of the bandwidth is reserved for control signal, audio transmission and burst traffic. On this basis, the benchmark quantization parameters are calculated in real time by the encoder's built-in bitrate control module based on the benchmark bitrate, the current screen resolution and the frame rate.

[0106] S440. Based on the modal type in the image attribute information, determine the modal differentiation offset of each image region according to the preset mapping rules.

[0107] Among them, the preset mapping rule refers to the set of pre-defined correspondences that map different modal types to different quantization parameter offsets.

[0108] The modal differentiation offset refers to the adjustment range relative to the baseline quantization parameter found by a preset mapping rule based on the modal type of the image area. This offset can be negative or positive, and is used to enable different modal types of image areas to obtain different degrees of image quality improvement or compression sacrifice relative to the baseline value during encoding.

[0109] Specifically, based on the modal types of each image region identified in the image attribute information, a lookup table or mapping operation can be performed according to a preset mapping rule. This preset mapping rule predefines the offset values ​​corresponding to different modal types, and different modal types are assigned different offset amounts due to differences in visual characteristics and clinical importance. After the above mapping, the modal differentiation offset of each image region can be determined, so that the actual coding quantization parameters of each region will have different degrees of offset from the baseline value.

[0110] When determining the modal differential offset of each screen region based on modal type, different modal types should be assigned different offset values ​​with different directions and ranges due to differences in their visual characteristics and clinical importance. In this embodiment, optionally, the specific implementation of determining the modal differential offset of each screen region based on modal type and preset mapping rules may include: when the modal type is the endoscope main screen, the modal differential offset is set to a baseline zero value; when the modal type is an ultrasound-assisted screen, the modal differential offset belongs to a first negative offset interval; when the modal type is a monitor screen, the modal differential offset belongs to a second negative offset interval; wherein, the absolute value of the second negative offset interval is generally greater than the absolute value of the first negative offset interval; when the modal type is a three-dimensional navigation screen, the modal differential offset belongs to a first positive offset interval; when the modal type is a non-clinical information area, the modal differential offset belongs to a second positive offset interval; wherein, the value of the second positive offset interval is generally greater than the value of the first positive offset interval.

[0111] For example, the endoscope main image serves as the baseline reference for the entire frame, and its modal differential offset is set to 0; the ultrasound-assisted image is a grayscale low-texture image with low tolerance for coding distortion, and a moderate negative offset can significantly improve the clarity of tissue edges and the smoothness of speckle areas, with its modal differential offset in the range of -2 to -4; the monitor image information is highly concentrated in the edge pixels of text and waveforms, with the highest clinical priority and extremely high requirements for recognition accuracy, and should be given the largest negative offset and the highest priority coding resources, with its modal differential offset in the range of -3 to -5; the 3D navigation image is a smooth curved surface rendering, and its sensitivity to coding distortion and clinical priority are slightly lower than that of the endoscope main image, with its modal differential offset in the range of +1 to +2; the non-clinical information area contains no clinical information, and positive offset can significantly reduce bitrate consumption, with the saved bitrate budget being redistributed to high-priority areas, with its modal differential offset in the range of +4 to +8.

[0112] The design basis for the aforementioned preset mapping rules is as follows: the statistical characteristics of different modal images determine their varying sensitivities to changes in quantization parameters. For example, grayscale, low-texture ultrasound images, after appropriately reducing quantization parameters, show a significantly greater improvement in edge sharpness than full-color, high-texture endoscopic images. Simultaneously, the clinical value weight of different images determines the protection priority; vital sign values ​​from monitors and ultrasound lesion boundaries, which are related to surgical safety, require the highest level of coding protection. Furthermore, different mapping rules can be preset according to different surgical types and different physician preferences.

[0113] The above mapping rules ensure that the endoscope main screen, as the core visual reference, maintains the baseline image quality. The monitor screen and ultrasound-assisted screen, due to the high concentration of clinical information in text, waveforms, and edge details, receive different degrees of negative offset to improve image quality. The 3D navigation screen and non-clinical information areas bear positive offset to save bitrate, thereby realizing the targeted transfer of coding resources from clinically redundant areas to clinically critical areas.

[0114] After determining the basic modal differentiation offset for each image region based on modal type, further enhanced protection can be applied to local sub-regions within certain modal types that have higher priority protection requirements. Specifically, when a lesion outline is marked in an ultrasound-assisted image or a vital sign value is marked in a monitor image, the clinical safety value of these high-priority protection sub-regions is far higher than that of other parts of their respective images, and modal-level offset configuration alone is insufficient to meet their image quality protection requirements. In this embodiment, optionally, determining the modal differentiation parameter offset for each image region based on the modal type in the image attribute information further includes: when the image attribute information contains a high-priority protection sub-region, adding a preset negative offset to the modal differentiation parameter offset of the image region to which the high-priority protection sub-region belongs, to determine the modal differentiation parameter offset of the high-priority protection sub-region.

[0115] The high-priority protection sub-region refers to the area within the ultrasound-assisted or monitor display that carries local visual information crucial to surgical safety. This area requires stronger coding protection because its clinical value far exceeds that of other parts of the display. The preset negative offset refers to a pre-set negative adjustment value.

[0116] Specifically, if a certain screen area contains a high-priority protected sub-region, an additional preset negative offset is superimposed on the already determined modal differentiation offset of the screen area. The superposition result is used as the modal differentiation offset of the high-priority protected sub-region itself, so that its actual coding quantization parameters are further reduced compared with the screen area to which it belongs, thereby obtaining a higher degree of image quality preservation than the screen area to which it belongs.

[0117] For example, for lesion sub-regions in ultrasound-assisted images or numerical sub-regions in monitor images, an additional preset negative offset can be superimposed on the already determined modal differential offset for that region. This additional preset negative offset has a value between -2 and -3. Through this method, the actual encoded quantization parameters of the high-priority protected sub-regions are further reduced compared to their corresponding image regions, achieving a higher degree of image quality protection and enabling limited bitrate resources to be refocused on the most critical clinical information.

[0118] S450: Based on the baseline quantization parameters and the modal differential offsets, determine the encoding quantization parameters corresponding to each screen region.

[0119] Specifically, a baseline quantization parameter can be used as a unified reference point for the entire frame. Then, the modal differentiation offsets for each frame region are superimposed on the baseline quantization parameter, causing the actual encoded quantization parameters for each frame region to shift around the baseline value, thereby achieving differentiated image quality allocation between regions of different modal types. This encoded quantization parameter is used to control the compression intensity of macroblocks in the corresponding region during subsequent differentiated coding.

[0120] In this embodiment, optionally, the specific implementation method for determining the coding quantization parameters corresponding to each image region based on the baseline quantization parameters and the modal differential offsets may include the following steps: The first step is to scale and adjust the modal differentiation parameter offset based on the available network bandwidth to determine the bandwidth adaptive parameter offset for each screen area.

[0121] Among them, the bandwidth adaptive parameter offset refers to the result obtained by dynamically scaling and adjusting the modal differentiation offset according to the sufficiency or scarcity of the current available network bandwidth.

[0122] Specifically, when the available network bandwidth is relatively abundant, the negative offset of each region is increased proportionally to provide higher image quality redundancy for key clinical regions; when the available network bandwidth is relatively scarce, the negative offset of each region is reduced step by step according to clinical priority, with priority given to reducing the positive offset of non-clinical information regions, and finally only the negative offset of high-priority protected sub-regions remains unchanged.

[0123] In this embodiment, when the available network bandwidth is higher than the preset bandwidth sufficiency threshold, the negative offset of each screen area is increased proportionally, so that the critical clinical areas can obtain image quality redundancy exceeding the baseline level under good network conditions; when the available network bandwidth is lower than the preset bandwidth shortage threshold, the negative offset of each screen area is reduced step by step according to the preset clinical priority order, and the positive offset of non-clinical information areas is reduced first, while only the negative offset of high-priority protected sub-areas remains unchanged.

[0124] For example, when the available network bandwidth is higher than the preset bandwidth sufficiency threshold, the negative offset of each region can be increased proportionally to 1.2 to 1.5 times, so that the key clinical regions can obtain image quality redundancy exceeding the baseline level under good network conditions; when the available network bandwidth is lower than the preset bandwidth shortage threshold, the negative offset of each region is reduced step by step according to the preset clinical priority order. First, the positive offset of the non-clinical information region is gradually adjusted from +8 to +4. Second, the negative offset of the endoscope main image is set to zero. Third, the negative offset of the ultrasound-assisted image is adjusted from -4 to -2.

[0125] After the above scaling adjustment, the bandwidth adaptive parameter offset of each screen area is obtained. This offset fully integrates the reference offset direction and magnitude determined by the modality type and the dynamic scaling constraint imposed by the current available network bandwidth, so that the actual coding quantization parameters of each subsequent area not only reflect the differentiated protection needs of clinical priority, but also adapt to the real-time changes in network conditions.

[0126] The second step is to determine the encoding quantization parameters for each frame region based on the baseline quantization parameters and the bandwidth adaptive parameter offset for each frame region.

[0127] Specifically, the baseline quantization parameter is used as a unified reference point for the entire frame. Then, the bandwidth adaptive parameter offset of each frame area is superimposed on the baseline value, so that the actual coding quantization parameters of each frame area are optimally allocated under the constraint of the available network bandwidth.

[0128] S460 performs differential encoding on the current frame based on the encoding control parameters of each screen region, and outputs the encoded video stream.

[0129] In this embodiment, the encoding control parameters may include encoding quantization parameters. Based on this, when determining the encoding control parameters for each frame region, the baseline quantization parameters for the entire frame are first determined according to the available network bandwidth, ensuring the overall bitrate level matches the current network capacity. Then, modal differentiation offsets are determined based on the modal type of each frame region, achieving initial image quality differentiation between regions of different clinical value. Finally, the baseline quantization parameters are superimposed with the modal differentiation offsets to obtain the encoding quantization parameters for each frame region. This ensures that the actual encoding quantization parameters for each region are offset to varying degrees from the baseline value, guaranteeing that the overall bitrate is limited by the available network bandwidth while ensuring that regions with higher clinical value receive lower encoding quantization parameters. This allows for the targeted transfer of encoding resources from non-clinical information regions to clinically critical regions within a limited bandwidth budget, achieving optimal matching between bitrate allocation and clinical priority.

[0130] Example 5: Figure 6This diagram illustrates an adaptive encoding control method for multimodal images in remote surgery, provided in an embodiment of this application. Based on the aforementioned embodiments, and considering that the encoding control parameters include a forced intra-frame refresh period, the implementation method for determining the encoding control parameters corresponding to each image region can be further detailed. Specific implementation methods can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 6 As shown, the method specifically includes the following steps: S510: Real-time acquisition of surgical video frame sequences in picture-in-picture mode.

[0131] The surgical video frame sequence is generated by overlaying the main image with at least one type of auxiliary modal image.

[0132] S520. Perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region.

[0133] The image attribute information includes bounding box coordinates and modal type.

[0134] In this embodiment, in addition to configuring encoding and quantization parameters for each screen region based on modality and clinical priority, attention can also be paid to the image quality stability and error accumulation issues of static or quasi-static key regions over time. Based on this, the encoding control parameters include a forced intra-frame refresh cycle. This parameter specifies that a specific screen region is forced to undergo a complete refresh every fixed number of frames using an encoding method independent of any preceding reference frame. This periodically prevents the continuous degradation of image quality caused by network packet loss or quantization errors accumulating in the reference frame, ensuring that the clinically critical information displayed on the remote end remains accurate and reliable over the long term.

[0135] S530. Based on the modal type in the screen attribute information, determine the screen area that needs to be refreshed within the frame and the forced intra-frame refresh period.

[0136] Specifically, based on the modal types of each screen region identified in the screen attribute information, it can be determined which regions require forced intra-frame refresh. For example, regions with modal types of monitor screens and ultrasound-assisted screens can be identified as areas requiring intra-frame refresh. After determining the refresh regions, a corresponding forced intra-frame refresh cycle is configured for each refresh region according to its modal type and whether it contains a high-priority protection sub-region. This ensures that regions with higher clinical value receive shorter refresh cycles, thereby preventing error accumulation at a higher frequency and ensuring that the static key information displayed on the remote end remains accurate and reliable over the long term.

[0137] In this embodiment, optionally, the specific implementation of determining the screen area requiring intra-frame refresh and the forced intra-frame refresh cycle based on the modal type in the screen attribute information may include: determining the areas with modal types of monitor screen and ultrasound-assisted screen in the screen attribute information as the screen areas requiring intra-frame refresh; determining the refresh cycle of the waveform area in the monitor screen as the first cycle; determining the refresh cycle of the entire ultrasound-assisted screen area as the second cycle; and performing staggered refresh frame allocation for each screen area requiring intra-frame refresh so that the refresh frames of each screen area do not overlap in time.

[0138] The second period is longer than the first period. For example, the first period can be set to the range of 25 to 35 frames, and the second period can be set to the range of 30 to 45 frames.

[0139] Specifically, when determining the image regions requiring intra-frame refresh, the selection is based on the modality type in the image attribute information. Regions with the modality types of monitor images and ultrasound-assisted images are specifically identified as requiring intra-frame refresh because these two types of images contain a large amount of static or quasi-static clinically critical information and are most sensitive to error accumulation. After determining the refresh regions, corresponding refresh cycles can be configured for different regions. The refresh cycle for the waveform region in the monitor image is determined as the first cycle, and the refresh cycle for the entire ultrasound-assisted image region is determined as the second cycle. Finally, to avoid multiple regions triggering forced intra-frame refreshes simultaneously in the same frame, resulting in excessively high instantaneous bitrate peaks, refresh frames for each region requiring intra-frame refresh are staggered, ensuring that the refresh frames for each region are evenly staggered in time without overlapping. This maintains the periodic error blocking effect of each region while smoothing the bitrate fluctuations of the encoded output.

[0140] After determining the basic refresh cycle for monitor and ultrasound-assisted images based on modal type, these two types of images may contain critical local information with higher clinical value and protection requirements than the overall area or waveform area. Specifically, the key numerical display area in the monitor image carries quantitative readings of vital signs, and the lesion outline area in the ultrasound-assisted image identifies the spatial range and boundaries of the lesion tissue. This local information plays a decisive role in surgical safety, and modal-level refresh cycle configuration alone is insufficient to meet the frequency requirements for error prevention. Therefore, after determining the image attribute information of each image area, the following is also included: When the modal type is monitor screen, identify the key value display area in the monitor screen and determine the key value display area as a high-priority protection sub-area.

[0141] The key value display area refers to the local display position on the monitor screen used to present quantitative readings of vital signs such as heart rate, blood pressure, and blood oxygen saturation. This area carries the core data for real-time quantitative assessment of the patient's condition.

[0142] Specifically, after classifying each screen area according to modal type, for areas identified as monitor screens, the key numerical display areas carrying quantitative vital sign readings can be further located within these areas. These values ​​are the most direct quantitative basis for remotely assessing the patient's condition. By locating this key numerical display area and designating it as a high-priority protection sub-area, this local area can obtain higher image quality protection and refresh rate than other parts of the monitor screen in subsequent encoding parameter configurations, ensuring the clarity and accuracy of vital sign readings.

[0143] Optionally, the method for identifying the key value display area in the monitor screen may include: inputting the monitor screen into a lightweight text detection network to identify the display position of key vital sign values ​​in the monitor screen; and extracting the bounding box coordinates of the key value display area based on the display position.

[0144] When the modality type is ultrasound-assisted image, the lesion contour region in the ultrasound-assisted image is identified and the lesion contour region is determined as a high-priority protection sub-region.

[0145] Among them, the lesion outline area refers to the boundary range of the lesion tissue marked or identified by the doctor in the ultrasound-assisted image. This area marks the pathological spatial location that needs to be observed and operated on.

[0146] Specifically, for areas identified as ultrasound-assisted images, the outlines of marked or presented lesions within these areas are further identified. These outlines define the spatial location and boundaries of the lesion tissue, serving as crucial visual references for precise manipulation during remote surgery. By identifying these lesion outlines and designating them as high-priority protected sub-regions, this local area can receive higher image quality protection than other parts of the ultrasound-assisted image in subsequent coding parameter configurations, ensuring the clarity of the lesion boundaries.

[0147] Optionally, the specific implementation of identifying the lesion contour region in the ultrasound-assisted image may include: inputting the ultrasound-assisted image into a lightweight segmentation network to generate a pixel-level mask of the lesion contour; and determining the bounding box coordinates of the lesion contour region based on the minimum bounding rectangle of the pixel-level mask.

[0148] Add the high-priority protected sub-region to the screen attribute information of the corresponding screen region.

[0149] Specifically, the bounding box coordinates and subtype labels of the high-priority protected sub-region can be incorporated into the image attribute information of the corresponding image region. In addition to the original bounding box coordinates and modal types of each image region, the updated image attribute information also records whether a high-priority protected sub-region exists within that region and its specific location. This allows for the application of stronger image quality protection and a shorter refresh cycle to that local area during subsequent encoding parameter configuration.

[0150] Based on the above embodiments, the implementation of the encoding control parameters including the forced intra-frame refresh cycle further includes: determining the forced intra-frame refresh cycle of the high-priority protection sub-region in the monitor screen as the third cycle; and determining the forced intra-frame refresh cycle of the high-priority protection sub-region in the ultrasound-assisted screen as the fourth cycle.

[0151] The third cycle is shorter than the refresh cycle of the waveform area in the monitor screen; the fourth cycle is between the third cycle and the refresh cycle of the entire ultrasound-assisted screen area. Based on the above example, the third cycle can be set to a range of 15 to 20 frames (approximately 250 to 330 ms refresh per second at 60 fps), and the fourth cycle can be set to a range of 20 to 30 frames (approximately 330 to 500 ms refresh per second at 60 fps).

[0152] Specifically, for the high-priority protection sub-region in the monitor screen, its forced intra-frame refresh cycle is determined to be the third cycle. This cycle value is less than the refresh cycle of the waveform area in the monitor screen, allowing the key numerical display area carrying vital sign readings to obtain a higher refresh frequency than the waveform area. For the high-priority protection sub-region in the ultrasound-assisted screen, its forced intra-frame refresh cycle is determined to be the fourth cycle. This cycle value is between the third cycle of the high-priority protection sub-region in the monitor screen and the second cycle of the overall ultrasound-assisted screen area, allowing the lesion contour area to obtain a higher refresh frequency than the overall ultrasound area but slightly lower than the monitor numerical area.

[0153] Through the above differentiated configuration, different priority areas form a gradient distribution in refresh frequency that matches clinical importance. The highest frequency is concentrated in the monitor value area, which is most decisive for surgical safety, followed by the ultrasound lesion outline area, then the monitor waveform area, and finally the overall ultrasound area.

[0154] The mechanism of forced intra-frame refresh is as follows: Video coding standards (H.264 / H.265 / VVC / AVS3, etc.) stipulate that macroblocks coded in intra-frame prediction mode do not depend on any preceding reference frames; their reconstructed pixels are entirely determined by the coded data of the current frame. By periodically applying intra-frame prediction coding to a specified area, even if the content of that area has not changed, the decoder will reconstruct the image of that area using the latest transmitted coded data, thereby blocking the path of accumulation and frame-by-frame transmission of network packet loss and quantization errors in the reference frame. This mechanism is fundamentally different from strategies that only reduce quantization parameters; reducing quantization parameters can only reduce quantization errors in a single encoding, but cannot eliminate the continuous impact of existing errors already present in the reference frame on subsequent frames.

[0155] S540 performs differential encoding on the current frame based on the encoding control parameters of each screen region and outputs the encoded video stream.

[0156] As an optional implementation, this application adopts a lightweight deep learning-based visual model as the core implementation method for picture-in-picture region perception and recognition. This model adopts an integrated architecture of "shared backbone network + multi-task head", with the overall parameter size controlled within 1M and the single-frame inference latency not exceeding 2ms on the embedded platform. At the same time, it meets the accuracy indicators of region detection mAP (mean Average Precision) ≥95% and modality classification accuracy ≥97%, so as to adapt to the stringent requirements of low latency, high robustness and high accuracy in remote surgical scenarios.

[0157] For the backbone network, the model can use lightweight feature extraction networks such as MobileNetV2 and its lightweight variants as a shared backbone. Taking the improved MobileNetV2-Lite preferred in this application as an example, it has been specifically pruned and optimized based on the general MobileNetV2: the number of channels has been greatly reduced (maximum channels compressed to 64), only four core depthwise separable convolutional blocks are retained, and a lightweight channel attention mechanism (SE module, Squeeze-and-Excitation module) is introduced in specific layers, thereby significantly reducing redundant parameters while maintaining good feature extraction capabilities for low-texture ultrasound images and high-contrast monitor images.

[0158] In terms of task heads, the model employs multiple lightweight output heads to perform different perception tasks. The target detection head, based on deep feature maps output from the backbone network, uses a minimalist single-detection-head structure (containing only two depthwise separable convolutional layers) with pre-defined anchor boxes to accurately locate and output confidence scores for the bounding boxes of independent regions within the picture-in-picture. The modality classification head, based on global pooling feature vectors from the backbone network, uses two cascaded fully connected layers to determine the modality type of the image (e.g., endoscope, ultrasound, monitor). Furthermore, for sub-regions with high clinical value, the model reuses shallow feature maps from the backbone network, adding extremely lightweight segmentation heads or traditional image processing heads to respectively extract ultrasound lesion contours and detect numerical regions on monitors. These multiple task heads, together with the shared backbone, constitute a complete end-to-end perception architecture.

[0159] The model takes as input a single frame of surgical footage synthesized from picture-in-picture (preferably downsampled to 512×288 resolution), and outputs the bounding box coordinates of each independent image region, the modality type label corresponding to each region, and the location information of high-priority sub-regions (such as ultrasound lesion regions and monitor value regions). This output information directly serves the subsequent differential coding control module, providing a perceptual basis for coding parameter allocation. The entire model employs a combination of pre-training, phased fine-tuning, and knowledge distillation during the training phase, and utilizes INT8 (8-bit integer) quantization and inference pipeline optimization via a deep learning inference optimizer during the deployment phase. Ultimately, it achieves real-time and stable inference performance on the surgical robot's embedded coding board.

[0160] It is generally believed in the industry that performing secondary region recognition and classification on the synthesized image would introduce additional latency, which does not meet the core requirement of low latency for remote surgery. This application controls the single-frame inference latency to within 2ms through a lightweight model architecture design, breaking through this inherent technical bias and is not a conventional technical choice in this field.

[0161] In this embodiment, the encoding control parameters may include a forced intra-frame refresh cycle. Based on this, when determining the encoding control parameters corresponding to each screen area, the monitor screen and ultrasound-assisted screen are selected as areas requiring intra-frame refresh based on the modal type in the screen attribute information. Different forced intra-frame refresh cycles are configured according to different area types and whether they contain high-priority protection sub-areas. This forces static or quasi-static clinical key screen areas to be refreshed at fixed frame intervals using an encoding method independent of reference frames. This periodically blocks the continuous degradation of image quality caused by network packet loss or quantization errors accumulating in the reference frame, ensuring that the monitoring values, waveforms, and ultrasound lesion boundaries displayed on the remote end remain accurate and reliable over a long period. At the same time, by staggering the refresh frame allocation, the refresh frames of each area are evenly staggered in time and do not overlap, avoiding the peak bit rate fluctuation caused by multiple areas triggering forced refresh in the same frame.

[0162] Example 6: Figure 7 This diagram illustrates an adaptive encoding control method for multimodal images in remote surgery, provided in an embodiment of this application. Based on the aforementioned embodiments, the specific implementation of differentiated encoding for the current frame can be further detailed. For detailed implementation methods, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 7 As shown, the method specifically includes the following steps: S610: Real-time acquisition of surgical video frame sequences in picture-in-picture mode.

[0163] The surgical video frame sequence is generated by overlaying the main image with at least one type of auxiliary modal image.

[0164] S620. Perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region.

[0165] The image attribute information includes bounding box coordinates and modal type.

[0166] S630. Based on the current available network bandwidth and screen attribute information, determine the encoding control parameters corresponding to each screen area.

[0167] S640. Sort each screen area according to the preset clinical priority order, and select a preset number of screen areas with the highest priority based on the sorting results.

[0168] Clinical priority refers to a pre-defined order in which different image regions are allocated in the encoding resources. Regions with higher clinical value are ranked higher and receive priority for image quality protection when the number of encoder-differentiated regions is limited or the bitrate is tight. For example, the specific clinical priority order is as follows: the monitor numerical value sub-region is highest, followed by the ultrasound lesion sub-region, the monitor waveform region, the overall ultrasound region, the 3D navigation image, the endoscope main image, and the background region is lowest. Regions with higher clinical value are ranked higher and receive priority for image quality protection when the number of encoder-differentiated regions is limited or the bitrate is tight.

[0169] Specifically, after obtaining the encoding control parameters corresponding to each frame region, since encoder hardware typically only supports a limited number of differentiated regions, it cannot apply differentiated encoding configurations to all independent frame regions identified in the current frame one by one. Therefore, the most important regions can be selected for actual configuration. Specifically, the frame regions are sorted according to a preset clinical priority order, which is pre-determined based on the clinical value weight of each region, with regions of higher clinical value ranked higher. Based on the ranking result, a preset number of frame regions with the highest priority are selected from the head of the sorting queue. This preset number is equal to the maximum number of differentiated regions supported by the encoder hardware. Thus, the bounding box coordinates, quantization parameter offsets, and intra-frame refresh periods of the selected frame regions can be converted into region of interest configuration instructions compatible with the encoder programming interface for differentiated encoding of the current frame.

[0170] After the above screening, only the selected image areas will receive differentiated encoding quantization parameter configurations in the subsequent encoding process, while the remaining areas will be encoded using the baseline quantization parameters of the entire frame. This way, the limited differentiated encoding capabilities will be concentrated on the image areas with the highest clinical value within the constraints of the encoder hardware.

[0171] S650: Convert the bounding box coordinates and encoding quantization parameters of each selected screen area into region of interest configuration instructions compatible with the encoder programming interface.

[0172] The encoder programming interface refers to the application programming interface provided by the encoder software development kit. Through this interface, externally configured parameters can be written into the encoder's internal registers, thereby controlling the encoder's actual encoding behavior.

[0173] The Region of Interest (ROI) configuration instruction refers to a set of configuration information organized according to the data format required by the encoder programming interface. It includes the bounding box coordinates of the specified region and its corresponding encoding quantization parameters. After receiving the instruction, the encoder can compress and encode the macroblocks in each region using the corresponding encoding quantization parameters.

[0174] Specifically, after prioritizing and selecting a preset number of image regions, each selected region contains bounding box coordinates and corresponding encoding quantization parameters. However, these two types of information cannot be directly recognized and executed by the encoder and need to be reorganized according to the data structure and format required by the encoder programming interface. Specifically, the bounding box coordinates and encoding quantization parameters of each selected image region are converted into region of interest (ROI) configuration instructions compatible with the encoder programming interface. This allows the encoder to correctly parse the spatial position of each selected region in the image and its corresponding encoding quantization parameter values, and configure these parameters in the encoder's internal registers, preparing for subsequent actual encoding execution.

[0175] S660: Write the region of interest configuration instruction into the encoder to control the encoder to encode macroblocks in each frame area with the corresponding encoding quantization parameters. When the forced intra-frame refresh period in the encoding control parameters is reached, the macroblocks in the corresponding frame area are forced to be encoded using the intra-frame prediction mode, and the encoded video stream is output.

[0176] Specifically, the encoder compresses macroblocks within each selected frame region using corresponding coding quantization parameters, resulting in differentiated image quality preservation levels for regions with different clinical priorities. Simultaneously, for frame regions that meet the mandatory intra-frame refresh cycle requirements of the coding control parameters, when the refresh frame is reached, the macroblocks within that region are forced to be encoded using an intra-frame prediction mode that does not rely on any preceding reference frame. This periodically prevents image quality degradation caused by network packet loss or quantization errors accumulating in the reference frame. After this differentiated coding process, the current frame is output as a video stream for remote decoding and display.

[0177] It should be noted that this embodiment is compatible with all mainstream video coding standards that support ROI coding / macroblock-level coding quantization parameter adjustment, including but not limited to H.264, H.265, VVC, AV1, AVS3 and other mainstream domestic and international coding standards.

[0178] During the procedure, the picture-in-picture layout may change in real time due to the opening, closing, position movement, or size adjustment of the auxiliary screen. At this time, the region list output by the image perception layer will be updated accordingly, and the strategy decision layer will generate a new region of interest configuration. However, if the new configuration is immediately and fully applied, the quantization parameter offset of each image region may abruptly change between adjacent frames. The encoder may misinterpret this as a scene switch due to large inter-frame jumps in macroblock quantization parameters, resulting in additional coding noise or bitrate fluctuations. Therefore, the method further includes the following steps: The first step is to obtain the newly generated region of interest configuration when a change in the picture-in-picture layout is detected.

[0179] Among them, the region of interest configuration refers to the set of encoding parameter instructions generated for each selected screen region in the current frame, including the bounding box coordinates, encoding quantization parameters, and forced intra-frame refresh period of each region.

[0180] During the procedure, the picture-in-picture layout may change in real time due to the opening, closing, repositioning, or resizing of the auxiliary screen. A layout change means that the spatial position, size, or number of each screen region in the current frame has changed, and the original region of interest configuration will no longer be applicable to the changed screen layout. For example, by comparing the screen perception results of adjacent frames, if the number of independent screen regions, the change in bounding box coordinates exceeds a preset threshold, or the modal type changes, it is determined that the picture-in-picture layout has changed. When such a layout change is detected, a newly generated region of interest configuration can be obtained. This is a set of encoding parameter instructions regenerated by the strategy decision layer based on the changed screen layout. This configuration includes information such as the bounding box coordinates, encoding quantization parameters, and forced intra-frame refresh cycle for each screen region after the change, and is used to replace the original old configuration to adapt to the differentiated encoding requirements after the layout change.

[0181] The second step is to gradually transition the current region of interest configuration to the newly generated region of interest configuration within the preset transition time window, so that the quantization parameter offset of each screen region gradually changes from the current value to the target value.

[0182] The preset transition time window refers to a pre-defined frame interval used to gradually adjust the encoding parameters from the current configuration to the new configuration when the picture-in-picture layout changes. The current region of interest configuration refers to the region of interest configuration that was in effect before the layout change occurred.

[0183] Here, the current value refers to the current encoding and quantization parameter value of each frame region in the current region of interest configuration. The target value refers to the target encoding and quantization parameter value of each frame region in the newly generated region of interest configuration.

[0184] After acquiring the newly generated Region of Interest (ROI) configuration, the current configuration cannot be immediately overwritten with the complete target value, as large jumps in the quantization parameter offsets of each frame region between adjacent frames could cause severe fluctuations in encoding quality. Therefore, within a preset transition time window, the current ROI configuration is gradually transitioned to the newly generated ROI configuration, allowing the quantization parameter offsets of each frame region to smoothly change from their current values ​​to the target values ​​over several frames. Specifically, the quantization parameter offset of each frame gradually approaches the target value with a preset step size, rather than jumping directly to the target value within a single frame. After frame-by-frame adjustments throughout the transition time window, it eventually stabilizes at the target value, thus avoiding encoding quality fluctuations caused by abrupt configuration changes.

[0185] The third step is to update the region of interest configuration parameters of the encoder frame by frame within the transition time window so that the quantization parameters of the current frame change smoothly in the time dimension.

[0186] Specifically, during the gradual transition from the current region of interest (ROI) configuration to the newly generated ROI configuration, the quantization parameter offset of each frame gradually approaches the target value rather than jumping directly within a single frame. To achieve this, before encoding begins for each frame within the transition time window, the intermediate state quantization parameter offset corresponding to the current frame is updated frame-by-frame in the encoder's ROI configuration parameters. This ensures that the encoder uses the corresponding intermediate state parameter value for each frame during actual encoding. Through frame-by-frame updates throughout the transition time window, the quantization parameters of each image region exhibit a smooth trajectory over time, rather than abruptly jumping in a single frame, thus ensuring a stable transition in image quality during layout changes.

[0187] For example, when a change in the picture-in-picture layout is detected, a newly generated region of interest (ROI) configuration is obtained. When applying the new configuration, it does not immediately take effect with the full target value. Instead, it gradually transitions from the current ROI configuration to the newly generated ROI configuration within a preset transition time window. This allows the quantization parameter offsets of each frame region to gradually change from their current values ​​to the target values ​​in a linear or stepwise manner within a transition time window of 3 to 5 frames. During the transition, the encoder's ROI configuration parameters are updated frame by frame, ensuring a smooth change in the distribution of quantization parameters across the frame over time. This avoids misinterpreting inter-frame abrupt changes in macroblock quantization parameters as scene transitions or generating encoding noise.

[0188] The above method introduces a gradual transition mechanism when the layout changes, so that the quantization parameters of each region are smoothly adjusted within a few frame windows, avoiding the oscillation of coding quality caused by sudden configuration changes, and ensuring the visual continuity and coding stability of the output bitstream during the switching process.

[0189] During continuous encoder operation, hardware status or software interface anomalies may occur, causing the region of interest configuration to fail to be written or executed correctly. To ensure the reliability of the encoding process, this method may further include the following steps: The first step is to monitor the configuration status of the region of interest returned by the encoder.

[0190] The region of interest configuration status refers to the execution result feedback returned by the encoder after receiving and applying the region of interest configuration instruction, which indicates whether the configuration has been successfully written and taken effect.

[0191] Specifically, after writing the region of interest (ROI) configuration command to the encoder and controlling it to start encoding, the encoder's response to the configuration command can be continuously monitored, i.e., the ROI configuration status returned by the encoder can be monitored. This configuration status reflects whether the encoder has successfully received and applied the written ROI configuration command, including whether the configuration has been correctly parsed and whether the corresponding differential encoding parameters have actually taken effect during the encoding process. By continuously monitoring the ROI configuration status returned by the encoder, any anomalies that may occur during the encoder's configuration application process can be detected in a timely manner, providing a basis for subsequent anomaly handling.

[0192] The second step is to clear all configuration settings for regions of interest and control the encoder to revert to the global unified encoding mode when the configuration status indicates configuration failure or encoding error multiple times in a row.

[0193] The global uniform coding mode refers to a coding method that does not perform differentiated processing on any region in the current frame. The entire picture is encoded using the same reference quantization parameter, and no quantization parameter offset or forced intra-frame refresh period is enabled in any region. All macroblocks are predicted between frames or within frames according to the encoder's default strategy.

[0194] Specifically, during continuous monitoring of the region of interest (ROI) configuration status returned by the encoder, if the configuration status repeatedly indicates configuration failure or encoding error, it means that the encoder is currently unable to receive or execute differential encoding configuration normally. Continuing to attempt to write differential configuration may further deteriorate the encoding status or even lead to abnormal bitstream output. At this time, all ROI configurations can be cleared, that is, all currently saved differential encoding parameter settings in the encoder can be cleared. At the same time, the encoder can be controlled to fall back to the global unified encoding mode, so that the encoder encodes the entire frame in a unified way without differentiating any region. This avoids the encoder outputting an incorrect bitstream due to configuration abnormalities, ensuring that the surgical image can still be continuously transmitted at the baseline quality even when the encoder status is abnormal.

[0195] The third step is to trigger an alarm signal to prompt the user to check the encoder status.

[0196] Among them, alarm signals are human-machine-aware prompts triggered when the encoder returns configuration failure or encoding error multiple times in a row. They are used to notify operators that the current encoder status is abnormal and prompt them to check the encoder.

[0197] Specifically, while controlling the encoder to revert to the globally unified encoding mode, an alarm signal can be triggered. This alarm signal is presented on the operation interface in the form of visual, audible, or text prompts to clearly indicate to the operator that the current encoder status is abnormal and the differentiated encoding function has been suspended. This reminds the operator to check and troubleshoot the encoder hardware or software interface, thereby providing a basis for manual intervention to restore the abnormal state while ensuring continuous image transmission through automatic reversion.

[0198] For example, the system monitors the region of interest configuration status returned by the encoder. If the encoder returns configuration failure or encoding error three times in a row, all region of interest configurations are automatically cleared and the system reverts to the global unified encoding mode. At the same time, an alarm signal is triggered through the system interface to prompt the system to check the encoder status.

[0199] The above method avoids encoder output errors due to configuration anomalies by automatically clearing differentiated configurations and reverting to the global unified mode when configuration fails. At the same time, it prompts the user to check the encoder status through alarm signals, ensuring the availability and reliability of surgical image transmission in abnormal scenarios.

[0200] During differentiated coding, each region is configured with corresponding coding quantization parameters based on the available network bandwidth and clinical priority. However, the actual bitrate output by the encoder may deviate from the preset target bitrate range due to factors such as fluctuations in the complexity of the image content. To ensure that the bitrate of the encoded output always adapts to the available network bandwidth, the method may further include the following steps: The first step is to monitor the actual bit rate output by the encoder in real time.

[0201] The actual bitrate refers to the data rate actually output by the encoder after differential encoding the current frame. This value varies with the complexity of the image content.

[0202] Specifically, during the process of the encoder performing differential encoding on the current frame and outputting the video bitstream, the actual data rate output by the encoder can be continuously acquired, i.e., the actual bitrate output by the encoder can be monitored in real time. This actual bitrate reflects the amount of data actually generated by the encoder under the complexity of the current frame's content and the current differential encoding parameter configuration. This value may fluctuate around the target bitrate due to dynamic changes in the content. By continuously acquiring the actual bitrate, it is possible to promptly detect whether the actual bitrate deviates from the preset target range, providing real-time data for subsequent bitrate feedback adjustments.

[0203] The second step is to report a bitrate overrun event when the actual bitrate of multiple consecutive frames exceeds the target bitrate.

[0204] The target bitrate refers to the upper limit of the expected data rate set for the entire frame based on the available network bandwidth, and is the control target value of the encoder output bitrate.

[0205] Among them, the bitrate exceeding event refers to a feedback signal triggered when the actual bitrate output by the encoder for multiple consecutive frames exceeds the target bitrate, which is used to notify the policy decision layer that the current bitrate has deviated from the target range.

[0206] Specifically, during real-time monitoring of the encoder's output bitrate, a momentary exceedance of the target bitrate in a single frame might be due to occasional fluctuations in the complexity of the image content, and is insufficient to indicate a persistent bitrate deviation in the current encoding parameter configuration. Therefore, the trend of actual bitrate changes over multiple consecutive frames can be observed. Only when the actual bitrate of multiple consecutive frames exceeds the target bitrate is it considered a persistent bitrate deviation event. At this point, a bitrate overshoot event is triggered and fed back to the strategy decision layer, informing it that the current differentiated encoding configuration has caused the actual bitrate to continuously exceed the target range, thereby initiating corresponding bitrate budget recovery operations to pull the bitrate back to the target range.

[0207] The third step is to respond to the bitrate exceeding event by gradually recovering the bitrate budget according to the preset clinical priority order until the actual bitrate converges to the target bitrate range.

[0208] Clinical priority order refers to the pre-defined order of each screen region in the allocation of encoding resources. Regions with higher clinical value are ranked higher. When the bit rate exceeds the budget, their negative offset is retained first. The specific order is as follows: first reduce the positive offset of non-clinical information regions, then reduce the negative offset of the endoscope main screen, then reduce the negative offset of the ultrasound-assisted screen, and finally retain the negative offset of the high-priority protection sub-regions in the monitor screen and the high-priority protection sub-regions in the ultrasound-assisted screen unchanged.

[0209] Specifically, the bitrate budget is reclaimed tier by tier according to a preset clinical priority order. This starts with the areas of the image with the lowest clinical value, gradually reducing their negative offsets or decreasing their positive offsets. This causes the coding quantization parameters in these areas to revert towards the baseline value, thereby releasing some bitrate budget. If the actual bitrate has not converged to the target bitrate range after the first tier of reclamation, the reclamation operation continues to the next lower level of clinical priority areas, progressing upwards until the actual bitrate falls back to the target bitrate range. Throughout the reclamation process, the highest clinical priority areas are only touched at the last tier, ensuring that the most critical clinical safety information is preserved to the greatest extent possible within a limited reclamation range.

[0210] For example, during the differential coding process, the actual bitrate output by the encoder can be monitored in real time. If the actual bitrate exceeds 110% of the target bitrate for 3 to 5 consecutive frames, a bitrate overshoot event is reported to the strategy decision layer. After receiving the feedback, the strategy decision layer recovers the bitrate budget step by step according to the preset clinical priority order. The closed-loop cycle of feedback and adjustment is 5 to 10 frames, until the actual bitrate converges to the target bitrate range within a short period of time.

[0211] The above method monitors the actual bitrate in real time and triggers tiered recycling when it exceeds the limit, so that the bitrate converges to the target range in a short time, while ensuring that the most critical clinical safety information can still be encoded and protected to the greatest extent under the most demanding bandwidth conditions.

[0212] The technical solution of this embodiment, when outputting the encoded video stream after differential coding of the current frame, sorts each frame region according to clinical priority before encoding execution, and selects a preset number of frame regions with the highest priority from the sorting results as the actual targets of differential coding. This concentrates the limited differential coding capability on the frame regions with the highest clinical value under the constraints of encoder hardware. Based on this, the bounding box coordinates and coding quantization parameters of each selected frame region are converted into region of interest configuration instructions compatible with the encoder programming interface and written into the encoder. This allows the encoder to compress macroblocks in each frame region with the corresponding coding quantization parameters, achieving differentiated image quality allocation for different clinical priority regions. Simultaneously, for frame regions that meet the mandatory intra-frame refresh cycle, the macroblocks are forced to adopt an intra-frame prediction mode independent of the reference frame when the refresh frame arrives, periodically blocking error accumulation. These mechanisms together ensure that differential coding resources are preferentially allocated to the frame regions with the highest clinical priority under encoder hardware constraints, while periodic refresh maintains the long-term image quality stability of static key regions, ultimately outputting a video stream that balances clinical safety and coding efficiency under limited bandwidth and hardware conditions.

[0213] Example 7: This example uses an ultrasound-guided picture-in-picture scenario during laparoscopic liver resection surgery as an example to illustrate the remote surgical multimodal image adaptive coding control method provided in this application. In this example, the surgeon needs to refer in real time to the images of blood vessels and lesion boundaries inside the liver obtained by the laparoscopic ultrasound probe. The output signal of the ultrasound device and the endoscopic video signal are superimposed by a picture synthesizer. The ultrasound image is fixedly displayed in the upper right corner of the screen in picture-in-picture format with a resolution of 320×240 pixels. The synthesized surgical field of view is a single video stream with a resolution of 1920×1080 and a frame rate of 60 frames per second, which is the core applicable scenario of this application.

[0214] This embodiment deploys the three-layer control architecture of this application on the video encoding board of the patient end of the proximal module of a remote surgical robot. The region detection network of the image perception layer adopts a lightweight shared feature extraction network, with the input resolution downsampled to 512×288, the number of parameters approximately 0.8M, and a single-frame inference latency of 1.2ms; the classification network shares the feature extraction backbone with the detection network, with an additional 0.3ms inference latency for the classification head; the ultrasound lesion segmentation module adopts a lightweight UNet architecture with an inference latency of 1.0ms. The encoder is an H.265 hardware encoder that supports macroblock-level coding quantization parameter offset. In the preset parameters, the base bit rate coefficient is 0.8, the bandwidth sufficiency threshold is set to 25Mbps, and the bandwidth constraint threshold is set to 10Mbps.

[0215] The image perception layer processes each frame of the synthesized image. The detection network outputs the bounding box coordinates of the ultrasound image region as the top left corner (1520, 60), with a width of 320 and a height of 240. The classification network confirms that the modality type of this region is ultrasound-assisted image, with a confidence level of 98.5%. The ultrasound lesion segmentation module further identifies the lesion contour region automatically drawn by the ultrasound device AI within the ultrasound image, takes its smallest bounding rectangle as a high-priority protected sub-region, and adds the bounding box coordinates to the region list.

[0216] The current available network bandwidth is measured at 20 Mbps. Based on this bandwidth, the policy decision layer sets the baseline bitrate to 16 Mbps at a preset ratio of 0.8, corresponding to a baseline quantization parameter of 28. Differential configurations are generated according to the two-dimensional mapping rules: the modal differential offset for the overall ultrasound region is -3, with an actual encoding quantization parameter of 25; the ultrasound lesion sub-region has an additional preset negative offset of -2 on top of -3, resulting in an actual encoding quantization parameter of 23; the modal differential offset for the endoscope main image is 0, with an actual encoding quantization parameter of 28; and the modal differential offset for the black border filling area around the image is +6, with an actual encoding quantization parameter of 34. In the forced refresh strategy, the ultrasound lesion sub-region is forced to refresh every 20 frames, and the overall ultrasound region is forced to refresh every 30 frames.

[0217] After applying the region of interest (ROI) configuration to the encoder, the average bitrate percentage of the ultrasound image area increased from 2.8% in the previous unified encoding mode to 5.2%. Image quality assessment showed a significant improvement in the edge sharpness of blood vessel walls in the ultrasound images, with the peak signal-to-noise ratio increasing from 34dB in unified encoding to 39dB; lesion outlines showed no mosquito-like noise or jagged edges. The forced refresh mechanism ensured that intermittent flickering and ghosting issues no longer occurred in the ultrasound image during the 60-minute surgery. The end-to-end latency remained stable within the range of 80 to 95ms throughout the procedure, without triggering any latency alarms.

[0218] At the 35-minute mark of the procedure, the surgeon temporarily removed the ultrasound probe from the abdominal cavity. The ultrasound equipment stopped outputting valid images, and the image synthesizer automatically removed the picture-in-picture. The image perception layer detected the disappearance of the ultrasound image, and the strategy decision layer output the cleared region of interest configuration. Within a 5-frame transition window, the encoding execution layer gradually reduced the quantization parameter offsets of all regions to zero, and the image smoothly switched back to the globally unified encoding mode. No sudden changes in bitrate or image stuttering were observed during the transition, and no encoder scene switching or I-frame insertion was triggered.

[0219] Example 8: This example uses a picture-in-picture scenario of a monitor in cardiovascular interventional surgery as an example to describe in detail the remote surgical multimodal image adaptive encoding control method provided in this application. In this example, the surgeon needs to overlay the vital signs values ​​and electrocardiogram waveforms of the monitor onto the main screen of digital subtraction angiography. The monitor image is fixed on the left edge of the screen in a vertical picture-in-picture format, with a width of 160 pixels and a height of 1080 pixels. The composite image is a single video stream with a resolution of 1920×1080 and a frame rate of 60 frames per second.

[0220] The image perception layer is optimized for this scenario. The region detection network identifies the overall bounding box of the monitor screen, with the bounding box coordinates being the top left corner (0, 0), a width of 160, and a height of 1080. The classification network confirms that the modal type of this region is the monitor screen. The text detection network identifies the precise display coordinates of the heart rate, systolic and diastolic blood pressure, and blood oxygen saturation values ​​on the screen, and extracts their bounding boxes as high-priority protection sub-regions.

[0221] The current available network bandwidth is measured at 15 Mbps. The policy decision layer sets the baseline bitrate to 12 Mbps based on a preset ratio of 0.8, corresponding to a baseline quantization parameter of 30. Differential configurations are generated according to the two-dimensional mapping rules: the modal differential offset for the overall monitor area is -3, with an actual encoding quantization parameter of 27; the heart rate, blood pressure, and blood oxygen sub-regions have an additional preset negative offset of -2 on top of -3, resulting in an actual encoding quantization parameter of 25; the modal differential offset for the monitor waveform area is -3, with an actual encoding quantization parameter of 27; the modal differential offset for the main screen of digital subtraction angiography is 0, with an actual encoding quantization parameter of 30; and the modal differential offset for the black border area of ​​the background is +5, with an actual encoding quantization parameter of 35. In the forced refresh strategy, the forced intra-frame refresh cycle for the heart rate sub-region is 15 frames, the forced intra-frame refresh cycle for the blood pressure sub-region is 15 frames, and the forced intra-frame refresh cycle for the waveform area is 25 frames.

[0222] In the unified encoding mode, the heart rate digits on the monitor occasionally appear blurry at the last digit due to quantization compression, requiring doctors to get close to the screen to see them. After enabling this solution, the heart rate and blood pressure values ​​remain clear and sharp, without any ghosting or blurring. Even when the network bandwidth briefly drops to 8Mbps and the policy decision layer triggers step-by-step bitrate reclamation, the modal differential offset of -5 in the monitor's numerical area is still fully preserved. Only the bitrate budget of the main digital subtraction angiography image is moderately reduced, and the image quality of core vital signs information is not affected in any way.

[0223] Example 9: This example uses a complex laparoscopic tumor resection surgery scenario that simultaneously includes ultrasound picture-in-picture and monitor picture-in-picture to illustrate the remote surgery multimodal image adaptive coding control method provided in this application. The ultrasound image is located in the upper right corner of the screen with a resolution of 320×240 pixels. The monitor image is located on the left edge of the screen in a vertical picture-in-picture format with a width of 160 pixels and a height of 1080 pixels. The composite image is a single video stream with a resolution of 1920×1080 and a frame rate of 60 frames per second.

[0224] The image perception layer simultaneously detects and classifies two picture-in-picture regions. The output region list includes: region A is the endoscope main image, region B is the ultrasound auxiliary image located in the upper right corner, and region C is the monitor image located on the left edge. It also outputs the coordinates of the ultrasound lesion high-priority protection sub-region and the monitor value high-priority protection sub-region.

[0225] The current available network bandwidth is measured at 18 Mbps. Based on this bandwidth, the policy decision layer sets the baseline bitrate to 14.4 Mbps at a preset ratio of 0.8, corresponding to a baseline quantization parameter of 29. Differential configurations are generated according to the two-dimensional mapping rules: the modal differential offset for the overall ultrasound region is -3; the total offset for the ultrasound lesion sub-region is -5 after adding a preset negative offset of -2 to the -3 value; the modal differential offset for the overall monitor region is -3; the total offset for the monitor value sub-region is -5 after adding a preset negative offset of -2 to the -3 value; the modal differential offset for the endoscope main screen is 0; and the modal differential offset for the background region is +6. The forced refresh strategy is executed according to the preset cycle corresponding to each region.

[0226] The encoder used in this embodiment supports eight rectangular regions of interest. The strategy decision layer distributes the region of interest configurations according to clinical priority: the heart rate value region of the monitor is the highest priority, followed by the blood pressure value region of the monitor, the lesion region of the ultrasound, the waveform region of the monitor, the overall ultrasound region, the overall monitor region, the non-operational region of the endoscope, and the background region. The above order completely covers the core protection requirements in this scenario.

[0227] As the simulated network bandwidth gradually decreased from 18Mbps to 6Mbps, the stepwise bitrate recovery process of the policy decision layer was as follows: When the bandwidth dropped to 14Mbps, the positive offset of the background area was reduced from +6 to +4; when the bandwidth dropped to 10Mbps, the modal difference offset of the endoscope main image was adjusted from 0 to +2, and its image quality was moderately reduced; when the bandwidth dropped to 8Mbps, the negative offset of the overall ultrasound area was reduced from -3 to -1; when the bandwidth dropped to 6Mbps, the negative offset of the monitor waveform area was reduced from -3 to -1, but the total offset of -5 of the monitor numerical sub-area and the ultrasound lesion sub-area was never reduced.

[0228] Example 10: This example is an alternative to a lightweight deep learning solution, employing traditional computer vision methods to achieve multimodal region recognition and classification. This example is applicable to low-end coding devices with limited computing power, where deep learning models cannot be deployed. The surgical scenario is an ultrasound-guided picture-in-picture scenario during laparoscopic liver resection. This example achieves the same region recognition and classification functions as deep learning solutions using traditional computer vision methods.

[0229] In terms of region detection and classification methods, prior region configuration is first performed, pre-configuring the preset display position and size range of the picture-in-picture (PIP) screen. In clinical practice, the PIP position is usually a fixed preset value, and candidate region coordinates are obtained accordingly. Then, image statistical features are extracted for each candidate region, including gray-level histogram distribution, texture complexity based on gradient amplitude variance, edge density, and color space distribution. After feature extraction, modality classification is performed based on preset feature thresholds. Specifically, the endoscopic main screen features full color, high texture, and high edge density; the ultrasound-assisted screen features single-channel grayscale, low texture, low edge density, and speckle noise; the monitor screen features high contrast, low texture, and significant text edge features; and the background region features complete black or single grayscale and zero texture. For the identification of high-priority protected sub-regions, edge detection and connected component analysis are used to identify digital regions in the monitor screen and strong-edge lesion contour regions in the ultrasound screen.

[0230] The encoding parameter configuration is completely consistent with that of Example 7. Testing shows that the region classification accuracy of this solution is no less than 95%, and the single-frame processing latency is no more than 1ms, fully meeting the low-latency requirements. The peak signal-to-noise ratio of the ultrasound image reaches 38dB, similar to the effect of deep learning solutions, and it is suitable for low-end devices with limited computing power.

[0231] Example 11: Figure 8 This is a schematic diagram of an adaptive encoding control device for remote surgical multimodal images provided in an embodiment of this application. The device includes: The data acquisition module 710 is used to acquire the surgical video frame sequence in picture-in-picture mode in real time; wherein, the surgical video frame sequence is generated by superimposing the main screen with at least one type of auxiliary modal screen; The multimodal region recognition module 720 is used to perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region; wherein, the image attribute information includes bounding box coordinates and modal type; The encoding parameter determination module 730 is used to determine the encoding control parameters corresponding to each screen area based on the current available network bandwidth and screen attribute information; The video differential encoding module 740 is used to perform differential encoding on the current frame based on the encoding control parameters of each screen region, and output the encoded video bitstream.

[0232] This application provides an adaptive encoding control device for multimodal images in remote surgery. The device includes: a data acquisition module for real-time acquisition of a surgical video frame sequence in picture-in-picture mode; wherein the surgical video frame sequence is generated by superimposing a main image with at least one type of auxiliary modal image; a multimodal region recognition module for real-time image perception of the current frame in the surgical video frame sequence, identifying each independent image region in the current frame, and determining the image attribute information of each image region; wherein the image attribute information includes bounding box coordinates and modal type; an encoding parameter determination module for determining the encoding control parameters corresponding to each image region based on the available network bandwidth and image attribute information; and a video differential encoding module for differential encoding of the current frame based on the encoding control parameters of each image region, and outputting the encoded video stream. The technical solution provided in this embodiment achieves independent encoding control of each region in the multimodal composite image by real-time sensing of the modal type and bounding box coordinates of each independent image region after picture-in-picture synthesis, and by determining differentiated encoding control parameters for each region in combination with the available network bandwidth. This solves the problem of incompatibility between existing encoding methods and multimodal composite images, improves the transmission fidelity of key clinical information under limited bandwidth and the targeted allocation of encoding resources, thereby ensuring the operational safety and stability of remote surgery.

[0233] Based on the above-mentioned device, optionally, a multimodal region recognition module 720 is used to: input the current frame into a shared feature extraction network, and determine the bounding box coordinates of each independent screen region in the current frame based on the region detection output head of the shared feature extraction network; capture the region map corresponding to each screen region in the current frame, and input each region image into a modal classification network to determine the confidence level of each region image belonging to each modal type included in the current picture-in-picture mode; determine the category with the highest confidence level as the modal type of the corresponding screen region; and generate screen attribute information of each screen region based on each bounding box coordinate and each modal type.

[0234] Based on the above-mentioned device, the adaptive encoding control device may optionally include: a control parameter reset module, used to obtain the classification confidence of each frame region; when the classification confidence of each region is lower than a preset threshold, clear all differential encoding configurations of the current frame and reset each encoding control parameter to the parameter value of the global unified encoding mode, and control the encoder to switch to the global unified encoding mode to output the video bitstream; and / or, obtain the manually specified protection region and its encoding parameters, merge the manually specified protection region with each independent frame region, and when there is overlap, overwrite the corresponding encoding control parameters with the manually specified encoding parameters, and perform differential encoding on the current frame based on the overwritten encoding control parameters.

[0235] Based on the above-mentioned device, optionally, the multimodal region recognition module 720 is further used to obtain prior region configuration information corresponding to the current frame; wherein, the prior region configuration information includes the preset position of each candidate image region in the current frame and the corresponding preset modality type; based on the preset position, the candidate bounding box coordinates of each candidate image region are determined; edge detection is performed on the current frame, and the candidate bounding box coordinates are corrected based on the edge detection results to determine the bounding box coordinates of each image region; based on each bounding box coordinate, the region image is cropped, and the image features of the region image are extracted and matched with the preset feature templates of each modality type to determine the modality type of each image region; based on each bounding box coordinate and each modality type, image attribute information of each image region is generated.

[0236] Based on the above-mentioned device, optionally, the encoding parameter determination module 730 includes: an encoding quantization parameter determination unit, used to determine the reference quantization parameters of the entire frame based on the current available network bandwidth; determine the modal differentiation offset of each frame region based on the modal type in the frame attribute information and according to the preset mapping rules; and determine the encoding quantization parameters corresponding to each frame region based on the reference quantization parameters and the modal differentiation offsets.

[0237] Based on the above-mentioned device, optionally, the encoding quantization parameter determination unit is used to scale and adjust the modal differentiation parameter offset based on the current available network bandwidth to determine the bandwidth adaptive parameter offset of each screen region; and to determine the encoding quantization parameter corresponding to each screen region based on the reference quantization parameter and the bandwidth adaptive parameter offset of each screen region.

[0238] Based on the above-mentioned device, optionally, the encoding quantization parameter determination unit is further used to superimpose a preset negative offset on the modal differentiation parameter offset of the screen area to which the high-priority protection sub-region belongs when the screen attribute information contains a high-priority protection sub-region, so as to determine the modal differentiation parameter offset of the high-priority protection sub-region.

[0239] Based on the above-mentioned device, optionally, the encoding parameter determination module 730 includes: a refresh cycle determination unit, used to determine the screen area that needs to be refreshed within the frame and the forced intra-frame refresh cycle based on the modal type in the screen attribute information.

[0240] Based on the above-mentioned device, optionally, a refresh cycle determination unit is used to determine the areas in the image attribute information whose modal type is monitor image and ultrasound-assisted image as the image areas that need to be refreshed within a frame; determine the refresh cycle of the waveform area in the monitor image as the first cycle; determine the refresh cycle of the entire ultrasound-assisted image area as the second cycle; wherein the second cycle is greater than the first cycle; and perform staggered refresh frame allocation for each image area that needs to be refreshed within a frame so that the refresh frames of each image area do not overlap in time.

[0241] Based on the above-mentioned device, the adaptive coding control device may optionally include: a high-priority area processing module, used to identify key value display areas in the monitor screen when the modal type is a monitor screen, and determine the key value display areas as high-priority protection sub-regions; when the modal type is an ultrasound-assisted screen, identify lesion contour areas in the ultrasound-assisted screen, and determine the lesion contour areas as high-priority protection sub-regions; add the high-priority protection sub-regions to the screen attribute information of the corresponding screen area; determine the forced intra-frame refresh cycle of the high-priority protection sub-regions in the monitor screen as the third cycle; wherein the third cycle is less than the refresh cycle of the waveform area in the monitor screen; determine the forced intra-frame refresh cycle of the high-priority protection sub-regions in the ultrasound-assisted screen as the fourth cycle; wherein the fourth cycle is between the third cycle and the refresh cycle of the entire ultrasound-assisted screen area.

[0242] Based on the above-mentioned device, optionally, a video differential coding module 740 is used to sort each screen region according to a preset clinical priority order, and select a preset number of screen regions with the highest priority based on the sorting result; convert the bounding box coordinates and coding quantization parameters of each selected screen region into a region of interest configuration instruction compatible with the encoder programming interface; write the region of interest configuration instruction into the encoder to control the encoder to encode macroblocks in each screen region with the corresponding coding quantization parameters, and when the forced intra-frame refresh period in the coding control parameters is reached, force the macroblocks in the corresponding screen region to be encoded using intra-frame prediction mode, and output the encoded video bitstream.

[0243] Based on the above-mentioned device, optionally, the video differential coding module 740 is further used to obtain the newly generated region of interest configuration when a change in the picture-in-picture layout is detected; within a preset transition time window, the current region of interest configuration is gradually transitioned to the newly generated region of interest configuration, so that the quantization parameter offset of each picture region gradually changes from the current value to the target value; and the region of interest configuration parameters of the encoder are updated frame by frame within the transition time window so that the quantization parameter distribution of the current frame changes smoothly in the time dimension.

[0244] Based on the above device, optionally, the video differentiation coding module 740 is further used to monitor the configuration status of the region of interest returned by the encoder; when the configuration status indicates configuration failure or encoding error multiple times in a row, clear all region of interest configurations and control the encoder to fall back to the global unified coding mode; trigger an alarm signal to prompt the encoder status to be checked.

[0245] Based on the above-mentioned device, the adaptive coding control device may optionally include: a bitrate overrun processing module, used to monitor the actual bitrate output by the encoder in real time; when the actual bitrate of multiple consecutive frames exceeds the target bitrate, a bitrate overrun event is fed back; in response to the bitrate overrun event, the bitrate budget is gradually recovered according to a preset clinical priority order until the actual bitrate converges to the target bitrate range. The preset clinical priority order includes: first reducing the positive offset of non-clinical information areas, then reducing the negative offset of the endoscope main screen, then reducing the negative offset of the ultrasound-assisted screen, and finally keeping the negative offset of the high-priority protection sub-regions in the monitor screen and the ultrasound-assisted screen unchanged.

[0246] The adaptive encoding control device for remote surgical multimodal images provided in this application can execute the adaptive encoding control method for remote surgical multimodal images provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0247] It is worth noting that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0248] Example 13: Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 A block diagram is shown of an exemplary electronic device 80 suitable for implementing embodiments of the present application. Figure 9 The electronic device 80 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Figure 9 As shown, the electronic device 80 is presented in the form of a general-purpose computing device. The components of the electronic device 80 may include, but are not limited to: one or more processors or processing units 801, system memory 802, and bus 803 connecting different system components (including system memory 802 and processing unit 801).

[0249] Bus 803 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Microchannel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus. Electronic device 80 typically includes various computer system readable media. These media can be any available media that can be accessed by electronic device 80, including volatile and non-volatile media, removable and non-removable media. System memory 802 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 807 and / or cache memory 805. Electronic device 80 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 806 can be used to read and write non-removable, non-volatile magnetic media (… Figure 9 Not shown; usually referred to as a "hard drive"). Although Figure 9Not shown, a disk drive for reading and writing to removable non-volatile disks (e.g., "floppy disks") and an optical disk drive for reading and writing to removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 803 via one or more data media interfaces. Memory 802 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application. A program / utility 808 having a set (at least one) of program modules 807 may be stored, for example, in memory 802. Such program modules 807 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 808 typically perform the functions and / or methods described in the embodiments of this application. Electronic device 80 can also communicate with one or more external devices 809 (e.g., keyboard, pointing device, display 810, etc.), and with one or more devices that enable a user to interact with the electronic device 80, and / or with any device that enables the electronic device 80 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 811. Furthermore, electronic device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 812. As shown, network adapter 812 communicates with other modules of electronic device 80 via bus 803. It should be understood that, although... Figure 9 Not shown, it can be combined with electronic device 80 to use other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0250] The processing unit 801 executes various functional applications and page processing by running programs stored in the system memory 802, such as implementing the adaptive encoding control method for remote surgical multimodal images provided in the embodiments of this application.

[0251] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0252] Note that the above description is merely a preferred embodiment and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this application, and the scope of this application is determined by the scope of the appended claims.

Claims

1. An adaptive coding control method for multimodal images in remote surgery, characterized in that, The method includes: Real-time acquisition of surgical video frame sequences in picture-in-picture mode; wherein, the surgical video frame sequence is generated by superimposing the main screen with at least one type of auxiliary modal screen; Real-time image perception is performed on the current frame in the surgical video frame sequence to identify each independent image region in the current frame and determine the image attribute information of each image region; wherein, the image attribute information includes bounding box coordinates and modal type; Based on the current available network bandwidth and the image attribute information, determine the encoding control parameters corresponding to each image region; Based on the encoding control parameters of each of the aforementioned screen regions, the current frame is differentially encoded, and the encoded video stream is output.

2. The method according to claim 1, characterized in that, The step of performing real-time image perception on the current frame in the surgical video frame sequence, identifying each independent image region in the current frame, and determining the image attribute information of each image region includes: The current frame is input into a shared feature extraction network, and the bounding box coordinates of each independent image region in the current frame are determined based on the region detection output head of the shared feature extraction network. Extract the corresponding region map of each of the aforementioned screen regions in the current frame, and input each of the aforementioned region images into a modality classification network to determine the confidence level of each of the aforementioned region images belonging to each of the various modal types included in the current picture-in-picture mode; The category with the highest confidence level is determined as the modality type of the corresponding image area; Based on the bounding box coordinates and modal types, image attribute information for each image region is generated.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the classification confidence score of each of the aforementioned image regions. When the classification confidence scores of all regions are lower than a preset threshold, clear all differential encoding configurations for the current frame, reset the encoding control parameters of each region to the parameter values ​​of the globally unified encoding mode, and control the encoder to switch to the globally unified encoding mode to output the video stream; and / or, Obtain the manually specified protected area and its encoding parameters, merge the manually specified protected area with each independent screen area, when there is overlap, overwrite the corresponding encoding control parameters with the manually specified encoding parameters, and perform differential encoding on the current frame based on the overwritten encoding control parameters.

4. The method according to claim 1, characterized in that, The step of performing real-time image perception on the current frame in the surgical video frame sequence, identifying each independent image region in the current frame, and determining the image attribute information of each image region includes: Obtain the prior region configuration information corresponding to the current frame; wherein, the prior region configuration information includes the preset position of each candidate image region in the current frame and the corresponding preset modal type; Based on the preset position, determine the candidate bounding box coordinates of each candidate image region; Edge detection is performed on the current frame, and the coordinates of the candidate bounding boxes are corrected based on the edge detection results to determine the bounding box coordinates of each of the image regions; Based on the coordinates of each bounding box, the image of the region is cropped, and the image features of the region image are extracted and matched with the preset feature templates of each modality type to determine the modality type of each image region. Based on the bounding box coordinates and modal types, image attribute information for each image region is generated.

5. The method according to claim 1, characterized in that, The encoding control parameters include encoding quantization parameters. Determining the encoding control parameters corresponding to each of the picture regions based on the current available network bandwidth and the picture attribute information includes: Based on the currently available network bandwidth, determine the baseline quantization parameters for the entire frame. Based on the modal type in the image attribute information, the modal differentiation offset of each image region is determined according to a preset mapping rule; Based on the baseline quantization parameters and the modal differential offsets, the encoding quantization parameters corresponding to each of the image regions are determined.

6. The method according to claim 5, characterized in that, The step of determining the encoding quantization parameters corresponding to each of the image regions based on the benchmark quantization parameters and the modal differential offsets further includes: Based on the current available network bandwidth, the modal differentiation parameter offset is scaled and adjusted to determine the bandwidth adaptive parameter offset for each of the image regions. Based on the baseline quantization parameters and the bandwidth adaptive parameter offset of each of the picture regions, the coding quantization parameters corresponding to each of the picture regions are determined.

7. The method according to claim 5, characterized in that, The step of determining the modal differentiation parameter offset of each of the image regions based on the modal type in the image attribute information further includes: When the image attribute information includes a high-priority protection sub-region, a preset negative offset is superimposed on the modal differentiation parameter offset of the image region to which the high-priority protection sub-region belongs, so as to determine the modal differentiation parameter offset of the high-priority protection sub-region.

8. The method according to claim 1, characterized in that, The encoding control parameters include a forced intra-frame refresh period. The step of determining the encoding control parameters corresponding to each of the picture regions based on the current available network bandwidth and the picture attribute information includes: Based on the modal type in the image attribute information, the image area that needs to be refreshed within the frame and the forced intra-frame refresh period are determined.

9. The method according to claim 8, characterized in that, The step of determining the screen area requiring intra-frame refresh and the forced intra-frame refresh period based on the modal type in the screen attribute information includes: The areas in the image attribute information whose modal type is monitor image and ultrasound-assisted image are identified as the image areas that need to be refreshed within the frame. The refresh cycle of the waveform area in the monitor screen is defined as the first cycle; The refresh cycle of the entire area of ​​the ultrasound-assisted image is defined as the second cycle; wherein the second cycle is greater than the first cycle; The refresh frames for each screen area that needs to be refreshed within a frame are staggered to ensure that the refresh frames of each screen area do not overlap in time.

10. The method according to claim 1, characterized in that, After determining the image attribute information of each of the image regions, the method further includes: When the modal type is a monitor screen, identify the key value display area in the monitor screen and determine the key value display area as a high-priority protection sub-area; When the modality type is ultrasound-assisted image, the lesion contour region in the ultrasound-assisted image is identified, and the lesion contour region is determined as a high-priority protection sub-region; Add the high-priority protected sub-region to the corresponding screen attribute information of the screen region; Accordingly, the encoding control parameters include a forced intra-frame refresh period, and the method further includes: The forced intra-frame refresh cycle of the high-priority protection sub-region in the monitor screen is determined as the third cycle; wherein, the third cycle is less than the refresh cycle of the waveform region in the monitor screen; The forced intra-frame refresh cycle of the high-priority protected sub-region in the ultrasound-assisted image is determined as the fourth cycle; wherein, the fourth cycle is between the third cycle and the refresh cycle of the entire ultrasound-assisted image region.

11. The method according to claim 1, characterized in that, The step of performing differential encoding on the current frame based on the encoding control parameters of each of the aforementioned screen regions, and outputting the encoded video stream, includes: The screen regions are sorted according to a preset clinical priority order, and a preset number of screen regions with the highest priority are selected based on the sorting results. The bounding box coordinates and encoding quantization parameters of each selected image region are converted into region of interest configuration instructions compatible with the encoder programming interface; The region of interest configuration instruction is written into the encoder to control the encoder to encode macroblocks in each of the picture regions with the corresponding encoding quantization parameters. When the forced intra-frame refresh period in the encoding control parameters is reached, the macroblocks in the corresponding picture region are forced to be encoded using the intra-frame prediction mode, and the encoded video stream is output.

12. The method according to claim 11, characterized in that, The method further includes: When a change in the picture-in-picture layout is detected, the newly generated region of interest configuration is obtained; Within a preset transition time window, the current region of interest configuration is gradually transitioned to the newly generated region of interest configuration, so that the quantization parameter offset of each of the image regions gradually changes from the current value to the target value; The region of interest configuration parameters of the encoder are updated frame by frame within the transition time window so that the quantization parameters of the current frame change smoothly in the time dimension.

13. The method according to claim 11, characterized in that, The method further includes: Monitor the region of interest configuration status returned by the encoder; When the configuration status indicates configuration failure or encoding error multiple times consecutively, clear all region of interest configurations and control the encoder to fall back to the global unified encoding mode; Trigger an alarm signal to prompt you to check the encoder status.

14. The method according to claim 1, characterized in that, The method further includes: Real-time monitoring of the actual bit rate output by the encoder; When the actual bitrate exceeds the target bitrate for multiple consecutive frames, a bitrate exceeding event is reported. In response to the bitrate exceeding event, the bitrate budget is recovered step by step according to the preset clinical priority order until the actual bitrate converges to the target bitrate range; The preset clinical priority order includes: first reducing the positive offset of non-clinical information areas, then reducing the negative offset of the endoscope main screen, then reducing the negative offset of the ultrasound-assisted screen, and finally keeping the negative offset of the high-priority protection sub-area in the monitor screen and the high-priority protection sub-area in the ultrasound-assisted screen unchanged.

15. An adaptive encoding control device for multimodal images in remote surgery, characterized in that, The device includes: The data acquisition module is used to acquire the surgical video frame sequence in picture-in-picture mode in real time; wherein, the surgical video frame sequence is generated by superimposing the main screen with at least one type of auxiliary modal screen; A multimodal region recognition module is used to perform real-time image perception on the current frame in the surgical video frame sequence, identify each independent image region in the current frame, and determine the image attribute information of each image region; wherein, the image attribute information includes bounding box coordinates and modal type; The encoding parameter determination module is used to determine the encoding control parameters corresponding to each of the screen regions based on the current available network bandwidth and the screen attribute information; The video differential encoding module is used to perform differential encoding on the current frame based on the encoding control parameters of each of the aforementioned screen regions, and output the encoded video bitstream.

16. An electronic device, characterized in that, Electronic devices include: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the adaptive coding control method for remote surgical multimodal images according to any one of claims 1-14.

Images