Multi-party visiting equipment and system based on VR panoramic visiting

Abstract

The embodiment of the application discloses a multi-party visiting device and system based on VR panoramic visiting, and relates to the technical field of medical visiting. The multi-party visiting device comprises: a ward end mobile visiting vehicle, which is provided with: a panoramic camera, which is used for collecting panoramic video of a ward in real time and supporting remote control of panoramic view angle rotation; a voice transmission module, which is integrated with a directional microphone and a noise reduction module, and is used for realizing multi-party intercom of patients, visitors and medical staff; a video communication module, which is used for realizing multi-party remote video communication of patients, visitors and medical staff; an information storage module, which is used for automatically storing visiting information; a handheld device, which is used for providing visiting reservation service and remote video service for visitors; and a visiting end VR device, which is used for providing VR video service for visitors. The embodiment realizes remote visiting by using VR technology and provides real visiting experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of medical visitation technology, and in particular to a multi-party visitation device and system based on VR panoramic visitation. Background Technology

[0002] With the increasing prevalence of cross-regional medical treatment, the need for emotional connection between patients and their families is becoming more prominent. However, traditional visitation models have the following problems:

[0003] 1. Due to geographical distance, family members in different locations find it difficult to make effective visits, and critically ill patients often lack family emotional support during their treatment, which hinders their psychological recovery process.

[0004] 2. Existing video visitation systems only support fixed-view calls, preventing family members from observing the entire ward and leading to significant biases in their understanding of the patient's actual condition;

[0005] 3. Frequent visits by family members to and from the hospital increase the burden on patient guidance, and medical staff are largely occupied by unnecessary visits, affecting the quality of core medical services. Summary of the Invention

[0006] This invention provides a multi-party viewing device and system based on VR panoramic viewing to solve at least one of the above problems.

[0007] In a first aspect, embodiments of the present invention provide a multi-party viewing device based on VR panoramic viewing, comprising:

[0008] The mobile patient visitation vehicle is equipped with: a panoramic camera for real-time acquisition of panoramic video of the ward and supports remote control of panoramic view rotation; a voice transmission module integrating a directional microphone and noise reduction module for multi-party communication between patients, visitors, and medical staff; a video communication module for multi-party remote video communication between patients, visitors, and medical staff, and providing a video display interface for patients; an information storage module for automatically storing visitation information; and an onboard control computer for controlling the coordinated operation of the panoramic camera, voice transmission module, video communication module, and information storage module.

[0009] The visitation terminal handheld device is used to provide visitation appointment services and remote video services for visitors;

[0010] VR devices for visiting patients are used to provide VR video services to visitors. Visitors can choose to receive audio and video from the ward through handheld devices or VR devices.

[0011] In this process, audio and video data from each end are transmitted via the SRT protocol, and the transmission bandwidth is periodically adjusted as follows: based on the changes in bandwidth allocated in the recent period, the short-term bandwidth in the future is predicted; the current bandwidth allocated by the SRT protocol is corrected based on the short-term bandwidth, and the corrected bandwidth is used as the transmission bandwidth for this period.

[0012] Secondly, embodiments of the present invention provide a multi-party viewing device based on VR panoramic viewing, comprising:

[0013] The aforementioned multi-party visitation equipment, as well as the cloud platform;

[0014] During system use, visitors submit identification via the handheld device, and the cloud platform verifies kinship.

[0015] After verification, the visitor selects a time slot through the handheld device, and medical staff will move the visiting vehicle to the patient's bed during the time slot.

[0016] Once the vehicle is in place, a panoramic video stream is activated, and a VR control interface is generated on the handheld device. Visitors can adjust the viewing angle by swiping their hands and conduct multi-party video and voice communication through the handheld device or VR device.

[0017] After the visit ends, the information storage module automatically stores the visit information.

[0018] Thirdly, embodiments of the present invention provide a multi-party visitation method based on VR panoramic visitation, which is applied to the cloud platform of the aforementioned multi-party visitation system, and the visitation vehicle has multiple units;

[0019] The method includes:

[0020] Based on the preset visitation duration, the visitation time of each vehicle is divided into multiple reservation time slots per day. Initially, the reservation time slots of each vehicle are distributed in the same way.

[0021] For each mobile visitor: if a visit ends early, the continuous time slots that are not booked after the end time will be re-divided into multiple booking time slots starting from the end time; and the final set of booking time slots corresponding to each mobile visitor will be displayed to the visitor.

[0022] In response to the target appointment time selected by the visitor in the new visitation request, determine whether there are any medical orders for this visit based on the patient's condition. If there are medical orders, medical staff need to be present during the visit; otherwise, medical staff do not need to be present during the visit.

[0023] Based on the distribution patterns of single visitation video traffic in historical data with and without medical orders, the traffic distribution of visitation video after the new visitation request is accessed is predicted.

[0024] Based on the predicted traffic distribution of other visitation vehicles' booked visitation tasks within the same time period, determine whether the maximum traffic of all visitation vehicles within the same time period exceeds the threshold limit after the new visitation request is received.

[0025] If the time limit is not exceeded, the new visitation request will be assigned to the target visitation vehicle corresponding to the target appointment time slot, and the appointment will be successful;

[0026] If the limit is exceeded, the unbooked appointment time slots of the target visitation vehicle starting from the target appointment time slot will be sequentially postponed until the maximum flow of all visitation vehicles in the same time slot as the postponed target appointment time slot falls within the upper limit of the threshold.

[0027] In summary, this invention provides a multi-party visitation device and system based on VR panoramic visitation. Utilizing VR technology, through high-precision tracking, low-latency panoramic video, spatial audio, and natural interaction, it significantly enhances the immersion, realism, and user experience of remote visitation, making it suitable for medical visitation scenarios. Specifically, the visitation vehicle uses a 360° panoramic camera to capture real-time high-definition video streams from the ward, supporting remote control of panoramic view rotation, breaking through the limitations of traditional fixed perspectives; it integrates a multi-directional audio system to enable multi-party communication between family members, medical staff, and patients, supporting simultaneous participation by multiple people and significantly improving the interactive experience; it enables remote real-time hospital visits through a video communication module; and it stores historical records through an information storage unit. The handheld device provides visitors with an online channel for submitting and verifying family member identities, displays available visitation slots, and transforms the panoramic video stream into an immersive screen operable by the handheld device, supporting rotation and swiping to view the ward panorama, and features multi-person conversation management, allowing three devices to simultaneously access and conduct voice conversations; the VR device provides an immersive visitation experience through VR technology, helping visitors observe the entire ward and understand the patient's actual condition. Attached Figure Description

[0028] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of a multi-party viewing system based on VR panoramic viewing provided in an embodiment of the present invention;

[0030] Figure 2 This is a technical architecture diagram of a multi-party visitation system based on VR panoramic visitation provided by an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of a multi-party exploration method based on VR panoramic exploration provided by an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0033] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0034] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] Figure 1 This is a schematic diagram of a multi-party viewing system based on VR panoramic viewing provided in an embodiment of the present invention. Figure 1 As shown, the system includes: multi-party visitation equipment based on VR (Virtual Reality) panoramic visitation, and a cloud platform. The multi-party visitation equipment includes a mobile visitation vehicle at the ward end, handheld devices at the visitation end, and VR devices at the visitation end.

[0036] The vehicle is freely movable and is equipped with a panoramic camera, voice transmission module, video communication module, information storage module, and onboard control computer.

[0037] The panoramic camera is used to capture 360° panoramic video of the ward in real time and supports remote control of the panoramic view rotation. Optionally, the panoramic camera consists of a ring array of two ultra-wide-angle lenses, which generates an 8K resolution seamless video stream through a parallax compensation algorithm. It supports 360° horizontal rotation and -30° to +45° vertical tilt adjustment, and supports intelligent supplemental lighting in low-light environments.

[0038] The voice transmission module integrates a directional microphone and a noise reduction module to enable multi-party communication between patients, visitors, and medical staff. Optionally, the voice transmission module uses beamforming technology to achieve directional sound pickup and is equipped with one noise-reducing microphone and two high-fidelity speakers, supporting real-time clear communication between family members, patients, and medical staff.

[0039] The video communication module enables multi-party remote video communication between patients, visitors, and medical staff, and provides a video display interface for patients.

[0040] The information storage module is used to automatically store visitation information for easy viewing later.

[0041] The onboard control computer is used to control the panoramic camera, voice transmission module, video communication module and information storage module to work together.

[0042] The handheld device for visitation is used to provide visitors with visitation appointment services and remote video services. The handheld device can be a mobile phone or tablet, and provides a user interface through an APP (Application) or web page. Optionally, the interface includes a smart appointment module, which provides a channel for visitors to submit their identity verification and displays available visitation slots; at the same time, it can also convert the panoramic video stream into an immersive screen that can be operated by the handheld device through a VR control module, supporting functions such as rotating and sliding to view the panoramic view of the ward, multi-person conversation management, etc., allowing three devices to access and conduct voice conversations simultaneously.

[0043] The VR device for visitation is used to provide VR video services to visitors. Visitors can choose to receive audio and video from the ward via either a handheld device or a VR device. Optionally, the VR device can be a VR head-mounted display device supporting 6 degrees of freedom tracking. This device is equipped with an LCD (Liquid Crystal Display) with a single-eye resolution of no less than 2160×2160, providing a horizontal field of view of approximately 105°. It has a built-in high-precision IMU (Inertial Measurement Unit) and an Inside-Out tracking system, and achieves sub-millimeter-level spatial positioning through four cameras. After receiving the video stream, the VR device decodes and renders it in the viewport, and dynamically renders the corresponding screen area in combination with IMU sensor data to enhance the immersive experience.

[0044] In one specific embodiment, the VR device integrates stereo speakers and a dual-microphone array with spatial audio technology, supporting noise suppression and clear voice pickup. Combined with an ergonomic design, it ensures comfort during extended use. Simultaneously, the VR device supports Wi-Fi 6, guaranteeing stable, low-latency transmission of high-definition video streams. Furthermore, the VR device incorporates the Pico 4 gesture recognition SDK (Software Development Kit). This SDK, based on a convolutional neural network using multi-frame image recognition and depth estimation, identifies hand gestures by calculating inter-frame differences and hand key point positions; it then maps the recognition results to virtual button inputs, allowing visitors to control button clicks and other functions during visits using natural movements.

[0045] Optionally, in the video display interface of each party, the main speaker's image occupies the central area of ​​the screen, and its image is automatically enlarged to 80% of the screen area, while the images of other participants are arranged in a ring at the bottom edge.

[0046] Furthermore, the multi-party visitation equipment also includes a medical staff management terminal, which may include handheld devices and VR devices, allowing medical staff to choose either one to access the visitation. Simultaneously, the medical staff management terminal also provides: a background monitoring function, allowing medical staff to view the visitation situation between family members and patients during the visitation period, including video and audio content; and an emergency interruption function, allowing medical staff to instantly terminate the connection in the background.

[0047] Of course, the aforementioned visitation and medical management devices also include voice transmission and video communication modules to enable multi-party audio and video transmission.

[0048] The cloud platform is used for identity verification, establishing encrypted data channels, video storage and scheduling, and other data processing operations.

[0049] Specifically, during system use, visitors submit identification via the handheld device, and the cloud platform verifies kinship. Optionally, visitors fill in identity and appointment information on the handheld device interface and submit an appointment request. Identity information includes the visitor's name, the patient's name, and the bed number to be visited; appointment information includes the appointment date and time slot. Hospital administrators review the application through the cloud platform, and upon approval, the system records the appointment. Visitors can initiate a visitation call request within the appointed time slot on the appointed date. The cloud platform is equipped with a kinship authentication module, which connects to the hospital information system and supports both automatic system review and manual verification to ensure secure and reliable visitation permissions.

[0050] After verification, the visitor selects a time slot via the handheld device, and medical staff moves the visiting vehicle to the patient's bed during that time. The visitor can initiate a call via a mobile app or VR device. To ensure security, the call message must carry a dynamic security key generated based on a timestamp, and the request protocol must be HTTPS to access microphone and camera permissions. The visiting devices (including handheld devices and VR devices) must also be connected to a WebSocket service and have a heartbeat mechanism activated. The signaling server on the cloud platform acts as a scheduling center, responsible for managing session connection information.

[0051] In response to a visitor's call request, the cloud platform generates a chat room serial number and sends it to the deployed visitation vehicle. Specifically, after receiving the call request, the signaling server generates a chat room serial number and sends a chat room authentication identifier and connection information to the visitor.

[0052] The visiting cart displays the call request on the patient's video display interface; after clicking "accept," the patient and visitor establish a connection and join the chat room based on the chat room serial number. Specifically, after the visiting cart logs in, if it receives a call request and clicks "accept," the signaling server notifies the visitor of the reception status, and both parties establish a connection and join the chat room based on the chat room serial number.

[0053] Upon joining the chat room, the mobile visitation vehicle automatically activates its core hardware (including a panoramic camera and a directional microphone array), starts a panoramic video stream, and pushes the audio and video to the chat room. Simultaneously, a VR control interface is generated on the handheld device; the visitor can adjust the viewing angle by using the VR control interface or by swiping a specific area using gesture recognition on the VR device, and can also conduct multi-party video and voice communication through the handheld device or VR device.

[0054] During the visit, the system initiates a countdown mechanism, allowing either party to actively hang up. After the session ends, all devices exit the chat room and terminate the audio and video streams. The information storage module and / or cloud platform can automatically store the visit information.

[0055] During the chat, healthcare workers or other authorized third parties can apply for a token to join the chat room. Once joined, they can receive all audio and video streams and speak, supporting real-time multi-party interaction. Third-party devices can leave at any time without affecting the main call flow. If the chat ends, the system automatically removes the third-party device, completing the loop. The entire technical approach is as follows: Figure 2 As shown.

[0056] Furthermore, to achieve a lower latency remote VR communication experience, this embodiment adopts a panoramic video transmission method based on the SRT (Secure Reliable Transport) protocol. SRT is a secure transmission protocol built on top of UDP (User Datagram Protocol), featuring low latency and packet loss resistance. Its sending and receiving ends achieve data reliability through automatic retransmission requests and selective retransmission mechanisms.

[0057] In particular, to meet the transmission requirements of "low latency, low packet loss, and high stability" in remote VR visitation scenarios, this embodiment proposes a bandwidth prediction-driven adaptive latency optimization method based on the traditional SRT protocol, and makes improvements in bandwidth estimation, dynamic buffer control, and packet loss repair.

[0058] Specifically, traditional SRT protocols use exponentially weighted averaging to smooth bandwidth. However, in hospital Wi-Fi or public network environments, bandwidth fluctuates drastically, and smoothing alone cannot provide a timely response. Therefore, this embodiment introduces a bandwidth prediction term to adjust the transmission bandwidth of audio and video data during visitation in the following way:

[0059] First, based on the recent changes in bandwidth allocated to the system, predict the short-term bandwidth in the future. Optionally, using the bandwidth data allocated in the recent period as sample data, fit a linear model of bandwidth changing over time using the least squares method; then extend the linear model to a short period in the future to predict the short-term bandwidth for that period.

[0060] Then, the current bandwidth allocated by the SRT protocol is corrected based on the short-term bandwidth, and the corrected bandwidth is used as the transmission bandwidth for this cycle. Optional:

[0061]

[0062] in, This indicates the corrected bandwidth. This represents the instantaneous bandwidth allocated by the SRT protocol. Indicates future short-term bandwidth. This represents the correction factor. You can take 0.15.

[0063] The above operations can be performed periodically by each audio / video transmitter and receiver in the system to achieve dynamic adjustment of transmission bandwidth. Furthermore, a dynamically adjustable latency control method is also implemented based on network characteristics, as follows:

[0064] If a predicted short-term bandwidth drop indicates insufficient bandwidth, the cloud platform will trigger a bitrate reduction strategy, such as reducing the resolution of the transmitted video.

[0065] If the packet loss rate increases, it indicates that there are missing frames or missing pixels in the received video data. The cloud platform triggers an interpolation compensation strategy, that is, to make up for the lost data through interpolation.

[0066] If the latency increases, it indicates that some frames are lagging. In this case, the cloud platform will trigger a buffering playback strategy, which means that it will wait until the last frame is received before playing the video to avoid video stuttering.

[0067] The cloud platform returns the triggered strategy to the video sending end or video playback end, enabling the video sending end or video playback end to send or play the video according to the corresponding strategy, thereby achieving dynamic smoothing of the video stream. The video sending end includes the video communication module in the mobile hospital vehicle or the video sending module of the cloud platform; the video playback end includes the video display interface in the mobile hospital vehicle, as well as the handheld device or VR device.

[0068] Optionally, the AES (Advanced Encryption Standard)-128 encryption algorithm can be used for end-to-end encryption of audio and video data, supporting custom keys and dynamic key update mechanisms to ensure data security during transmission.

[0069] In summary, this embodiment provides a multi-party visitation system based on VR panoramic visitation, including a cloud platform and multi-party visitation devices based on VR panoramic visitation. The multi-party visitation devices include a mobile visitation vehicle at the ward end, a handheld device at the visitation end, and a VR device at the visitation end. The visitation vehicle uses a 360° panoramic camera to capture real-time high-definition video streams from the ward, supporting remote control of panoramic view rotation, breaking through the limitations of traditional fixed perspectives; it integrates a multi-directional audio system to enable multi-party communication between family members, medical staff, and patients, supporting simultaneous participation of multiple people and significantly improving the interactive experience; it enables remote real-time hospital visits through a video communication module; and it stores historical records through an information storage unit. The handheld device provides visitors with an online channel for submitting and verifying family member identities, displays available visitation slots, converts the panoramic video stream into an immersive screen operable by the handheld device, supports rotating and sliding to view the ward panorama, and features multi-person conversation management, allowing three devices to simultaneously access and conduct voice conversations. The VR device provides an immersive visitation experience through VR technology, helping visitors observe the entire ward and understand the patient's actual condition. In summary, this system utilizes VR technology to significantly enhance the immersion, realism, and user experience of remote visits through high-precision tracking, low-latency panoramic video, spatial audio, and natural interaction. It is suitable for medical visitation scenarios, while the device's enclosed design helps protect patient privacy.

[0070] Based on the above system, Figure 3This is a flowchart illustrating a multi-party visitation method based on VR panoramic viewing, provided by an embodiment of the present invention. It can adjust the reservation time slots of each visitation vehicle in real time according to network conditions, and provide more reservation options without affecting network quality. This method can be executed by the cloud platform in the aforementioned system. Figure 3 As shown, the method specifically includes:

[0071] S110. Based on the preset visitation duration, the visitation time of each visitation vehicle is divided into multiple reservation time slots per day. In the initial state, the reservation time slots of each visitation vehicle are distributed in the same way.

[0072] The visitation duration can be determined by the average visitation duration and adjusted according to the hospital's management needs. For example, the finalized visitation duration could be 10 minutes. Assuming the daily visitation period is from 2:00 PM to 5:00 PM, the daily visitation period can be divided into multiple scheduled time slots of 30 minutes each. The first 10 minutes and the last 10 minutes of each scheduled time slot are used for medical staff to transfer the visitor vehicle, etc., and the middle 10 minutes are the visitation time.

[0073] In this embodiment, the initial allocation of appointment slots for each mobile hospital visit vehicle is consistent. For example, each slot is from 14:00 to 14:30, the second from 14:31 to 15:00, and so on. For ease of description, this embodiment refers to the set of appointment slot distributions for each mobile hospital visit vehicle in the initial state as the initial appointment slot distribution. It should be noted that the above times are only examples and should be adjusted as needed in actual applications.

[0074] S120. Provide visitation services according to the reservation time slots of each visitation vehicle in the initial state, record the maximum visitation traffic in the same time slot, and construct a standard curve of video traffic changing with time in a single visitation.

[0075] Under the aforementioned initial appointment time distribution, the audio and video data from each mobile home visit vehicle are transmitted at relatively similar times. After a period of use, historical data can be used to determine the maximum traffic threshold that ensures video communication quality when all mobile home visit devices initiate visits. For example, if statistics show that the total visit traffic exceeds a certain percentage (e.g., 110%), the communication quality of at least one mobile home visit vehicle will be affected; this value will then be defined as the maximum visit traffic. Furthermore, when this disruption occurs frequently, some mobile home visit devices may need to be temporarily idled to ensure video communication quality.

[0076] Furthermore, after a period of use, a curve showing the video traffic variation over time during a single visit can be constructed based on historical data. This curve reflects the general pattern of video traffic variation over time during most visits, and in this embodiment, it is also referred to as the standard curve. The reason for constructing such a curve is that, in practical applications, it has been found that although the total duration of each visit varies, and the video traffic variation is not entirely the same for each visit, in the early stages of a visit, because the visitor is eager to understand the ward environment, they often need to constantly adjust the video angle. For differentially encoded video transmission protocols, this frequent change in video imagery leads to a peak in video traffic. After understanding the environment, a better video angle is often chosen to communicate with the patient, at which point the video image remains stable, and the video traffic is relatively low. After the visitor and patient have basically finished communicating, depending on the patient's condition, two scenarios can be considered:

[0077] Scenario 1: This visit is a visit with a doctor's order, indicating that medical staff are required to access the visit and communicate the doctor's orders with the patient and family. For this type of visit, this embodiment sets a visit reminder time, for example, at the 7th minute of a 10-minute visit. After this reminder time, the system automatically prompts medical staff to access the visit for doctor's order communication. If the visitor and patient have finished their basic communication before the reminder time has arrived, the visitor or patient can also proactively request medical staff to access the visit. However, the access of medical staff will increase video traffic due to the increased number of video devices, resulting in another video traffic peak.

[0078] Scenario 2: If this visit is without a doctor's order, it means that medical staff are not required to participate in the visit. In this case, both the patient and medical staff can interrupt the current video call.

[0079] Therefore, although each visitation is different, the video traffic changes for a single visitation follow certain patterns; however, the patterns differ between visits with and without a medical order. For example, most visits with a medical order experience two traffic peaks, one at the beginning and one in the middle to late stages of the video, with the second peak often being larger than the first; while most visits without a medical order experience one traffic peak at the beginning of the video, followed by relatively stable video traffic.

[0080] Utilizing the aforementioned patterns, this embodiment categorizes the video traffic curves of single visits accumulated from historical data into two types: those with and without medical orders. Based on the video traffic curves with medical orders from historical data, a standard curve is constructed for the case with medical orders; similarly, a standard curve is constructed for the case without medical orders from historical data. Specifically, the second construction method for both standard curves is the same. Taking the standard curve with medical orders as an example, a univariate polynomial curve or a bimodal Gaussian curve can be fitted based on the characteristic that each video traffic flow has two peaks. The curve parameters are to be determined. Then, a method similar to least squares is used to solve for the undetermined parameters.

[0081] The least squares method refers to solving for model parameters by minimizing the sum of the squares of the differences between the values ​​on each historical curve and the standard curve at the same time. The optimization objective is:

[0082]

[0083] in, The parameters to be optimized are... For indexing historical curves, For each moment in the visit duration, Indicates the first A historical curve at a certain moment The value, This indicates the standard curve at time [time]. The value of .

[0084] In this embodiment, since the duration of each visit and the specific times when the two peaks occur are not exactly the same, the sum of squares of the differences in the least squares method is improved. Instead, it is changed to constrain the standard curve and each historical curve. The modified optimization objective is to find the model parameters by minimizing the sum of distances in Dynamic Time Warping (DW).

[0085]

[0086] in, Indicates the first Historical curves and standard curves distance. Distance focuses more on the similarity of the two curves in terms of their changing patterns, while ignoring differences such as the duration of curve coverage and the lead-lag between curves. Therefore, it can better reflect the general pattern of changes in the survey flow and greatly reduce the influence of factors such as curve length and specific peak position.

[0087] S130. Based on the standard curve, adjust the available time slots for each visitation vehicle.

[0088] Once the maximum visitation volume and standard curve are obtained, the initial appointment time distribution can be dynamically adjusted.

[0089] In one specific implementation, the following operations can be performed on each mobile visitor: If a visit ends early, the continuous time slots that have not been booked after the end time are re-divided into multiple booking slots starting from the end time. Essentially, this involves sequentially bringing forward the remaining time slots after the early-ending slot, thereby utilizing the remaining time of the early-ending slot and increasing the number of booking slots. After this adjustment, the distribution of booking slots for each mobile visitor will no longer be uniform. In this embodiment, after each adjustment, the final set of booking slots for all mobile visitors is displayed to the visitor to provide more booking options.

[0090] Under the new appointment time slot distribution, in response to the appointment time slot selected by the visitor in a new visitation request, it can be determined whether there are any medical orders for this visit based on the patient's condition. This "with medical orders" or "without medical orders" label is set in advance by medical staff based on the patient's condition. For ease of distinction and description, this embodiment refers to the appointment time slot selected by the visitor here as the target appointment time slot.

[0091] If the visit is "with a doctor's order," then based on the historical data showing the distribution pattern of video traffic for a single visit when a doctor's order is present, the video traffic distribution after the new visit request is received is predicted. Specifically, the standard curve for cases with a doctor's order represents the video traffic distribution that will occur after the new visit request is received. Similarly, if the visit is "without a doctor's order," then based on the historical data showing the distribution pattern of video traffic for a single visit when a doctor's order is not present, the video traffic distribution after the new visit request is received is predicted.

[0092] Then, based on the predicted traffic distribution of other visitation vehicle appointments scheduled within the same time period, it is determined whether the maximum traffic of all visitation vehicles within the same time period exceeds the threshold limit (i.e., the maximum visitation traffic determined in S120) after a new visitation request is received. Specifically, based on whether the visitation tasks scheduled by other visitation vehicles have medical orders or not, the standard curves for those with or without medical orders are used as the predicted traffic distribution for that task; based on these predicted traffic distributions, the total traffic of all assigned tasks and the visitation videos that will be generated after the new visitation request is received is calculated for each moment within that time period; the maximum value among the total traffic at all moments is taken as the maximum traffic that can be predicted within the target appointment time period.

[0093] If the maximum predictable traffic within the target reservation period does not exceed the threshold limit, it indicates that the new visitation request is unlikely to affect the video communication quality of the visitation during the same period. Therefore, the new visitation request is assigned to the visitation vehicle belonging to the target reservation period, and the reservation is successful. Specifically, as mentioned above, after the initial reservation period distribution is adjusted, the reservation periods for each visitation vehicle are no longer uniform. Therefore, this embodiment determines the visitation vehicle belonging to the target reservation period and refers to it as the target visitation vehicle. When the target reservation period arrives, the visitation task is completed by the target visitation vehicle.

[0094] If the predicted high traffic volume within the target reservation period exceeds the upper limit of the threshold, indicating that the new visitation request is highly likely to affect the video communication quality of visits during the same period, then all consecutive unreserved reservation periods in the target visitation vehicle starting from the target reservation period will be postponed until, after the postponement, the maximum traffic volume of all visitation vehicles in the same period as the postponed target reservation period is calculated again, and this traffic volume falls within the upper limit of the threshold. That is, if the target reservation period is currently unavailable, then this period and all subsequent consecutive unoccupied reservation periods will be postponed sequentially until the maximum predicted traffic volume in the first postponed period, calculated using the above method, recovers to within the upper limit of the threshold. At this point, the final reservation periods for all visitation vehicles will be displayed to the visitor for them to choose again.

[0095] It should be emphasized that, unless otherwise specified, the reservation time slot in this embodiment is an unoccupied reservation time slot (i.e., the time slot is not assigned a visitation task).

[0096] In summary, this embodiment provides an intelligent scheduling method for appointment slots based on the hospital's network conditions and the distribution patterns of visitation video traffic. While ensuring communication quality, it suggests utilizing the remaining time when a call ends early, enriching the available appointment slots. It also allows for staggered distribution of appointment slots for different visitation vehicles, reducing concurrent traffic peaks and utilizing surplus traffic to handle more visitation requests and vehicles, thereby improving visitation efficiency and equipment utilization. This method optimizes the allocation of hospital visitation resources (including network resources, equipment resources, and time resources), enhancing the hospital's visitation service capabilities.

[0097] Finally, it should be noted that the user data (including patient data and visitor data) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A multi-party peeking method based on VR panorama peeking, characterized in that, The cloud platform is applied to a multi-party visiting system based on a VR panoramic visiting, and the multi-party visiting system further comprises a plurality of ward-end mobile visiting vehicles; The method comprises: According to a preset visiting duration, the visiting time period of each visiting vehicle in a day is divided into a plurality of booking time periods, and the booking time periods of each visiting vehicle are uniformly distributed in an initial state; For each visiting vehicle: in the case that a visit is ended in advance, the continuous time period after the ending time which is not booked is re-divided into a plurality of booking time periods starting from the ending time; and the final corresponding booking time period set of each visiting vehicle is displayed to the visitor; In response to a target booking time period selected by the visitor in a new visiting request, it is determined according to the patient condition whether there is a medical order for this visit, wherein a medical order needs a medical staff to access the visit, and no medical order does not need a medical staff to access the visit; According to the distribution law of the video flow of a single visit in the case of a medical order and a non-medical order in the historical data, the flow distribution of the visiting video after the new visiting request is accessed is predicted; wherein the flow distribution reflects the change law of the video flow with time; According to the predicted flow distribution of the visiting tasks booked by other visiting vehicles in the same time period, it is determined whether the maximum flow of all visiting vehicles in the same time period after the new visiting request is accessed exceeds the threshold upper limit; If not, the new visiting request is assigned to the target visiting vehicle to which the target booking time period belongs, and the booking is successful; If it exceeds, each booking time period of the target visiting vehicle which is not booked starting from the target booking time period is sequentially postponed, until the maximum flow of all visiting vehicles in the same time period as the postponed target booking time period falls within the threshold upper limit.

2. The multi-party peek method of claim 1, wherein, Before the step of predicting the flow distribution of the visiting video after the new visiting request is accessed according to the distribution law of the video flow of a single visit in the case of a medical order and a non-medical order in the historical data, the method further comprises: Setting a medical order reminder node for the visit with a medical order, and reminding the medical staff to access the video for communication when the node is reached, or initiating a medical order communication request to the medical staff by the patient or the visitor; Recording the change curve of the video flow with time of each visit to form historical data; Dividing the historical data into two cases of a medical order and a non-medical order, fitting a standard curve of the change of the video flow with time in a single visit based on the historical data of each case, so that the independent variable range of the standard curve is the preset visiting duration, and the sum of the dynamic time warping distances of the standard curve to each change curve in the historical data of the corresponding case is minimum.

3. The multi-party peek method of claim 1, wherein, The multi-party visiting system comprises a multi-party visiting device, and the multi-party visiting device comprises: The ward end mobile visiting vehicle is provided with a panoramic camera for collecting panoramic video of the ward in real time and supporting remote control of panoramic view rotation, a voice transmission module integrating a directional microphone and a noise reduction module for realizing multi-party intercom of patients, visitors and medical staff, a video communication module for realizing multi-party remote video communication of patients, visitors and medical staff and providing a video display interface for patients, and an information storage module for automatically storing visiting information; and a vehicle-mounted control computer is used for controlling cooperation of the panoramic camera, the voice transmission module, the video communication module and the information storage module. The visiting end handheld device is used for providing visiting reservation service and remote video service for visitors. The visiting end VR device is used for providing VR video service for visitors, and the visitors can select to receive audio and video from the ward through the handheld device or the VR device. The audio and video data of each end are transmitted through the SRT protocol, and the transmission bandwidth is periodically adjusted in the following manner: the short-time bandwidth in the future is predicted according to the bandwidth allocated in the recent period of time; the current bandwidth allocated by the SRT protocol is corrected according to the short-time bandwidth, and the corrected bandwidth is taken as the transmission bandwidth of the current period.

4. The multi-party peek method of claim 3, wherein, The method for predicting the short-time bandwidth in the future according to the bandwidth allocated in the recent period of time comprises: Taking the bandwidth data allocated in the recent period of time as sample data, a linear model of the change of bandwidth with time is fitted by using the least square method; The short-time bandwidth in the future is predicted by using the linear model.

5. The multi-party peek method of claim 3, wherein, The method for correcting the current bandwidth allocated by the SRT protocol according to the short-time bandwidth comprises: wherein, denotes the corrected bandwidth, denotes the current bandwidth allocated by the SRT protocol, denotes the short-time bandwidth, denotes the correction factor.

6. The multi-party peek method of claim 3, wherein, The vertical pitch angle adjustment range of the panoramic camera includes [-30°, +45°], and intelligent light compensation in a low-light environment is supported; In the video display interface of each party, the picture of the main speaker occupies the center area of the screen, and the pictures of other participants are arranged in a ring shape at the edge.

7. The multi-party visiting method according to claim 3, wherein The VR device is equipped with a single-eye high-resolution LCD display screen, a high-precision IMU and an Inside-Out tracking system, and can realize sub-millimeter level spatial positioning through a four-camera lens; After receiving the video stream, the VR device decodes and viewport renders, and dynamically renders the corresponding picture area in combination with IMU data.

8. The multi-party visiting method according to claim 3, wherein During use of the system, the visitor submits an identity certificate through the handheld device, and the cloud platform verifies the kinship; After verification, the visitor selects a reservation period through the handheld device, and the medical staff moves the visiting vehicle to the patient's bed during the reservation period; After the visiting vehicle is positioned, the panoramic video stream is started, a VR control interface is generated in the handheld device, the visitor adjusts the view angle through gesture sliding, and multi-party video and voice communication is realized through the handheld device or the VR device; After the visit is completed, the information storage module automatically stores the visit information.

9. The multi-party peek method of claim 3, wherein, After the visiting vehicle is positioned, the panoramic video stream is started, comprising: In response to the call request of the visitor, the cloud platform generates a chat room serial number and sends it to the visit vehicle; The visit vehicle displays the call request to the patient in the video display interface; after clicking to accept, the patient and the visitor establish a connection and join the chat room according to the chat room serial number; After joining the chat room, the visit vehicle starts panoramic video streaming and pushes audio and video to the chat room.

10. The multi-party peek method of claim 3, wherein, In addition, the method further comprises: If the short-time bandwidth decreases, the cloud platform triggers a code rate reduction strategy; If the packet loss rate rises, the cloud platform triggers an interpolation compensation strategy; If the delay time rises, the cloud platform triggers a buffer playback strategy; The cloud platform returns the triggered strategy to the video sending end or the video playing end to achieve dynamic smoothing of the video stream; The video sending end includes the video communication module in the visit vehicle or the video sending module of the cloud platform; the video playing end includes the video display interface in the visit vehicle, as well as the handheld device or the VR device.

Images