Video inquiry processing method and device

Abstract

The invention discloses a video inquiry processing method and device, and relates to the technical field of internet medical treatment. A specific embodiment of the method comprises the steps of transmitting an abnormal event to a robot server in response to monitoring that the abnormal event occurs in an object, so as to judge whether to generate a video inquiry instruction based on the abnormal event through the robot server; and in response to a received video inquiry instruction transmitted by the robot server side, calling a local doctor inquiry service to transmit a video inquiry request to a doctor side, so that the doctor side establishes a video inquiry link between the distributed doctor and the robot side after the doctor is distributed. According to the embodiment of the invention, the method can meet the requirement that the household robot actively initiates video inquiry when automatically discovering that the object is abnormal in a daily household monitoring scene.

Description

Technical Field

[0001] The present invention relates to the field of Internet medical technology, and in particular to a video consultation processing method and device. Background Art

[0002] The current video call technology is relatively mature, and video calls between two or more parties can be realized through technologies such as WebRTC. Video consultation is an important application scenario of video calls. The two parties in the call are doctors and patients. When both parties initiate a video call, the consultation service can be carried out.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: the existing video consultation solutions all rely on the user's active trigger, that is, the service can only be started when the doctor or patient actively initiates a video consultation request. This mode cannot meet the needs of initiating a video consultation when the home robot detects an abnormal situation at home. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a video consultation processing method and device, which can at least solve the problem that the existing technology relies on the user's active triggering and cannot meet the need to initiate a video consultation when a home robot detects an abnormal situation at home.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a video consultation processing method applied to a robot side is provided, comprising:

[0006] In response to monitoring an abnormal event of the subject, the abnormal event is transmitted to the robot server, so that the robot server determines whether to generate a video consultation instruction based on the abnormal event;

[0007] In response to receiving the video consultation instruction transmitted by the robot service end, the local doctor consultation service is called to transmit the video consultation request to the doctor end, so that after the doctor end assigns a doctor, a video consultation link is established between the assigned doctor and the robot end.

[0008] Optionally, the determining, by the robot server, whether to generate a video consultation instruction based on the abnormal event includes:

[0009] Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, receive the voice information transmitted by the robot server and broadcast it; wherein the voice information is obtained by converting the text information of the inquiry whether to initiate the video consultation instruction;

[0010] Acquire the reply information of the object, transmit the reply information to the robot server for intention recognition processing, and obtain the intention recognition result;

[0011] Among them, when the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, the robot server generates a video consultation instruction and transmits it to the robot side.

[0012] Optionally, after establishing the video consultation link between the assigned doctor and the robot terminal, the method further includes:

[0013] Receiving a three-party video instruction transmitted by the robot server; wherein the three-party video instruction is generated by the robot server after receiving the video consultation information transmitted by the doctor, based on the abnormal event, the conference number of the video consultation link, and the mobile terminal identifier; the mobile terminal identifier is determined by the basic information of the object;

[0014] Based on the mobile terminal identifier and the conference number, a three-party video conference request is generated, and the three-party conference interface of the local doctor consultation service is called to transmit the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number to join the mobile terminal to the video consultation conference.

[0015] To achieve the above object, according to one aspect of an embodiment of the present invention, a video consultation processing method applied to a robot server is provided, comprising:

[0016] Receive abnormal events related to the object transmitted by the robot end;

[0017] In response to determining to generate a video medical consultation instruction based on the abnormal event, acquiring basic information of the subject, and generating a video medical consultation instruction based on the abnormal event and the basic information;

[0018] The video consultation instruction is transmitted to the robot side, so that the robot side calls the local doctor consultation service and transmits the video consultation request to the doctor side. After the doctor side assigns a doctor, a video consultation link is established between the assigned doctor and the robot side.

[0019] Optionally, the determining to generate a video medical consultation instruction based on the abnormal event includes:

[0020] Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, convert the text information of the inquiry whether to initiate the video consultation instruction into voice information, and transmit the voice information to the robot end for broadcasting;

[0021] Receiving reply information of the object transmitted by the robot end, performing intent recognition on the reply information, and obtaining an intent recognition result;

[0022] When the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, it is determined to generate a video consultation instruction.

[0023] Optionally, the method further comprises:

[0024] Receiving video consultation information transmitted by the doctor end; wherein the video consultation information includes the conference number of the video consultation link;

[0025] Determine other objects associated with the object based on the basic information, obtain mobile terminal identifiers of the other objects, and then generate a three-party video instruction based on the mobile terminal identifier, the abnormal event, and the conference number;

[0026] The three-party video instruction is transmitted to the robot side, so that the robot side generates a three-party video conference request based on the mobile terminal identifier and the conference number, calls the three-party conference interface of the local doctor consultation service, and sends the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number, and joins the mobile terminal to the video consultation conference.

[0027] Optionally, the method further includes an intermediate platform, wherein the intermediate platform is used to perform one or more of the following operations:

[0028] The text message asking whether to initiate a video consultation instruction is converted into voice information and transmitted to the robot end for broadcast;

[0029] Receive the reply voice information transmitted by the robot end, perform intent recognition on the reply voice information, obtain the intent recognition result and transmit it to the robot server end;

[0030] Receive the video consultation instruction transmitted by the robot server, and transmit the video consultation instruction to the robot end;

[0031] Receive the three-party video instructions transmitted by the robot server and transmit the three-party video instructions to the robot.

[0032] Optionally, the method further comprises:

[0033] Generate a link identifier;

[0034] After receiving the video consultation information transmitted by the doctor, insert a record in the event table based on the link identifier to record the video consultation information;

[0035] After receiving the three-party video call information transmitted by the doctor, insert another record in the event table based on the link identifier to record the three-party video call information;

[0036] In response to the end of the three-party video call, the records associated with the link identifier in the event table are integrated to obtain a full-link video consultation record corresponding to the object.

[0037] To achieve the above object, according to another aspect of an embodiment of the present invention, a video consultation processing device applied to a robot side is provided, comprising:

[0038] A monitoring module, configured to transmit, in response to detecting an abnormal event of the subject, the abnormal event to the robot server, so that the robot server determines whether to generate a video consultation instruction based on the abnormal event;

[0039] The first consultation module is used to call the local doctor consultation service in response to receiving the video consultation instruction transmitted by the robot service end, so as to transmit the video consultation request to the doctor end, so that after the doctor end assigns a doctor, a video consultation link is established between the assigned doctor and the robot end.

[0040] Optionally, the monitoring module is used to:

[0041] Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, receive the voice information transmitted by the robot server and broadcast it; wherein the voice information is obtained by converting the text information of the inquiry whether to initiate the video consultation instruction;

[0042] Acquire the reply information of the object, transmit the reply information to the robot server for intention recognition processing, and obtain the intention recognition result;

[0043] Among them, when the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, the robot server generates a video consultation instruction and transmits it to the robot side.

[0044] Optionally, the device further includes a first third-party video module, configured to:

[0045] Receiving a three-party video instruction transmitted by the robot server; wherein the three-party video instruction is generated by the robot server after receiving the video consultation information transmitted by the doctor, based on the abnormal event, the conference number of the video consultation link, and the mobile terminal identifier; the mobile terminal identifier is determined by the basic information of the object;

[0046] Based on the mobile terminal identifier and the conference number, a three-party video conference request is generated, and the three-party conference interface of the local doctor consultation service is called to transmit the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number to join the mobile terminal to the video consultation conference.

[0047] To achieve the above object, according to another aspect of an embodiment of the present invention, a video consultation processing device applied to a robot server is provided, comprising:

[0048] A receiving module, used for receiving abnormal events related to the object transmitted by the robot end;

[0049] An instruction module, configured to generate a video consultation instruction in response to determining to generate a video consultation instruction based on the abnormal event, obtain basic information of the object, and generate a video consultation instruction based on the abnormal event and the basic information;

[0050] The second consultation module is used to transmit the video consultation instruction to the robot side, so that the robot side calls the local doctor consultation service and transmits the video consultation request to the doctor side. After the doctor side assigns a doctor, it establishes a video consultation link between the assigned doctor and the robot side.

[0051] Optionally, the instruction module is used to:

[0052] Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, convert the text information of the inquiry whether to initiate the video consultation instruction into voice information, and transmit the voice information to the robot end for broadcasting;

[0053] Receiving reply information of the object transmitted by the robot end, performing intent recognition on the reply information, and obtaining an intent recognition result;

[0054] When the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, it is determined to generate a video consultation instruction.

[0055] Optionally, the device further includes a second third-party video module, configured to:

[0056] Receiving video consultation information transmitted by the doctor end; wherein the video consultation information includes the conference number of the video consultation link;

[0057] Determine other objects associated with the object based on the basic information, obtain mobile terminal identifiers of the other objects, and then generate a three-party video instruction based on the mobile terminal identifier, the abnormal event, and the conference number;

[0058] The three-party video instruction is transmitted to the robot side, so that the robot side generates a three-party video conference request based on the mobile terminal identifier and the conference number, calls the three-party conference interface of the local doctor consultation service, and sends the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number, and joins the mobile terminal to the video consultation conference.

[0059] Optionally, the device further comprises an intermediate platform, and the intermediate platform is used to perform one or more of the following operations:

[0060] The text message asking whether to initiate a video consultation instruction is converted into voice information and transmitted to the robot end for broadcast;

[0061] Receive the reply voice information transmitted by the robot end, perform intent recognition on the reply voice information, obtain the intent recognition result and transmit it to the robot server end;

[0062] Receive the video consultation instruction transmitted by the robot server, and transmit the video consultation instruction to the robot end;

[0063] Receive the three-party video instructions transmitted by the robot server and transmit the three-party video instructions to the robot.

[0064] Optionally, the device further includes a link module, configured to:

[0065] Generate a link identifier;

[0066] After receiving the video consultation information transmitted by the doctor, insert a record in the event table based on the link identifier to record the video consultation information;

[0067] After receiving the three-party video call information transmitted by the doctor, insert another record in the event table based on the link identifier to record the three-party video call information;

[0068] In response to the end of the three-party video call, the records associated with the link identifier in the event table are integrated to obtain a full-link video consultation record corresponding to the object.

[0069] To achieve the above objective, according to another aspect of an embodiment of the present invention, a video medical consultation processing electronic device is provided.

[0070] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned video consultation processing methods.

[0071] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned video medical consultation processing methods is implemented.

[0072] To achieve the above object, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product of an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the video consultation processing method provided by the embodiment of the present invention is implemented.

[0073] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: when the robot detects an abnormal situation of the object, it will immediately report the abnormal information to the robot server for processing. The server evaluates the abnormal type and decides whether to start a video consultation through voice inquiry or directly. This operation can meet the needs of the home robot to actively initiate a video consultation when an abnormal situation is found in the daily home monitoring scenario.

[0074] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0076] Figure 1 This is a schematic diagram of the main flow of a video consultation processing method applied to a robot end according to an embodiment of the present invention;

[0077] Figure 2 is a flowchart of a video consultation processing method applied to a robot server according to an embodiment of the present invention;

[0078] FIG3 (a) is a schematic diagram of a process for generating video consultation instructions based on abnormality type judgment;

[0079] Figure 3 (b) is a schematic diagram of the process of initiating a video consultation;

[0080] Figure 4 is a flow chart of an optional video consultation processing method according to an embodiment of the present invention;

[0081] Figure 5 It is a flowchart of three-party video processing;

[0082] Figure 6 is a flowchart of a specific video consultation processing method according to an embodiment of the present invention;

[0083] Figure 7 is a schematic diagram of main modules of a video medical consultation processing device applied to a robot end according to an embodiment of the present invention;

[0084] Figure 8is a schematic diagram of main modules of a video medical consultation processing device applied to a robot server according to an embodiment of the present invention;

[0085] Fig. 9 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0086] Fig.10 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0087] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0088] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0089] In the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The acquisition, transmission, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations, and are used for legal and reasonable purposes, and are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities.

[0090] For user information, necessary measures should be taken to prevent illegal access to such personal information data, ensure that persons who have access to personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting data. When applicable, including in certain relevant applications, protect user privacy by de-identifying data, such as by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the specific address level), controlling how data is stored, and / or other methods of de-identification.

[0091] See also Figure 1 , which shows a main flow chart of a video consultation processing method applied to a robot end provided by an embodiment of the present invention, including the following steps:

[0092] S101: In response to monitoring an abnormal event of an object, transmitting the abnormal event to a robot server, so that the robot server determines whether to generate a video consultation instruction based on the abnormal event;

[0093] S102: In response to receiving the video consultation instruction transmitted by the robot service end, calling the local doctor consultation service to transmit the video consultation request to the doctor end, so that after the doctor end assigns a doctor, a video consultation link is established between the assigned doctor and the robot end.

[0094] See also Figure 2 , which shows a main flow chart of a video consultation processing method applied to a robot server provided by an embodiment of the present invention, including the following steps:

[0095] S201: receiving abnormal events related to the object transmitted by the robot end;

[0096] S202: In response to determining to generate a video medical consultation instruction based on the abnormal event, obtaining basic information of the subject, and generating a video medical consultation instruction based on the abnormal event and the basic information;

[0097] S203: The video consultation instruction is transmitted to the robot side, so that the robot side calls the local doctor consultation service and transmits the video consultation request to the doctor side. After the doctor side assigns a doctor, a video consultation link is established between the assigned doctor and the robot side.

[0098] for Figure 1 and Figure 2 As shown in the steps, the home robot (i.e., the robot side) is mainly used for health and safety monitoring in the home environment, especially for the elderly at home. Its core responsibilities include continuously monitoring the health of the elderly, such as vital signs such as heart rate and blood pressure, and using specific algorithms to identify potential dangers or abnormal situations, such as falls, long periods of inactivity, etc. Once an abnormality is detected, the robot side will immediately report the relevant information to the cloud (i.e., the robot server, or robot cloud) for further processing.

[0099] Abnormal events are divided into multiple abnormal types. Taking falls and emergencies as examples, the first abnormal type is falls, and the second abnormal type is emergencies. In addition, there can be other abnormal types customized by the staff, but different abnormal types require different processing methods. For the monitoring technology of falls, it can be determined by the CV detection algorithm (target detection algorithm in computer vision) of the robot body's visual sensor, or it can be used with millimeter-wave radar or camera combined with AI (Artificial Intelligence) algorithm for real-time posture analysis. For the monitoring technology of emergencies, it can be realized through a variety of sensors and methods. For example, key physiological parameters such as respiratory rate, heart rate and body temperature can be monitored through hand shape, facial expression, millimeter-wave radar and other technologies, which will not be elaborated in detail here.

[0100] If the abnormal type is a fall, the robot server needs to transmit the "inquiry whether to initiate a video consultation command" to the robot side. The command contains the abnormal scene information and the voice information that needs to be broadcast. However, before sending, the "inquiry whether to initiate a video consultation command" needs to be converted into TTS (Text-to-Speech) voice, and the inquiry voice is broadcast on the robot side. After the robot side waits for the object's reply information, it will transmit the collected information (such as voice information, or text information, such as some elderly people who are inconvenient to speak) to the robot server for intent recognition, and obtain the intent recognition result. If the intent recognition result is to initiate a video consultation command, a video consultation command is generated and sent to the robot side. However, if the abnormal type is an emergency, a video consultation is directly initiated. Similarly, if the intent recognition result of the fall scene is that a video consultation is required, a video consultation also needs to be initiated.

[0101] Among them, TTS voice technology, namely text-to-speech technology, is a technology that converts text information into audible sound. It can convert text content on electronic devices into sound output that humans can understand, and is widely used in many fields such as reading assistance, navigation prompts, virtual assistants, etc. TTS systems usually include multiple steps such as text analysis, language processing and sound synthesis to ensure that the generated speech sounds natural and smooth.

[0102] As shown in Figure 3 (a), this solution can also set up an intermediate platform between the robot side and the robot server side, such as the AVS (Audio Voice Service) platform. AVS is a channel that can convert text and voice TTS to each other and supports intent recognition. If the abnormal type is a fall, the robot server side needs to transmit the inquiry command whether to initiate a video consultation through the AVS platform. The command contains the abnormal scene information and the voice information to be broadcast. After receiving the command, the AVS platform will convert the text information into TTS voice information and broadcast the inquiry voice information on the robot side. After the robot side waits for the object's reply information, it will transmit the collected information to the AVS platform for intent recognition, obtain the intent recognition result, and determine whether the object needs a video consultation. After that, the AVS platform encapsulates the intent recognition result into an event to notify the robot server side, and the robot server side performs subsequent processing. Among them, intent recognition: a processing method for obtaining results after natural language processing of text.

[0103] When the robot server determines that a video consultation needs to be initiated, it will generate a video consultation instruction, including abnormal scene information (such as falls, emergencies, etc.), basic information of the locally stored object (i.e., archival information, including name, age, health records, etc.), urgency information (this is defined by the product, and falls and emergencies are currently the most urgent), and bussinessId. Among them, bussinessId is the unique link identifier, which records the entire link from initiation to end of the video consultation. The identifier can be generated based on the snowflake algorithm, self-increment ID, etc. The snowflake algorithm is preferred. This algorithm is suitable for use in distributed systems and can efficiently generate unique identifiers. In addition, the snowflake algorithm relies on timestamp information, which is convenient for subsequent tracking and analysis. Therefore, generating link identifiers through the snowflake algorithm can ensure that each business operation in the video consultation process has a unique, orderly and efficient identifier.

[0104] The robot server can directly send the video consultation instruction to the robot, or send it to AVS first. After receiving the video consultation instruction, AVS sends the instruction to the robot. In any case, after receiving the video consultation instruction, the robot needs to call the pre-integrated local family doctor video consultation SDK (Software Development Kit) to initiate a video consultation request and transmit it to the family doctor side (for family doctor services, it is the doctor side, video consultation is conducted at the doctor's workstation), and pass the bussinessId through to achieve full-link identification. At this time, the robot will call up the fake call page (referring to the waiting page before the video call is connected), and the family doctor workstation will allocate doctors, as shown in Figure 3 (b). After the allocation, a two-way call will be established between the robot and the allocated doctor, so that the doctor can provide consultation services to the object on the robot.

[0105] The method provided in the above embodiment can achieve the purpose of the household robot actively initiating a video consultation when an abnormal situation is found in the daily home monitoring scenario. When the robot detects that an abnormal situation has occurred in the object, it will immediately report the abnormal information to the robot server for processing. The server evaluates the type of abnormality and decides whether to start the video consultation through voice inquiry or directly. For example, in the case of a fall of an elderly person at home, the robot can automatically identify this abnormality and ask the elderly whether a video consultation is needed; and when an emergency medical condition is detected in the elderly, the video consultation function can be started immediately. In addition, this solution uses the AVS platform to achieve efficient interaction between the robot and the server, and through the support of functions such as command issuance, intent recognition and message notification, it ensures that the video consultation service can be carried out in a timely and effective manner in various monitoring scenarios.

[0106] See also Figure 4 , shows a schematic flow chart of an optional video consultation processing method according to an embodiment of the present invention, comprising the following steps:

[0107] S401: receiving video consultation information transmitted by a doctor, wherein the video consultation information includes a conference number of the video consultation link;

[0108] S402: Determine other objects associated with the object based on the basic information, obtain mobile terminal identifiers of the other objects, and then generate a three-party video instruction based on the mobile terminal identifier, the abnormal event, and the conference number;

[0109] S403: Transmitting the three-party video command to the robot end;

[0110] S404: The robot generates a three-party video conference request based on the mobile terminal identifier and the conference number, calls a three-party conference interface of the local doctor consultation service, and sends the three-party video conference request to the doctor terminal;

[0111] S405: Based on the conference number, the doctor sends a video request to the mobile terminal corresponding to the mobile terminal identifier, and joins the mobile terminal to the video consultation conference.

[0112] In the above implementation, after assigning a doctor, the family doctor will establish a two-way call between the doctor and the robot, and the robot can then provide consultation services. At the same time, the family doctor will execute a callback notification and send the video consultation information to the robot asynchronously through a message queue (MQ, full name Message Queue) to ensure that the information is not lost, and the businessId is placed in the extended field of the MQ message as a business attribute of the MQ message. Here, MQ can also be regarded as a form of information transmission.

[0113] Video consultation information includes information such as meetingType, meetingId, and memberStatus. Among them, meetingType describes the type of meeting, which can be set to video conference in this solution. meetingId is used to uniquely identify the ID of a meeting. Each meeting has a unique meetingId, which is used to identify and manage specific meetings in the system. It is usually a string or number, which can be generated when the meeting is created and used in operations such as meeting invitation and joining. memberStatus describes the status of meeting members, such as which specific doctor, object information, whether to join (i.e. whether online), etc.

[0114] After receiving the MQ information, the robot server inserts a record in the event table based on the bussinessId to record the video conference information. In addition, based on the basic information of the object, other objects associated with the object are obtained from the local database, such as children, emergency contacts (such as neighbors, the other party of the couple, other friendly people), etc. Then the mobile terminal identification PIN of other objects is obtained. The mobile terminal identification PIN can take many forms, depending on the design requirements and security considerations of the system, such as mobile phone number, device ID (each mobile device has a unique device ID), user ID (a unique ID assigned to each user within the system), email address, custom PIN code, etc. Considering security, uniqueness and ease of use, the mobile phone number is a common and reasonable choice.

[0115] The robot server generates a three-party video command based on the abnormal event, mobile PIN, meetingId and bussinessId. It should be noted that bussinessId is set to facilitate subsequent record search. If bussinessId is not set, the three-party video command does not include bussinessId. The robot server can send the three-party video command directly to the robot, or send it to AVS first. After receiving the three-party video command, AVS sends the command to the robot.

[0116] After receiving the three-party video conference command, the robot generates a three-party video conference request based on the mobile terminal PIN and meetingId, calls the family doctor's video consultation SDK to pull up the three-party conference interface, and transparently transmits the businessId. After receiving the three-party video conference request sent by the robot, the family terminal will initiate a video request on the mobile terminal based on the conference number and wait for the mobile terminal to access the video. See Figure 5 At the same time, the robot will display the three-party video interface, and the mobile terminal will be in a waiting state. After the mobile terminal connects to the video, a three-party video call is established on the home doctor side. The robot will display the three-party video interface and complete the three-party video consultation.

[0117] The family doctor sends the three-party video call information to the robot server through MQ, including information such as meetingType, meetingId, and memberStatus, and the bussinessId is placed in the extended field of the MQ message. After receiving the MQ message, the robot server inserts another record in the event table based on the bussinessId, that is, a new record is added each time. After the three-party video consultation is completed, the robot server integrates the records related to the bussinessId in the event table based on the bussinessId to obtain the final video consultation record.

[0118] After the mobile terminal accesses the video, the family doctor workstation establishes a three-party video call and notifies the robot server of the information through MQ, including information such as meetingType, meetingId and memberStatus. The bussinessId is placed in the extended field of the MQ message. After the cloud receives the MQ message, it inserts a record in the event table. Subsequently, all events of the consultation can be queried through the meetingId; the robot side will display the three-party video interface to complete the three-party video consultation.

[0119] The method provided in the above embodiment reuses the video consultation capability of the family doctor. In view of the fact that the existing audio and video technology is relatively mature, the family doctor service can be replaced by introducing an open source audio and video SDK to achieve business decoupling. Considering that some subjects may not express themselves clearly when responding to questions, this solution further introduces a three-party video call function. Specifically, the associated mobile terminal PIN code is determined based on the basic information of the elderly, and the relevant emergency contacts (such as children, etc.) are notified through the PIN code, and they are added to the video consultation process to form a three-party video consultation service. This operation not only improves the accuracy of the consultation, but also provides strong data support for subsequent medical decisions.

[0120] See also Figure 6 , shows a schematic flow chart of a specific video consultation processing method according to an embodiment of the present invention, comprising the following steps:

[0121] 1. The robot uses the monitoring algorithm to monitor the monitored object in real time, obtains abnormal events, and reports the abnormal events to the robot server;

[0122] 2. The robot server determines the abnormal type of the abnormal event, and if the abnormal type is the first abnormal type, sends a text message to the AVS intermediate platform asking whether to initiate a video consultation instruction;

[0123] 3. The AVS intermediate platform converts text information into voice information and transmits the voice information to the robot end for broadcast;

[0124] 4. The robot collects the object's reply information, which can be voice information, input text information, or emoticons, such as OK gestures; and sends the reply information to the AVS intermediate platform;

[0125] 5. The AVS intermediate platform performs intent recognition on the reply information, obtains the intent recognition result, encapsulates the intent recognition result as an event, and sends it to the robot server;

[0126] 6. The robot server determines to generate a video consultation instruction when the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction;

[0127] 7. The robot server sends the video consultation instruction to the AVS intermediate platform, and the AVS intermediate platform sends the video consultation instruction to the robot;

[0128] 8. The robot side calls the local doctor consultation service to transmit the video consultation request to the doctor side, so that after the doctor side assigns a doctor, a video consultation link is established between the assigned doctor and the robot side;

[0129] 9. The doctor initiates a callback and sends the video consultation information to the robot server in the form of MQ; wherein the video consultation information includes the conference number of the video consultation link;

[0130] 10. The robot server determines other objects associated with the object based on the basic information, obtains the mobile terminal identifiers of the other objects, and then generates a three-party video instruction based on the mobile terminal identifier, the abnormal event, and the conference number;

[0131] 11. The robot server transmits the three-party video command to the AVS intermediate platform, and the AVS intermediate platform sends the three-party video command to the robot;

[0132] 12. The robot generates a three-party video conference request based on the mobile terminal identifier and the conference number, calls the three-party conference interface of the local doctor consultation service, and sends the three-party video conference request to the doctor terminal;

[0133] 13. Based on the conference number, the doctor sends a video request to the mobile terminal corresponding to the mobile terminal identifier, and joins the mobile terminal to the video consultation conference;

[0134] 14. The doctor initiates a callback and sends the video call information to the robot server in the form of MQ.

[0135] See also Figure 7 , which shows a schematic diagram of main modules of a video medical consultation processing device 700 applied to a robot end provided by an embodiment of the present invention, including:

[0136] The monitoring module 701 is used for transmitting the abnormal event to the robot server in response to the abnormal event detected by the object, so that the robot server determines whether to generate a video consultation instruction based on the abnormal event;

[0137] The first consultation module 702 is used to call the local doctor consultation service in response to receiving the video consultation instruction transmitted by the robot service end, so as to transmit the video consultation request to the doctor end, so that after the doctor end assigns a doctor, a video consultation link is established between the assigned doctor and the robot end.

[0138] In the implementation device of the present invention, the monitoring module 701 is used to:

[0139] Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, receive the voice information transmitted by the robot server and broadcast it; wherein the voice information is obtained by converting the text information of the inquiry whether to initiate the video consultation instruction;

[0140] Acquire the reply information of the object, transmit the reply information to the robot server for intention recognition processing, and obtain the intention recognition result;

[0141] Among them, when the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, the robot server generates a video consultation instruction and transmits it to the robot side.

[0142] The implementation device of the present invention further includes a first third-party video module, which is used to:

[0143] Receiving a three-party video instruction transmitted by the robot server; wherein the three-party video instruction is generated by the robot server after receiving the video consultation information transmitted by the doctor, based on the abnormal event, the conference number of the video consultation link, and the mobile terminal identifier; the mobile terminal identifier is determined by the basic information of the object;

[0144] Based on the mobile terminal identifier and the conference number, a three-party video conference request is generated, and the three-party conference interface of the local doctor consultation service is called to transmit the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number to join the mobile terminal to the video consultation conference.

[0145] See also Figure 8 , which shows a schematic diagram of main modules of a video consultation processing device 800 applied to a robot server provided by an embodiment of the present invention, including:

[0146] The receiving module 801 is used to receive abnormal events related to the object transmitted by the robot end;

[0147] Instruction module 802, for generating a video consultation instruction in response to determining to generate a video consultation instruction based on the abnormal event, acquiring basic information of the subject, and generating a video consultation instruction based on the abnormal event and the basic information;

[0148] The second consultation module 803 is used to transmit the video consultation instruction to the robot side, so that the robot side calls the local doctor consultation service and transmits the video consultation request to the doctor side. After the doctor side assigns a doctor, it establishes a video consultation link between the assigned doctor and the robot side.

[0149] In the implementation device of the present invention, the instruction module 801 is used to:

[0150] Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, convert the text information of the inquiry whether to initiate the video consultation instruction into voice information, and transmit the voice information to the robot end for broadcasting;

[0151] Receiving reply information of the object transmitted by the robot end, performing intent recognition on the reply information, and obtaining an intent recognition result;

[0152] When the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, it is determined to generate a video consultation instruction.

[0153] The implementation device of the present invention further includes a second third-party video module, which is used to:

[0154] Receiving video consultation information transmitted by the doctor end; wherein the video consultation information includes the conference number of the video consultation link;

[0155] Determine other objects associated with the object based on the basic information, obtain mobile terminal identifiers of the other objects, and then generate a three-party video instruction based on the mobile terminal identifier, the abnormal event, and the conference number;

[0156] The three-party video instruction is transmitted to the robot side, so that the robot side generates a three-party video conference request based on the mobile terminal identifier and the conference number, calls the three-party conference interface of the local doctor consultation service, and sends the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number, and joins the mobile terminal to the video consultation conference.

[0157] The implementation device of the present invention also includes an intermediate platform, which is used to perform one or more of the following operations:

[0158] The text message asking whether to initiate a video consultation instruction is converted into voice information and transmitted to the robot end for broadcast;

[0159] Receive the reply voice information transmitted by the robot end, perform intent recognition on the reply voice information, obtain the intent recognition result and transmit it to the robot server end;

[0160] Receive the video consultation instruction transmitted by the robot server, and transmit the video consultation instruction to the robot end;

[0161] Receive the three-party video instructions transmitted by the robot server and transmit the three-party video instructions to the robot.

[0162] The implementation device of the present invention also includes a link module, which is used to:

[0163] Generate a link identifier;

[0164] After receiving the video consultation information transmitted by the doctor, insert a record in the event table based on the link identifier to record the video consultation information;

[0165] After receiving the three-party video call information transmitted by the doctor, insert another record in the event table based on the link identifier to record the three-party video call information;

[0166] In response to the end of the three-party video call, the records associated with the link identifier in the event table are integrated to obtain a full-link video consultation record corresponding to the object.

[0167] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0168] Fig. 9 An exemplary system architecture 900 to which embodiments of the present invention may be applied is shown, including terminal devices 901 , 902 , 903 , a network 904 and a server 905 (only an example).

[0169] Terminal devices 901, 902, 903 can be various electronic devices with display screens and supporting web browsing, and various communication client applications are installed. Users can use terminal devices 901, 902, 903 to interact with server 905 through network 904 to receive or send messages, etc.

[0170] The network 904 is used to provide a medium for communication links between the terminal devices 901, 902, 903 and the server 905. The network 904 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0171] The server 905 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 901, 902, and 903 (for example only). The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 905, and accordingly, the device is generally set in the server 905.

[0172] It should be understood that Fig. 9 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0173] Reference below Fig.10 , which shows a schematic diagram of the structure of a computer system 1000 of a terminal device suitable for implementing an embodiment of the present invention. Fig.10 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0174] like Fig.10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0175] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0176] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present invention are executed.

[0177] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0178] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0179] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, may be described as: a processor includes a monitoring module and a first consultation module. The names of these modules do not, in some cases, constitute a limitation on the modules themselves, for example, the first consultation module may also be described as a "consultation module".

[0180] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes any of the above video consultation processing methods.

[0181] The computer program product of the present invention includes a computer program, and when the computer program is executed by a processor, the video medical consultation processing method in the embodiment of the present invention is implemented.

[0182] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video consultation processing method, characterized in that: Applied to the robot side, including: In response to monitoring an abnormal event of the subject, the abnormal event is transmitted to the robot server, so that the robot server determines whether to generate a video consultation instruction based on the abnormal event; In response to receiving the video consultation instruction transmitted by the robot service end, the local doctor consultation service is called to transmit the video consultation request to the doctor end, so that after the doctor end assigns a doctor, a video consultation link is established between the assigned doctor and the robot end.

2. The method according to claim 1, characterized in that The determining, by the robot server, whether to generate a video consultation instruction based on the abnormal event includes: Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, receive the voice information transmitted by the robot server and broadcast it; wherein the voice information is obtained by converting the text information of the inquiry whether to initiate the video consultation instruction; Acquire the reply information of the object, transmit the reply information to the robot server for intention recognition processing, and obtain the intention recognition result; Among them, when the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, the robot server generates a video consultation instruction and transmits it to the robot side.

3. The method according to claim 1 or 2, characterized in that: After establishing the video consultation link between the assigned doctor and the robot terminal, the method further includes: Receiving a three-party video instruction transmitted by the robot server; wherein the three-party video instruction is generated by the robot server after receiving the video consultation information transmitted by the doctor, based on the abnormal event, the conference number of the video consultation link, and the mobile terminal identifier; the mobile terminal identifier is determined by the basic information of the object; Based on the mobile terminal identifier and the conference number, a three-party video conference request is generated, and the three-party conference interface of the local doctor consultation service is called to transmit the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number to join the mobile terminal to the video consultation conference.

4. A video consultation method, characterized in that: Applied to the robot server, including: Receive abnormal events related to the object transmitted by the robot end; In response to determining to generate a video medical consultation instruction based on the abnormal event, acquiring basic information of the subject, and generating a video medical consultation instruction based on the abnormal event and the basic information; The video consultation instruction is transmitted to the robot side, so that the robot side calls the local doctor consultation service and transmits the video consultation request to the doctor side. After the doctor side assigns a doctor, a video consultation link is established between the assigned doctor and the robot side.

5. The method according to claim 4, characterized in that The step of determining and generating a video consultation instruction based on the abnormal event includes: Determine the abnormal type of the abnormal event, and when the abnormal type is the first abnormal type, convert the text information of the inquiry whether to initiate the video consultation instruction into voice information, and transmit the voice information to the robot end for broadcasting; Receiving reply information of the object transmitted by the robot end, performing intent recognition on the reply information, and obtaining an intent recognition result; When the exception type is the second exception type, or the exception type is the first exception type and the intention recognition result is to initiate a video consultation instruction, it is determined to generate a video consultation instruction.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Receiving video consultation information transmitted by the doctor end; wherein the video consultation information includes the conference number of the video consultation link; Determine other objects associated with the object based on the basic information, obtain mobile terminal identifiers of the other objects, and then generate a three-party video instruction based on the mobile terminal identifier, the abnormal event, and the conference number; The three-party video instruction is transmitted to the robot side, so that the robot side generates a three-party video conference request based on the mobile terminal identifier and the conference number, calls the three-party conference interface of the local doctor consultation service, and sends the three-party video conference request to the doctor side, so that the doctor side sends a video request to the mobile terminal corresponding to the mobile terminal identifier based on the conference number, and joins the mobile terminal to the video consultation conference.

7. The method according to claim 6, characterized in that The method further includes an intermediate platform, wherein the intermediate platform is used to perform one or more of the following operations: The text message asking whether to initiate a video consultation instruction is converted into voice information and transmitted to the robot end for broadcast; Receive the reply voice information transmitted by the robot end, perform intent recognition on the reply voice information, obtain the intent recognition result and transmit it to the robot server end; Receive the video consultation instruction transmitted by the robot server, and transmit the video consultation instruction to the robot end; Receive the three-party video instructions transmitted by the robot server and transmit the three-party video instructions to the robot.

8. The method according to claim 6, characterized in that The method further comprises: Generate a link identifier; After receiving the video consultation information transmitted by the doctor, insert a record in the event table based on the link identifier to record the video consultation information; After receiving the three-party video call information transmitted by the doctor, insert another record in the event table based on the link identifier to record the three-party video call information; In response to the end of the three-party video call, the records associated with the link identifier in the event table are integrated to obtain a full-link video consultation record corresponding to the object.

9. A video consultation processing device, characterized in that: Applied to the robot side, including: A monitoring module, configured to transmit, in response to detecting an abnormal event of the subject, the abnormal event to the robot server, so that the robot server determines whether to generate a video consultation instruction based on the abnormal event; The first consultation module is used to call the local doctor consultation service in response to receiving the video consultation instruction transmitted by the robot service end, so as to transmit the video consultation request to the doctor end, so that after the doctor end assigns a doctor, a video consultation link is established between the assigned doctor and the robot end.

10. A video medical consultation processing device, characterized in that: Applied to the robot server, including: A receiving module, used for receiving abnormal events related to the object transmitted by the robot end; An instruction module, configured to generate a video consultation instruction in response to determining to generate a video consultation instruction based on the abnormal event, obtain basic information of the object, and generate a video consultation instruction based on the abnormal event and the basic information; The second consultation module is used to transmit the video consultation instruction to the robot side, so that the robot side calls the local doctor consultation service and transmits the video consultation request to the doctor side. After the doctor side assigns a doctor, it establishes a video consultation link between the assigned doctor and the robot side.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images