A multimodal model-driven method based on visual collaborative interaction technology

Abstract

The present invention relates to a multimodal model driving method based on visual collaborative interaction technology applied in the field of medical information. By setting an online multi-person interactive platform, using the medical card number, the name of the patient and the attending doctor as identity authentication information, completing user login, and conducting online communication in the corresponding attending doctor communication interface, the procedures, precautions, postoperative rehabilitation and the like during medical treatment are effectively understood. In the case that the attending doctor fails to fully consider the patient's emotions and conditions due to the large number of patients, the patient can effectively reduce negative emotions such as anxiety through online communication. At the same time, the patient's voice can be effectively heard through online communication, and corresponding medical treatment improvements can be made, which effectively promotes the harmony of the doctor-patient relationship. In addition, through the coordinated use of a multimodal input device, special people can input communication content through sign language or lip language, and play an effective facial privacy protection effect during the input process to maintain their mental health.

Description

Technical Field

[0001] The present invention relates to a model-driven method, and in particular to a multimodal model-driven method based on a visualized collaborative interaction technology applied in the field of medical information. Background Art

[0002] In the modern medical system, with the advancement of medical technology and the popularization of electronic medical records, the amount of medical data has shown explosive growth. By using a variety of different information collection, processing and transmission methods to meet the diverse needs of different patients, these methods include but are not limited to text, images, sounds, biological signals, etc., which together constitute a complex and large multimodal medical information system.

[0003] For example, the specification of Chinese patent CN202410350480.8 discloses a remote consultation system based on a multimodal large language model, which includes a multimodal large language model module, a data processing module, a user interface, a communication module and an intelligent recommendation module; the multimodal large language model module is used to process and analyze medical data from different modalities, parse and understand the natural language communication between doctors and patients, identify pathological images, interpret test reports and analyze patient historical data, provide medical reports and diagnostic suggestions generated in natural language, and provide comprehensive diagnostic support; the data processing module is used to process real-time data from various medical devices, encrypt data based on the latest standard dynamic encryption technology, protect patient data, and ensure the security of data transmission and storage; the communication module is used for doctors and patients to interact through video conferencing, instant messaging or email.

[0004] As the existing medical system is large and complex, the medical process has become increasingly complicated, with more and more procedures. Patients may experience delays in treatment due to their lack of understanding of the process. In addition, due to the large number of medical patients, it is difficult for doctors to take into account the emotions and conditions of all patients during medical treatment, which can easily lead to disharmony between doctors and patients. Summary of the invention

[0005] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is that due to factors such as the large number of existing medical patients and the complicated medical process, patients are prone to delays in treatment and disharmonious doctor-patient relationships due to their lack of understanding of the medical process.

[0006] To solve the above problems, the present invention provides a multimodal model driving method based on visual collaborative interaction technology, comprising the following steps:

[0007] S1. The user successfully logs into the multi-person interactive platform of Hospital A by entering the corresponding medical card number, patient name and attending physician name in Hospital A, and enters the communication interface of the attending physician;

[0008] S2, the user inputs the communication content into the multi-person interaction platform through the multimodal input device, and then the vocabulary recognition module performs feature recognition on the communication content to detect whether there are privacy words and negative words;

[0009] When there are privacy words related to people, step S3 is performed; when no privacy words and negative words are identified, step S4 is performed; when there are no privacy words but negative words related to people or diseases are present, steps S4 and S5 are performed in sequence;

[0010] S3, annotating the privacy words in the communication content through the prompt feedback module, and inputting the annotated communication content back to the multimodal input device, so that the user knows and re-enters the new communication content, and then performs the feature recognition process in step S2 again;

[0011] S4, displaying the communication content on the interface of the multi-person interaction platform through the output display module for other users to view, thereby realizing the interaction process;

[0012] S5. Collect and record the communication contents with negative words published by users for later review by attending physicians and authenticity verification process;

[0013] When the content has a certain degree of authenticity, improvement suggestions can be sent to the corresponding attending physician. When the content lacks authenticity, the verification result can be sent to the user and the user is informed that the verification has been completed.

[0014] As a further supplement to this application, the authenticity verification process in step S5 includes the following steps:

[0015] S5-1. In the communication interface of the corresponding attending physician, randomly select N users (N is a positive integer greater than 2), and then convert the communication content with negative words published by the user in step S5 into question sentences, and send them to the above N users in the form of platform private messages, and wait for user feedback;

[0016] S5-2. When the opinions fed back by users exceeding n / N (n is a positive integer less than N) are consistent with the communication content published by the user in step S5, it indicates that the communication content has a certain degree of authenticity; otherwise, it lacks authenticity.

[0017] As a further supplement to the present application, the multi-person interaction platform includes a login module, a multimodal input module, a vocabulary recognition module, a prompt feedback module, an output display module, a recording module and a content verification module. The multimodal input module is connected to the multimodal inputter and is used to receive the communication content transmitted by the multimodal inputter. The recording module is used to record the communication content with negative vocabulary published by the user. The content verification module is used to perform the authenticity verification process in step S5.

[0018] As a further supplement to the present application, the login module is connected to the department database, patient database and doctor database. The department database stores all department information of Hospital A, the patient database stores all patient information of Hospital A, and the doctor database stores all active doctor information of Hospital A.

[0019] As a further supplement to the present application, the multimodal input device includes a terminal body, the front end of the terminal body is rotatably connected to a folding plate, and the rotational connection position of the two is located in the lower end area of ​​the terminal body, the outer end of the folding plate is fixedly connected to an operating screen, the operating screen is located on the upper end surface of the folding plate in a horizontal state, and the front end of the terminal body is fixedly connected to a camera.

[0020] As another improvement of the present application, the end of the folding plate away from the terminal body is rotatably connected to a folding frame, and the folding frame in the horizontal state is located at the upper end of the folding plate, the operation screen is located on the inner side of the frame of the folding frame, and the inner end of the folding frame is connected to a pair of rolling curtains, and the pair of rolling curtains are respectively located in the upper and lower areas of the folding frame in the vertical state.

[0021] As another improved supplement of the present application, the rolling blind includes a rotating rod, a blocking cloth and a moving rod, the two ends of the rotating rod are rotatably connected to the inside of the folding frame, one end of the blocking cloth is fixedly connected to the outer end of the rotating rod, and the other end is fixedly connected to the moving rod, and slideways are provided at the left and right ends of the folding frame, and the two ends of the moving rod are respectively slidably connected to the inside of a pair of slides and extend to the outside.

[0022] As another improvement supplement of the present application, the roller shutter also includes a pair of bolts, the inner wall of the slide is provided with a plurality of screw holes communicating with the outside, the pair of bolts are respectively threadedly connected to the inside of the pair of screw holes, and the ends of the bolts are in contact with the moving rod.

[0023] As another improved supplement of the present application, a pair of vertical grooves are provided at the front end of the terminal body, and when the folding frame is in contact with the surface of the terminal body, the bolts are located inside the vertical grooves.

[0024] As another improvement supplement of the present application, the front end of the terminal body and the outer end of the folding frame are coated with a pair of magnetic coatings. When the folding frame is in contact with the surface of the terminal body, the pair of magnetic coatings on the folding frame are respectively in contact with and adsorbed with the pair of magnetic coatings on the terminal body.

[0025] To summarize, the present application innovatively sets up a multi-person interactive platform on which multiple people can communicate online simultaneously. It uses the medical card number, the patient's and the attending physician's name as identity authentication information, allowing users to log in and enter the corresponding attending physician's communication interface for online communication, and effectively understand the procedures, precautions, and postoperative rehabilitation status during medical treatment and hospitalization. When the attending physician fails to fully consider the patient's emotions and conditions due to the large number of patients, online communication can effectively reduce the patient's anxiety and other negative emotions. At the same time, through online communication, the patient's voice can be effectively heard, and corresponding medical improvements can be made, which effectively promotes the harmony of the doctor-patient relationship. In addition, through the use of a multimodal input device, special groups of people can input online communication content through sign language or lip reading, and effectively protect their facial privacy during the input process to maintain their mental health. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the first and second implementation methods of this application;

[0027] Figure 2 This is a system diagram of the first and second implementation modes of the present application;

[0028] Figure 3 The stereoscopic multimodal input device in the first and second embodiments of the present application Figure 1 ;

[0029] Figure 4 The stereoscopic multimodal input device in the first and second embodiments of the present application Figure 2 ;

[0030] Figure 5 The stereoscopic multimodal input device in the first and second embodiments of the present application Figure 3 ;

[0031] Figure 6 The stereoscopic display of the multimodal input device in the second embodiment of the present application Figure 1 ;

[0032] Figure 7 The side structure diagram of the multi-modal input device in the second embodiment of the present application when in use Figure 1 ;

[0033] Figure 8 The stereoscopic display of the multimodal input device in the second embodiment of the present application Figure 2 ;

[0034] Fig. 9 The side structure diagram of the multi-modal input device in the second embodiment of the present application when in use Figure 2 .

[0035] Description of the numbers in the figure:

[0036] 1 terminal body, 101 vertical slot, 2 folding plate, 3 operation screen, 4 camera, 5 folding frame, 501 slide, 502 screw hole, 6 rolling curtain, 61 rotating rod, 62 cloth stop, 63 moving rod, 64 bolt, 7 magnetic coating. DETAILED DESCRIPTION

[0037] The following describes two implementation modes of the present application in detail with reference to the accompanying drawings.

[0038] The first implementation method:

[0039] The present invention provides a multimodal model driving method based on visual collaborative interaction technology, please refer to Figure 1 and Figure 2 , including the following steps:

[0040] S1. The user successfully logs into the multi-person interactive platform of Hospital A by entering the corresponding medical card number, patient name and attending physician name in Hospital A, and enters the communication interface of the attending physician;

[0041] Through the above login method, users entering the multi-person interactive platform are generally patients themselves or their relatives, thereby improving the security and privacy of the multi-person interactive platform;

[0042] S2, the user inputs the communication content into the multi-person interaction platform through the multimodal input device, and then the vocabulary recognition module performs feature recognition on the communication content to detect whether there are privacy words and negative words;

[0043] Privacy words include but are not limited to words and information related to identity privacy, physical privacy, etc. Negative words include but are not limited to "annoying", "fierce", "bad attitude" and other words related to medical care;

[0044] When there are privacy words related to people, step S3 is performed; when no privacy words and negative words are identified, step S4 is performed; when there are no privacy words but negative words related to people or diseases are present, steps S4 and S5 are performed in sequence;

[0045] S3, annotating the privacy words in the communication content through the prompt feedback module, and inputting the annotated communication content back to the multimodal input device, so that the user knows and re-enters the new communication content, and then performs the feature recognition process in step S2 again;

[0046] Through step S4, when users communicate online, it is not easy to disclose their own and others' private information, which effectively provides privacy for online interactions;

[0047] S4, displaying the communication content on the interface of the multi-person interaction platform through the output display module for other users to view, thereby realizing the interaction process;

[0048] S5. Collect and record the communication contents with negative words published by users for later review by attending physicians and authenticity verification process;

[0049] When the content has a certain degree of authenticity, improvement suggestions can be sent to the corresponding attending physician. When the content lacks authenticity, the verification result can be sent to the user and the user is informed that the verification has been completed.

[0050] The authenticity verification process in step S5 includes the following steps:

[0051] S5-1. In the communication interface of the corresponding attending physician, randomly select N users (N is a positive integer greater than 2), and then convert the communication content with negative words published by the user in step S5 into question sentences, and send them to the above N users in the form of platform private messages, and wait for user feedback;

[0052] S5-2. When the opinions fed back by users exceeding n / N (n is a positive integer less than N) are consistent with the communication content published by the user in step S5, it indicates that the communication content has a certain degree of authenticity; otherwise, it lacks authenticity.

[0053] For the communication content with negative words published by users, this application adopts the method of randomly selecting users to conduct online questionnaires, which can effectively obtain the patient's approval or disapproval of the communication content, thereby effectively reflecting the authenticity of the communication content;

[0054] When the communication content with negative words published by the user is proven to be true, the patient's feedback will be sent to the corresponding attending physician, and improvement suggestions will be sent to him. Compared with the existing technology, this application can effectively listen to the patient's voice through this multi-person interactive platform, and make corresponding improvements, which effectively promotes the harmony of the doctor-patient relationship; at the same time, different patient users who are seeing the same attending physician can effectively understand the treatment procedures required for medical treatment and hospitalization, as well as precautions, postoperative rehabilitation, etc. through online communication, so that patients can have a general understanding of their own disease condition (in actual situations, due to factors such as the large number of patients and busy treatment, doctors often do not have enough time to explain the condition of each patient in such detail, or patients tend to forget after talking), so as to carry out the treatment process in a timely and effective manner, and are not prone to negative emotions such as anxiety.

[0055] See also Figure 2The multi-person interaction platform includes a login module, a multimodal input module, a vocabulary recognition module, a prompt feedback module, an output display module, a recording module and a content verification module. The multimodal input module is connected to the multimodal input device and is used to receive the communication content transmitted by the multimodal input device. The recording module is used to record the communication content with negative vocabulary published by the user. The content verification module is used to perform the authenticity verification process in step S5.

[0056] The login module is connected to the department database, patient database and doctor database. The department database stores all department information of Hospital A, the patient database stores all patient information of Hospital A, and the doctor database stores all active doctor information of Hospital A.

[0057] See also Figure 3 The multimodal input device includes a terminal body 1, a folding plate 2 is rotatably connected to the front end of the terminal body 1, and the rotational connection position of the two is located in the lower end area of ​​the terminal body 1, an operation screen 3 is fixedly connected to the outer end of the folding plate 2, and the operation screen 3 is located on the upper end surface of the folding plate 2 in a horizontal state, and a camera 4 is fixedly connected to the front end of the terminal body 1. The rotatably arranged folding plate 2 can protect the operation screen 3 and the camera 4. When the patient communicates online, he can use the multimodal input device to select a usage method suitable for himself to input the communication content, such as: touch screen operation can be performed on the operation screen 3 to realize text input or voice input, and image acquisition and recognition can also be performed through the camera 4 to realize lip language input or sign language input.

[0058] Second implementation method:

[0059] Based on Implementation 1, this implementation adds the following structure, which effectively improves the security and privacy of special users when they input communication content. For details, please refer to Figure 4 and Figure 5 The folding plate 2 is rotatably connected to one end away from the terminal body 1 with a folding frame 5, and the folding frame 5 in a horizontal state is located at the upper end of the folding plate 2, the operation screen 3 is located inside the frame of the folding frame 5, and the inner end of the folding frame 5 is connected to a pair of rolling curtains 6, and the pair of rolling curtains 6 are respectively located in the upper and lower areas of the folding frame 5 in the vertical state.

[0060] Combination Figure 6 and Figure 8The rolling curtain 6 includes a rotating rod 61, a blocking cloth 62 and a movable rod 63. Both ends of the rotating rod 61 are rotatably connected to the inside of the folding frame 5. One end of the blocking cloth 62 is fixedly connected to the outer end of the rotating rod 61, and the other end is fixedly connected to the movable rod 63. Slideways 501 are provided at both left and right ends of the folding frame 5. Both ends of the movable rod 63 are slidably connected to the inside of a pair of slideways 501 and extend to the outside. The blocking cloth 62 is pulled by the movable rod 63 to realize the unwinding and winding of the blocking cloth 62 on the rotating rod 61. The rolling curtain 6 also includes a pair of bolts 64. The inner wall of the slideway 501 is provided with a plurality of screw holes 502 that communicate with the outside. The pair of bolts 64 are respectively threadedly connected to the inside of the pair of screw holes 502, and the ends of the bolts 64 are abutted against the movable rod 63. After the movable rod 63 is moved to a suitable position along the slideway 501, the movable rod 63 is positioned by the bolts 64, and the unwound blocking cloth 62 is fixed.

[0061] Usage for special users 1. When the user needs to input sign language (usually people with language impairment), such as Figure 6 and Figure 7 As shown, the lower side baffle 62 is first unrolled to move it from bottom to top to cover the hollow area in the lower half of the folding frame 5; then the person can align his face with the inner side of the folding frame 5, at which time the unrolled baffle 62 can cover the lower half of the person's face, and the person's eyes can observe the interface of the operation screen 3 through the hollow area in the upper half of the folding frame 5, and then the person can use both hands to make the sign language movements that need to be input, and after being collected and recognized by the camera 4, the sign language content is converted into text and displayed on the operation screen 3 for the person to confirm. In addition, when the person with speech dysfunction also has text recognition difficulties, the text can be converted into speech by operating the operation screen 3 and played out for the special user to confirm. When the text content is consistent with the meaning of the content to be expressed, the content can be sent to the multi-person interaction platform, that is, the process of "the user inputs communication content to the multi-person interaction platform through the multimodal input device" recorded in step S2 is completed;

[0062] Through the above process, the lower half of the face is physically shielded during sign language input, which effectively protects the facial privacy of special users and makes their facial information less likely to be leaked online.

[0063] Special user usage method 2: When the user needs to input lip reading (also a person with language impairment), such as Figure 8 and Fig. 9As shown, the upper baffle 62 is unrolled to move from top to bottom to cover the hollow area of ​​the upper half of the folding frame 5; then the person aligns his face with the inner side of the folding frame 5, and the unrolled baffle 62 can cover the upper half of the person's face, and the person's mouth is captured by the camera 4 through the hollow area of ​​the lower half of the folding frame 5. When the person performs lip reading, the camera 4 collects and recognizes it, and converts the lip reading content into text form and displays it on the operation screen 3, or converts it into voice and plays it for the person to confirm. When the content is consistent with the meaning of the content to be expressed, the text content can be sent to the multi-person interaction platform, and the process of "the user inputs communication content to the multi-person interaction platform through the multi-modal input device" recorded in step S2 is also completed;

[0064] Through the above process, the upper part of the face is physically shielded during lip reading input, which effectively protects the facial privacy of special users and also makes their facial information less likely to be leaked online.

[0065] Combination Figure 3 and Figure 6 As shown, a pair of vertical grooves 101 are provided at the front end of the terminal body 1. When the folding frame 5 is in contact with the surface of the terminal body 1, the bolts 64 are located inside the vertical grooves 101. By providing the vertical grooves 101, the bolts 64 can be stored conveniently during the folding process of the folding plate 2 and the folding frame 5. Figure 3 and Figure 4 As shown, the front end of the terminal body 1 and the outer end of the folding frame 5 are coated with a pair of magnetic coatings 7. When the folding frame 5 is in contact with the surface of the terminal body 1, the pair of magnetic coatings 7 on the folding frame 5 are respectively in contact with and adsorbed on the pair of magnetic coatings 7 on the terminal body 1, and the magnetic coatings 7 make the folded folding plate 2 more stable.

[0066] In view of current practical needs, the above-mentioned implementation mode adopted in this application is not limited to the scope of protection. Various changes made within the knowledge scope of technical personnel in this field without departing from the concept of this application still fall within the scope of protection of the present invention.

Claims

1. A multimodal model-driven method based on visual collaborative interaction technology, characterized in that: The following steps are involved: S1. The user successfully logs into the multi-person interactive platform of Hospital A by entering the corresponding medical card number, patient name and attending physician name in Hospital A, and enters the communication interface of the attending physician; S2, the user inputs the communication content into the multi-person interaction platform through the multimodal input device, and then the vocabulary recognition module performs feature recognition on the communication content to detect whether there are privacy words and negative words; When there are privacy words related to people, step S3 is performed; when no privacy words and negative words are identified, step S4 is performed; when there are no privacy words but negative words related to people or diseases are present, steps S4 and S5 are performed in sequence; S3, annotating the privacy words in the communication content through the prompt feedback module, and inputting the annotated communication content back to the multimodal input device, so that the user knows and re-enters the new communication content, and then performs the feature recognition process in step S2 again; S4, displaying the communication content on the interface of the multi-person interaction platform through the output display module for other users to view, thereby realizing the interaction process; S5. Collect and record the communication contents with negative words published by users for later review by attending physicians and authenticity verification process; When the content has a certain degree of authenticity, improvement suggestions can be sent to the corresponding attending physician. When the content lacks authenticity, the verification result can be sent to the user and the user is informed that the verification has been carried out. The multimodal input device comprises a terminal body (1), the front end of the terminal body (1) is rotatably connected to a folding plate (2), and the rotatable connection position of the two is located in the lower end area of ​​the terminal body (1), the outer end of the folding plate (2) is fixedly connected to an operation screen (3), and the operation screen (3) is located on the upper end surface of the folding plate (2) in a horizontal state, the front end of the terminal body (1) is fixedly connected to a camera (4), the end of the folding plate (2) away from the terminal body (1) is rotatably connected to a folding frame (5), and the folding frame (5) in a horizontal state is located at the upper end of the folding plate (2), the operation screen (3) is located on the inner side of the frame of the folding frame (5), and the inner end of the folding frame (5) is connected to a pair of rolling curtains (6), and the pair of rolling curtains (6) are respectively located in the upper and lower areas of the folding frame (5) in a vertical state; The rolling curtain (6) comprises a rotating rod (61), a blocking cloth (62) and a movable rod (63); the two ends of the rotating rod (61) are rotatably connected to the inside of the folding frame (5); one end of the blocking cloth (62) is fixedly connected to the outer end of the rotating rod (61), and the other end is fixedly connected to the movable rod (63); slideways (501) are provided at both left and right ends of the folding frame (5); the two ends of the movable rod (63) are respectively slidably connected to the inside of a pair of slideways (501) and extend to the outside; the blocking cloth (62) is pulled by the movable rod (63) moving up and down along the slideways (501), thereby realizing unwinding and rewinding of the blocking cloth (62) on the rotating rod (61); The rolling curtain (6) further comprises a pair of bolts (64); the inner wall of the slideway (501) is provided with a plurality of screw holes (502) communicating with the outside; the pair of bolts (64) are respectively threadedly connected to the inside of the pair of screw holes (502), and the ends of the bolts (64) are in contact with the movable rod (63); after the movable rod (63) is moved to a suitable position along the slideway (501), the movable rod (63) is positioned by the bolts (64), thereby fixing the unrolled blocking cloth (62).

2. The multimodal model driven method based on visual collaborative interaction technology according to claim 1, characterized in that: The authenticity verification process in step S5 includes the following steps: S5-1. In the communication interface of the corresponding attending physician, randomly select N users, where N is a positive integer greater than 2, and then convert the communication content with negative words published by the user in step S5 into question sentences, and send them to the above N users in the form of platform private messages, and wait for user feedback; S5-2. When the opinions fed back by users exceeding n / N are consistent with the communication content published by the user in step S5, n is a positive integer less than N, indicating that the communication content has a certain degree of authenticity, otherwise, it lacks authenticity.

3. The multimodal model driven method based on visual collaborative interaction technology according to claim 1, characterized in that: The multi-person interaction platform includes a login module, a multimodal input module, a vocabulary recognition module, a prompt feedback module, an output display module, a recording module and a content verification module. The multimodal input module is connected to the multimodal input device and is used to receive the communication content transmitted by the multimodal input device. The recording module is used to record the communication content with negative vocabulary published by the user. The content verification module is used to perform the authenticity verification process in step S5.

4. The multimodal model driven method based on visual collaborative interaction technology according to claim 3 is characterized by: The login module is connected to a department database, a patient database and a doctor database. The department database stores all department information of Hospital A, the patient database stores all patient information of Hospital A, and the doctor database stores all on-duty doctor information of Hospital A.

5. The multimodal model driven method based on visual collaborative interaction technology according to claim 1, characterized in that: A pair of vertical slots (101) are provided at the front end of the terminal body (1); when the folding frame (5) is in contact with the surface of the terminal body (1), the bolts (64) are located inside the vertical slots (101).

6. The multimodal model driven method based on visual collaborative interaction technology according to claim 1, characterized in that: The front end of the terminal body (1) and the outer end of the folding frame (5) are both coated with a pair of magnetic coatings (7); when the folding frame (5) is in contact with the surface of the terminal body (1), the pair of magnetic coatings (7) on the folding frame (5) are respectively in contact with and adsorbed on the pair of magnetic coatings (7) on the terminal body (1).

Images