Infrared thermal image assisted diagnosis system and method based on multi-modal large model

Abstract

The application relates to the field of artificial intelligence and medical equipment, in particular to an infrared thermal image auxiliary diagnosis system and method based on a multi-modal large model; the infrared thermal image auxiliary diagnosis system comprises an infrared collector, a visible light optical camera, a voice module, a physical sign detection module and a multi-modal large model; the infrared thermal image and clinical symptoms of a patient are cross-analyzed in combination with a multi-modal technology, and efficient, accurate and intelligent disease diagnosis effects on the patient are achieved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and medical devices, and in particular to an infrared thermogram-assisted diagnostic system and method based on a multimodal large model. Background Technology

[0002] With the continuous development of medical technology and people's increasing attention to health, the application prospects of infrared thermal imaging technology in the medical field are becoming increasingly broad. However, traditional disease diagnosis based on infrared thermography mainly relies on professional physicians, resulting in low diagnostic efficiency and reliance on physician experience for diagnostic results. Therefore, conducting research and consideration on disease diagnosis methods based on infrared thermography has certain practical value and progressive significance for improving the automation and intelligence of diagnosis, and enhancing diagnostic efficiency and effectiveness. Summary of the Invention

[0003] In view of this, this application provides an infrared thermography-assisted diagnostic system and method based on a multimodal large model, which aims to overcome the limitations of traditional infrared thermography in disease diagnosis and achieve efficient, accurate and intelligent disease diagnosis for patients.

[0004] Firstly, this application provides an infrared thermal imaging-assisted diagnostic system based on a multimodal large model: This infrared thermal imaging-assisted diagnostic system includes: Infrared sensor, used to collect infrared thermal images of the human body; Visible light optical cameras are used to acquire visible light images of the human body from multiple angles. The voice module is used to obtain the patient's chief complaint and pain information. The vital signs detection module is used to detect actions in the affected area; The multimodal large model includes a preprocessing module, a 3D reconstruction module, a mapping and matching module, and a disease diagnosis module; The preprocessing module is used to preprocess the human infrared thermal image; The three-dimensional reconstruction module is used to construct a three-dimensional point cloud image based on human visible light images acquired from multiple angles. The disease diagnosis module is used to determine the diseased area based on the preprocessed human infrared thermogram and the chief complaint information; The mapping and matching module is used to map the diseased area to a three-dimensional point cloud image to obtain a three-dimensional infrared thermogram; The disease diagnosis module is also used to receive pain information from the patient based on the detection action collected by the voice module, thereby determining the pain area based on the pain information; and outputting the diagnosis result based on the pain area.

[0005] Optionally, the preprocessing module includes a view selection module, an image restoration module, and an image enhancement module; The view selection module is used to select one or more human infrared thermal images from the front and rear data captured by the infrared thermal imaging scanner according to the chief complaint information. The image restoration module is used to determine the temperature difference between the patient's affected side and normal side based on the chief complaint information, and restores the human infrared thermal image selected by the view selection module into an RGB pseudo-color image based on the center temperature and temperature difference; The image enhancement module is used to convert RGB pseudo-color images into HSV images. After enhancing the hue and saturation channels, it is then converted back into RGB images.

[0006] Optionally, the view selection module consists of a pre-trained first BERT model and a text classification model; the pre-trained first BERT model converts the main complaint information into a first embedding vector; the text classification model predicts the human infrared thermal image to be selected based on the first embedding vector.

[0007] Optionally, the image restoration module includes a grayscale image restorer, a human pose recognition model, a pre-trained second BERT model, and a pseudo-color image generation network; The grayscale image restorer is used to map temperature data from a selected human infrared thermal image to the RGB pixel space from 0 to 255 to generate a grayscale image; The human pose recognition model is used to identify key points of the human body in grayscale images, including twenty parts: left upper arm aa, left forearm ab, left hand ac, right upper arm ba, right forearm bb, right hand bc, left thigh ca, left calf cb, left foot cc, right thigh da, right calf db, right foot dc, left chest ea, right chest eb, left abdomen ec, right abdomen ed, left buttock ee, right buttock ef, left brain fa, and right brain fb. The pre-trained second BERT model is used to encode the chief complaint information and generate a second embedding vector annotation to guide the localization of the affected area; The pseudo-color image generation network is used to generate RGB pseudo-color images from temperature data with the goal of maximizing the color difference between the affected and normal sides.

[0008] Optionally, the pseudo-color image generation network is configured with two learnable variables, namely center temperature (CT) and temperature width (TW). During training, the pseudo-color image generation network adopts self-supervised learning to adjust the values ​​of center temperature and temperature width with the goal of maximizing the color difference between the affected side and the normal side.

[0009] Optionally, the image enhancement module includes: Set two learnable parameters, Hz, Ho and Sz, So, for hue and saturation respectively, where H represents hue, S represents saturation, z represents scaling factor, and o represents offset; input a set of RGB pseudo-color images, and convert the RGB pseudo-color images into HSV images using the following formula:

[0010] In the formula, H' and S' represent the enhanced hue and saturation, respectively; Finally, the HSV image is restored to an RGB image.

[0011] Optionally, the 3D reconstruction module includes a feature extraction network, a feature alignment module, a multi-view fusion network, and a PointNet decoder; The feature extraction network extracts 2D semantic features from visible light images of the human body from each viewpoint and generates corresponding feature vectors; The feature alignment module is used to ensure that multi-view features remain consistent in space by minimizing the alignment loss, thereby achieving cross-view consistency of multi-view fused features. Multi-view fusion networks fuse image features from multiple perspectives into a unified 3D human body description through feature encoding. The PointNet decoder is used to decode the target human body mesh to obtain a 3D point cloud image that can represent a 3D human body model.

[0012] Optionally, the mapping and matching module includes a 3D-2D projection module, a pose alignment module, and a 2D-3D matching module; The 3D-2D projection module is used to project the vertices of a 3D point cloud image onto the view of an infrared acquisition instrument; The pose alignment module is used to calculate the pose transformation matrix to align the pose of the 3D point cloud image to the human pose space of the human infrared thermogram. The 2D-3D matching module is used to match 2D points of the diseased area to 3D projection points to obtain a three-dimensional infrared thermogram.

[0013] Secondly, this application provides an infrared thermographic-assisted diagnostic method based on a multimodal large model: This infrared thermography-assisted diagnostic method includes: Collect patient's chief complaint information, human infrared thermograms, and human visible light images from multiple angles; Constructing 3D point cloud images based on multi-angle visible light images of the human body; Preprocessing of human infrared thermal images; The diseased area was determined based on the pre-processed human infrared thermogram and the patient's chief complaint information; The diseased area is mapped onto a 3D point cloud image to obtain a 3D infrared thermal image; The procedure involves examining the affected area. Receive pain information from the patient based on the detection action, and determine the pain area based on the pain information; Diagnostic results are output based on the pain area.

[0014] Thirdly, this application provides a diagnostic device, including a processor, a memory, and a communication bus. The communication bus is used to establish a communication connection between the processor and the memory. The processor is used to execute a computer program stored in the memory to implement the infrared thermography-assisted diagnostic method described in the second aspect above.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program; the computer program can be executed by a processor to implement the infrared thermography-assisted diagnostic method as described in the second aspect above.

[0016] Fifthly, this application also provides a computer program product, including a computer program that can be executed by a processor to implement the infrared thermography-assisted diagnostic method as described in the second aspect above.

[0017] This application includes at least the following beneficial technical effects: 1. Traditional infrared thermography diagnosis requires physicians to manually analyze image features, which is time-consuming and depends on the physician's professional skills and work experience; this application adopts a multimodal large model, which can complete image preprocessing, feature extraction and disease probability prediction within seconds.

[0018] 2. This application is applicable to large-scale screening scenarios (such as occupational disease screening and chronic disease follow-up), improving efficiency by dozens of times and significantly alleviating the problem of strained medical resources.

[0019] 3. This application combines multimodal technology to cross-analyze infrared thermograms with patients' clinical symptoms, which can more accurately distinguish between rheumatoid arthritis and common joint inflammation, and can effectively reduce the misdiagnosis rate. Attached Figure Description

[0020] Figure 1 A schematic diagram of the framework of an infrared thermogram-assisted diagnostic system based on a multimodal large model provided in this application embodiment; Figure 2 This application provides a schematic diagram of the arrangement of a visible light optical camera and a vital sign detection module in an infrared thermography-assisted diagnostic system. Figure 3 A schematic diagram of an infrared thermogram-assisted diagnostic process based on a multimodal large model is provided for an embodiment of this application; Figure 4This application provides a schematic diagram of the module structure of an infrared thermogram-assisted diagnostic system based on a multimodal large model. Figure 5 This is a schematic diagram of the human infrared thermal image preprocessing process provided in the embodiments of this application; Figure 6 This is a schematic diagram of human body region segmentation provided in an embodiment of this application; Figure 7 This is a schematic diagram of the construction of a 3D point cloud image provided in an embodiment of this application; Figure 8 This is a schematic diagram of pain area calculation provided in an embodiment of this application; Figure 9 This is a schematic diagram of the diagnostic result output provided in the embodiments of this application; Figure 10 This application provides a schematic flowchart of an infrared thermogram-assisted diagnostic method based on a multimodal large model. Figure 11 This is a schematic diagram of a diagnostic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0023] In view of the limitations of traditional infrared thermography in disease diagnosis, the infrared thermography-assisted diagnostic system and method based on multimodal large model provided in this application can effectively solve the above-mentioned technical problems and achieve efficient, accurate and intelligent disease diagnosis for patients.

[0024] The infrared thermographic-assisted diagnostic system and method based on a multimodal large model provided in this application can be used as follows: Figure 1 The system architecture implementation shown mainly includes: The system includes an infrared thermal imaging scanner, a visible light optical camera array, a bed frame, a vital signs detection module, a multimodal large model, and a voice module. The visible light optical camera array is arranged around the bed frame, with all cameras facing the bed frame. The vital signs detection module includes a self-sensing robotic arm, which is located on the side of the bed frame.

[0025] The multimodal large model fuses and analyzes features from infrared thermal images captured by an infrared thermal imaging scanner and patient complaints collected by a voice module to identify abnormal temperature areas in the infrared thermal images, which are then designated as diseased areas. A visible light optical camera array captures multi-angle visible light images of the patient lying on a bed frame. The multimodal large model generates a 3D model of the patient (i.e., a 3D point cloud image) based on these multi-angle visible light images using a 3D reconstruction module. The multimodal large model maps the diseased areas in the infrared thermal images to the 3D point cloud image using a mapping and matching module, obtaining a 3D infrared thermal image, which is denoted as Ω. A vital sign detection module controls a robotic arm to hammer and press on the corresponding diseased area of ​​the patient within the Ω region of the 3D infrared thermal image, while the voice module records whether the patient experiences pain or abnormalities to determine the pain area. The multimodal large model outputs diagnostic results based on the patient's responses, including disease type, diseased area, and treatment plan, where the treatment plan includes treatment methods and treatment points.

[0026] Based on the multimodal large model-based infrared thermographic-assisted diagnostic system provided in this application, the automatic collection and analysis of disease information can be realized, and diagnostic results can be output. This solves the limitations of traditional infrared thermographic disease diagnosis and achieves efficient, accurate and intelligent disease diagnosis for patients.

[0027] In an optional embodiment of this application, reference is made to Figure 2 The visible light optical camera array contains at least five synchronized cameras, which can be arranged at the head, tail, left and right sides of the bed frame and above the bed frame to achieve omnidirectional acquisition of visible light images of the human body from five directions. Specifically, the upper camera is located 2m directly above the bed frame, and the other cameras are 1m away from the edge of the bed frame and 0.5m above the bed frame, while maintaining a 30-degree downward tilt to ensure that the entire area of ​​the human body is covered. An FPGA synchronization board is used to ensure that multiple cameras capture images simultaneously.

[0028] In an optional embodiment of this application, the Zhang Zhengyou calibration method is used before the visible light optical camera array is used to calculate the camera focal length, principal point, and distortion coefficient using a high-precision checkerboard grid to determine the camera's intrinsic and extrinsic parameters.

[0029] In an optional embodiment of this application, the vital signs detection module includes two robotic arms, respectively positioned on the left and right sides of the bed frame. Each robotic arm comprises a six-axis robotic arm and a self-sensing robotic hand. Based on the patient's affected area, a trajectory planning algorithm controls the movement trajectory of the robotic arms, ensuring that the robotic arms do not collide with the affected area. The self-sensing robotic hand is mounted at the end of the six-axis robotic arm and can perform both pressing and hammering actions. The self-sensing robotic hand has tactile sensors on its fingers and ulnar side to sense the force of the pressing or hammering, preventing injury to the patient.

[0030] In an optional embodiment of this application, the infrared thermal imaging scanner consists of an infrared darkroom and an infrared camera. The infrared camera scans the patient's body, capturing infrared thermal images of the front, back, left, and right sides of the body by the patient turning around. After the images are captured, the infrared thermal image data is uploaded to a multimodal large model.

[0031] The voice module collects the patient's chief complaint and pain information, converts it into text, and then uploads the text information to the multimodal large model. The voice module can also receive text information output by the multimodal large model, convert it into a speech signal, and play it to prompt the patient to describe their chief complaint in more detail or to remind the patient of the next steps.

[0032] The bed frame is used to assist in the diagnosis of patients' vital signs. The voice module can remind patients of their posture (supine, lateral, prone) and direction (head of bed, foot of bed) while lying on the bed frame according to diagnostic needs.

[0033] like Figure 3 As shown, this embodiment provides an infrared thermogram-assisted diagnostic process based on a multimodal large model. The specific process includes: S1: The patient enters the infrared thermal imaging scanner to capture infrared thermal images of the front, back, left and right sides of the body, and the infrared thermal images of the body are transmitted to a multimodal large model.

[0034] S2: The patient describes their chief complaint, which is then converted into text information by the voice module and transmitted to the multimodal big data model. The multimodal big data model determines the patient's disease type and affected area based on the human body infrared thermogram and the chief complaint.

[0035] S3: The patient lies on the bed frame, and a visible light optical camera array takes a full-body photo of the patient; the three-dimensional reconstruction module performs three-dimensional reconstruction on the captured visible light image of the human body to obtain a three-dimensional point cloud image of the patient's body.

[0036] S4: The diseased area in the human infrared thermogram is mapped to the three-dimensional point cloud image through the mapping and matching module to obtain the diseased area in the three-dimensional infrared thermogram.

[0037] S5: The vital signs detection module performs pressing or hammering actions on the affected area. The voice module records the patient's voice feedback data on the detection actions and converts it into text, which is then transmitted to the multimodal large model in real time.

[0038] S6: The multimodal large model combines infrared thermograms, patient complaints and physical signs to output diagnostic results, including the patient's treatment plan, such as moxibustion, acupuncture and other treatment methods and treatment points.

[0039] During the diagnosis process, the multimodal large model integrates the patient's infrared thermogram and chief complaint information through the disease diagnosis module, and infers the affected area in the patient's infrared thermogram. In addition, it can also integrate the patient's vital signs information (pain area) to further obtain suggested treatment methods and treatment points. The 3D reconstruction module constructs a 3D point cloud image of the patient by calculating the visible light image of the human body captured by the visible light optical camera array. The mapping and matching module integrates the affected area calculated by the disease diagnosis module and the 3D point cloud image data of the patient constructed by the 3D reconstruction module, and calculates the affected area in the patient's 3D infrared thermogram. The vital sign detection module plans and calculates the motion trajectory based on the affected area in the 3D infrared thermogram and its own spatial position through a robotic arm control algorithm, so as to control the robotic arm to move to the patient's affected area to perform detection actions such as hammering or pressing to detect the patient's vital signs and obtain the vital sign detection results (i.e., pain area). The disease diagnosis module outputs the diagnosis results based on the pain area.

[0040] refer to Figure 4 This application provides an infrared thermal imaging-assisted diagnostic system based on a multimodal large model, comprising: Infrared sensor, used to collect infrared thermal images of the human body; Visible light optical cameras are used to acquire visible light images of the human body from multiple angles. The voice module is used to obtain the patient's chief complaint and pain information. The vital signs detection module is used to detect actions in the affected area; The multimodal large model includes a preprocessing module, a 3D reconstruction module, a mapping and matching module, and a disease diagnosis module. The preprocessing module is used to preprocess the human infrared thermogram. The 3D reconstruction module is used to construct a 3D point cloud image based on the human visible light images acquired from multiple angles. The disease diagnosis module is used to determine the diseased area based on the preprocessed human infrared thermogram and the patient's complaint information. The mapping and matching module is used to map the diseased area to the 3D point cloud image to obtain a 3D infrared thermogram. The disease diagnosis module is also used to receive pain information from the patient based on the detected actions collected by the voice module, thereby determining the pain area based on the pain information and outputting the diagnosis result based on the pain area.

[0041] Optionally, the multimodal large model incorporates knowledge of disease physiological mechanisms during the pre-training stage to enhance the accuracy and professionalism of the model in diagnosing diseases. The multimodal large model first uses a preprocessing module to screen, visualize, and enhance the features of infrared thermograms, and then combines the preprocessed infrared thermograms with the patient's complaints to deduce the patient's disease type and affected area.

[0042] Optionally, during the pre-training phase, the multimodal large model uses real patient complaints and infrared thermograms as input, and disease types and disease regions diagnosed by professional physicians as standard outputs. A fine-tuning-based approach is then used to enable the large model to learn methods related to disease diagnosis. It should be noted that the output of the disease region is not a directly generated image, but rather a text-based coordinate point, which is then post-processed to transform it into a closed disease region.

[0043] By fine-tuning a large multimodal model, we can make it output the content we want. Specifically, during fine-tuning, we need to prepare an infrared heatmap and the subject's complaint as input, and disease type and disease region as the standard output. Initially, the model's output will differ from the standard output, at which point it will be "penalized" (i.e., by a loss function), forcing it to move closer to the standard output. After some fine-tuning, the model's output will be essentially consistent with the standard output (at which point the model gains the ability to output disease type and disease region). In principle, the large model uses an image encoder and a text encoder to encode the infrared heatmap and the subject's complaint respectively (the text encoder generates dynamic text feature vectors by capturing inter-word dependencies, while the image encoder generates image feature vectors by extracting spatial features of the image, such as object contours and textures). Then, a cross-attention mechanism is used to fuse these two types of features. Finally, the text decoder receives the multimodal features and generates text word by word in an autoregressive manner.

[0044] refer to Figure 5The preprocessing module includes a view selection module, an image restoration module, and an image enhancement module. The view selection module is used to select one or more human infrared thermal images from the front and rear data captured by the infrared thermal imaging scanner based on the patient's complaint information. The image restoration module is used to determine the temperature difference between the affected side and the normal side of the patient and between the affected area and the adjacent area based on the patient's complaint information, and restores the human infrared thermal image selected by the view selection module into an RGB pseudo-color image based on the center temperature and temperature difference. The image enhancement module is used to convert the RGB pseudo-color image into an HSV image, enhance the hue and saturation channels, and then convert it back into an RGB image.

[0045] Optionally, the view selection module consists of a pre-trained first BERT model and a text classification model. The pre-trained first BERT model transforms the subject information into a first embedding vector. The text classification model predicts the human infrared thermal image to be selected based on the first embedding vector. Specifically, the first BERT model first segments the text information (i.e., the subject), with each segmented word serving as a token. Then, each token is converted into a 768-dimensional vector, i.e., the input vector, which contains Token Embedding (i.e., lexical semantic encoding), Segment Embedding (i.e., sentence classification), and Position Embedding (i.e., positional information). Finally, a multi-layer Transformer encoder is used to calculate the dependencies between different input vectors and generate the first embedding vector.

[0046] Optionally, the text classification model consists of a Transformer decoder based on an autoregressive mechanism. It utilizes self-attention and cross-attention mechanisms to calculate the probability distribution of the next word in the output sentence based on the first embedding vector generated by the first BERT model, selects the word with the highest probability, and finally forms the infrared heatmap selection result.

[0047] Optionally, the pre-trained first BERT model and text classification model use patient complaints as input and infrared thermogram selection results from professional physicians as standard output during the pre-training phase. A fine-tuning-based method is used to enable the pre-trained first BERT model and text classification model to learn relevant methods for infrared thermogram selection.

[0048] Optionally, the image restoration module includes a grayscale image restorer, a human pose recognition model, a pre-trained second BERT model, and a pseudo-color image generation network. The grayscale image restorer maps the temperature data from the selected human infrared thermal image to the RGB pixel space from 0 to 255 to generate a grayscale image. The human pose recognition model identifies key points of the human body in the grayscale image and then divides the human body into regions based on the key point information, including twenty parts: left upper arm aa, left forearm ab, left hand ac, right upper arm ba, right forearm bb, right hand bc, left thigh ca, left calf cb, left foot cc, right thigh da, right calf db, right foot dc, left chest ea, right chest eb, left abdomen ec, right abdomen ed, left buttock ee, right buttock ef, left brain fa, and right brain fb. (See reference) Figure 6 A pre-trained second BERT model is used to encode the patient's complaint information and generate a second embedding vector to guide the localization of the affected area. A pseudo-color image generation network is used to generate an RGB pseudo-color image from the temperature data, aiming to maximize the color difference between the affected and normal sides, as well as between the affected area and its neighboring areas. For example, if a patient reports left abdominal pain for 3 days, the affected side (affected area) would be the left abdomen (ec), the normal side would be the right abdomen (ed), and the neighboring areas would be the left chest (ea) and left buttock (ee).

[0049] Optionally, the pre-trained second BERT model uses real patient complaints during the pre-training phase and uses the judgment of professional physicians on the affected side and normal side of the patient as the standard output. Fine-tuning is used to enable the pre-trained second BERT model and the region selection module of the pseudo-color image generation network to learn relevant methods for selecting the affected side and normal side.

[0050] Optionally, the human pose recognition model uses a pre-trained OpenPose model, which is trained on 10,000 grayscale human images and can recognize 18 key points of the human body, including nose 0, neck 1, right shoulder 2, right elbow 3, right wrist 4, left shoulder 5, left elbow 6, left wrist 7, right hip 8, right knee 9, right ankle 10, left hip 11, left knee 12, left ankle 13, right eye 14, left eye 15, right ear 16, and left ear 17.

[0051] Optionally, the pseudo-color image generation network consists of a region selection module and a temperature parameter adjustment module. The region selection module comprises a Transformer decoder based on an autoregressive mechanism, which decodes the embedding vectors generated from the pre-trained second BERT model into textual information containing the region selection results. These results include the location information of the affected side and normal side of the patient, as well as the affected area and adjacent regions. It should be noted that the affected side and the affected area refer to the same location.

[0052] Optionally, the temperature parameter adjustment module of the pseudo-color image generation network is set with two learnable variables: center temperature (CT) and temperature width (TW). During training, the pseudo-color image generation network adopts self-supervised learning to adjust the values ​​of center temperature and temperature width with the goal of maximizing the color difference between the affected side and the normal side, as well as between the affected area and its neighboring areas.

[0053] Optionally, the pre-trained second BERT model uses real patient complaints during the pre-training phase and uses the judgment of professional physicians on the affected side and normal side of the patient as the standard output. Fine-tuning is used to enable the pre-trained second BERT model and the region selection module of the pseudo-color image generation network to learn relevant methods for selecting the affected side and normal side.

[0054] Optionally, the image enhancement module includes: Set two learnable parameters, Hz, Ho and Sz, So, for hue and saturation respectively, where H represents hue, S represents saturation, z represents scaling factor, and o represents offset; input a set of RGB pseudo-color images, and convert the RGB pseudo-color images into HSV images using the following formula:

[0055] In the formula, H' and S' represent the enhanced hue and saturation, respectively; Finally, the HSV image is restored to an RGB image. For example, the hsv_to_rgb function in the Kornia library can be used to restore an HSV image to an RGB image.

[0056] Optionally, the 3D reconstruction module includes a feature extraction network, a feature alignment module, a multi-view fusion network, and a PointNet decoder. The feature extraction network extracts 2D semantic features from the image at each viewpoint and generates corresponding feature vectors. The feature alignment module minimizes the alignment loss to ensure spatial consistency of multi-view features, achieving cross-view consistency of multi-view fusion features. The multi-view fusion network fuses image features from multiple views into a unified 3D human body description through feature encoding. The PointNet decoder decodes the target human body mesh to obtain a 3D point cloud image that can represent a 3D human body model. Figure 7 As shown.

[0057] Optionally, the feature extraction network consists of a ResNet network, encoded using the following formula:

[0058] in, For feature extraction networks, This represents the spatial resolution after downsampling. For the number of channels, For the number of views, For the first Zhang image.

[0059] Optionally, the feature alignment module will align each feature map Alignment is performed based on the camera pose transformation to a shared 3D spatial coordinate system. Specifically, for a given point in an image, another image containing that point is selected, and they are denoted as follows: In the corresponding pixel If the point is detected by feature matching, its spatial coordinates can be estimated by triangulation. :

[0060] in For camera projection function, For depth. Then the feature map Each pixel The corresponding points are back-projected into the camera coordinate system and then projected again into the world coordinate system, as shown in the following formula:

[0061]

[0062] in, For the first Internal parameters of a camera, and For camera external parameters, The 3D position of the point in the camera coordinate system. This refers to the 3D coordinates of a point transformed from the camera coordinate system back to the world coordinate system. Finally, all... The average fusion method is used to map the data to a unified 3D mesh, denoted as... The formula is as follows:

[0063] in, This is a mapping function.

[0064] Optionally, the multi-view fusion network uses Transformer as the base network to fuse the aligned features as follows:

[0065] in, For Transformer networks, The fused feature vector It is the number of channels of 3D spatial features obtained after fusing multiple perspectives.

[0066] Optionally, a point cloud can be generated from the fused features using the PointNet decoder:

[0067] in, is the PointNet decoder, and P is the final point cloud.

[0068] Optionally, the 3D reconstruction module uses three loss functions to supervise the 3D reconstruction process. Specifically, these include chamfering loss, alignment loss, and surface smoothing loss, with the following formulas:

[0069] Where G is the true value of the point cloud. and These are the real point cloud and the predicted point cloud, respectively. and Features mapped from different perspectives Let E be the normal vector of the adjacent point, and E be the edge set of the point cloud. For the total loss, , , These represent the proportions of the three types of losses.

[0070] chamfer loss L CD Alignment loss L measures the distance between the reconstructed point cloud and the true point cloud. Reducing this loss allows the reconstructed point cloud to more closely resemble the true shape, optimizing the overall geometry. align Constrained reconstruction results are aligned with multi-view images, improving the model's consistency and accuracy across different perspectives and solving the multi-view matching problem. Surface smoothing loss L smooth This process aims to smooth reconstructed surfaces, suppress noise and unreasonable bumps, enhance the visual plausibility of the model's surface, and conform to the characteristics of real-world object surfaces. These three aspects work together to improve geometric accuracy and multi-view accuracy. Figure 1 The model optimizes three aspects: consistency, surface quality, and reliability, thereby improving the realism and reliability of the 3D point cloud model.

[0071] Optionally, the disease diagnosis module outputs a set of pixels in the diseased area based on the preprocessed human infrared thermogram and the patient's complaint, and then uses image masking technology to mask the diseased area.

[0072] Optionally, the mapping and matching module includes a 3D-2D projection module, a pose alignment module, and a 2D-3D matching module; the 3D-2D projection module is used to project the vertices of the 3D point cloud image onto the infrared acquisition instrument view; the pose alignment module is used to calculate the pose transformation matrix to align the pose of the 3D point cloud image to the human pose space of the human infrared thermogram; the 2D-3D matching module is used to match the 2D points of the diseased area to the 3D projection points, output the matched 3D target area, and thus obtain the 3D infrared thermogram.

[0073] Optionally, the process of projecting the vertices of the 3D point cloud image onto the human infrared thermal image specifically involves: for each vertex in the 3D point cloud image... Calculate its in the first Projected coordinates of an infrared camera view The formula is as follows:

[0074]

[0075]

[0076]

[0077]

[0078] in, For camera focal length, and These are the width and height, which are the actual physical dimensions of a single pixel, respectively. and These respectively represent the camera at and Principal point along the axis, and The first The rotation and translation matrices of the extrinsic parameters of an infrared camera. These are the coordinates of vertices in the 3D model after transformation to the camera coordinate system. , and These represent the camera coordinates in the X, Y, and Z axes, respectively.

[0079] Furthermore, the specific process of the pose alignment module is as follows: Minimize the projected joints using an iterative optimization algorithm. With infrared joints The distance is given by the following formula:

[0080] The above optimization problem is solved using existing related algorithms to obtain the attitude transformation matrix. It can align the pose of a 3D point cloud image to the human pose space of a human infrared thermal image.

[0081] Furthermore, the process by which the 2D-3D matching module matches 2D points of the diseased area to 3D projection points is as follows: given the target points of the diseased area in the infrared thermal image... Calculate the projected coordinates of all visible vertices. and Given the pixel distance, find the vertex with the smallest distance, as shown in the following formula:

[0082] In an optional embodiment of this application, the process of the 2D-3D matching module matching 2D points of the diseased area to 3D projection points is specifically as follows: for two-dimensional points of the target area in the infrared image... First, represent it as homogeneous coordinates. Using the attitude transformation matrix and camera internal reference Perform the inverse transform, the formula is as follows:

[0083] in These are the corresponding points in the 3D point cloud image. The ultimate goal is to map the diseased areas in the infrared image to the 3D point cloud image.

[0084] The self-sensing robotic arm can perform both pressing and hammering actions. Specifically, the self-sensing robotic arm first hammers the diseased area and records the areas where the patient feels pain. Then, it presses on the same area and marks the points where the patient still feels pain, thus identifying the specific pain points.

[0085] The self-sensing robotic arm has tactile sensors on its fingers and ulnar side, which can sense the force of pressing or hammering to avoid injuring the patient. The self-sensing robotic arm can move to the patient's affected area according to the calculation results of the mapping and matching module, and automatically calculate the movement trajectory according to the current task (hammering, pressing). Specifically, any existing related path planning algorithm can be used, as long as it can realize the action of detecting the affected area. This embodiment does not limit this.

[0086] The voice module collects pain information from patients based on their actions during detection, thereby determining the pain region. For example, when a detection action is performed on a point in the patient's affected area, if the patient reports pain, this indicates that the point is a pain point. All pain points are clustered using a 3-centimeter reference point to form a point set. For a point set with only one point, a circle with that point as its center and a radius of 1 centimeter is selected as the pain region. For a point set with two points, a line segment with the two points as endpoints and a width of 2 centimeters is selected as the pain region. For point sets with three or more points, a convex hull algorithm is used to calculate the minimum edge-enclosed polygon of the point set, and this polygon is selected as the pain region. (Reference) Figure 8 .

[0087] The disease diagnosis module outputs diagnostic results based on the pain area, such as... Figure 9 As shown in the figure. In this embodiment, the diagnostic results include the disease type and the suggested treatment plan, where the treatment plan may include treatment methods and treatment points, such as moxibustion, acupuncture, and other treatment methods and treatment points. Specifically, the multimodal large language model in the disease diagnosis module incorporates a multi-round question-and-answer process training during the pre-training stage. The first round input of the large model is the patient's chief complaint and heat map, and the output is the patient's disease type and disease area. The second round input of the large model is the patient's physical sign detection results, and the output is the treatment method and treatment point. In particular, the large model determines the treatment method based on the disease type and physical sign detection results, according to the medical knowledge base. The selection of treatment points is related to the treatment method. When using moxibustion treatment, the multimodal large model selects treatment points based on the moxibustion thermodynamic model and the pain area to ensure that the pain area receives heat therapy evenly. When using acupuncture treatment, the pain point is directly selected as the treatment point.

[0088] In this embodiment, the large-scale model determines the disease type and affected area (which is directly present in the infrared thermogram) based on the infrared thermogram and the patient's complaint. The vital sign detection module identifies the pain point or pain area based on the affected area. Finally, the large-scale model comprehensively considers the disease type and the pain point / pain area to determine the treatment point. For example, if a patient's complaint is "I have had right shoulder pain for 3 months," and the infrared thermogram shows a warming change in the right shoulder, the multimodal large-scale model can determine that the patient has frozen shoulder and identify the affected area. The vital sign detection module detects the patient's shoulder area and obtains the following results: Figure 8 The results for the pain area are shown. Finally, this result is input into a larger model (patient experiences shoulder tenderness upon palpation, more pronounced on the right side).

[0089] Based on the infrared thermography-assisted diagnostic system provided in this embodiment, by adopting a multimodal large model, image preprocessing, feature extraction, and disease probability prediction can be completed within seconds. By combining multimodal technology and cross-analyzing infrared thermography with patients' clinical symptoms, it is possible to more accurately distinguish between rheumatoid arthritis and common joint inflammation, which can effectively reduce the diagnosis rate.

[0090] Based on the above-described infrared thermogram-assisted diagnostic system embodiment, this embodiment also provides an infrared thermogram-assisted diagnostic method based on a multimodal large model.

[0091] refer to Figure 10 An infrared thermogram-assisted diagnostic method based on a multimodal large model includes: S101. Collect patient's chief complaint information, human infrared thermogram, and human visible light images from multiple angles; S102. Construct a three-dimensional point cloud image based on multi-angle visible light images of the human body; S103. Preprocess the infrared thermal image of the human body; S104. Determine the diseased area based on the preprocessed human infrared thermogram and the chief complaint information; S105. Map the diseased area onto a three-dimensional point cloud image to obtain a three-dimensional infrared thermal image; S106. Perform detection actions on the affected area; S107. Receive pain information from the patient based on the detection action, and determine the pain area based on the pain information; S108. Output diagnostic results based on the pain area.

[0092] Through the foregoing detailed description of the system, those skilled in the art can clearly understand the specific implementation process of the method in this embodiment. For the sake of brevity, it will not be described in detail here.

[0093] To better execute the above methods, embodiments of this application also provide a diagnostic device, such as... Figure 11 As shown, the diagnostic device includes a processor, a memory, and a communication bus for enabling communication between the processor and the memory.

[0094] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the infrared thermography-assisted diagnostic method provided in the above embodiments, etc. The data storage area may store data involved in the infrared thermography-assisted diagnostic method provided in the above embodiments, etc.

[0095] Optionally, the memory may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory exists independently and is connected to the processor via a communication bus, or the memory is integrated with the processor.

[0096] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0097] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0098] In an optional embodiment, the diagnostic device may further include a communication interface (not shown) for communication between the diagnostic device and other devices.

[0099] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the infrared thermal imaging-assisted diagnostic method of the above embodiments.

[0100] This application also provides a computer program product comprising a computer program tangibly contained on a readable medium thereof, the computer program containing program code for performing any of the infrared thermography-assisted diagnostic methods described in this application, the computer program being downloadable and installable over a network, and / or installed from a removable medium (such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc.).

[0101] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. An infrared thermographic-assisted diagnostic system based on a multimodal large model, characterized in that, The infrared thermography-assisted diagnostic system includes: Infrared sensor, used to collect infrared thermal images of the human body; Visible light optical cameras are used to acquire visible light images of the human body from multiple angles. The voice module is used to obtain the patient's chief complaint and pain information. The vital signs detection module is used to detect diseased areas in three-dimensional infrared thermograms. The multimodal large model includes a preprocessing module, a 3D reconstruction module, a mapping and matching module, and a disease diagnosis module; The preprocessing module is used to preprocess human infrared thermal images, including a view selection module, an image restoration module, and an image enhancement module. The view selection module selects one or more human infrared thermal images from the front and rear data captured by the infrared thermal imaging scanner based on the patient's complaint information. The image restoration module determines the temperature difference between the affected and normal sides of the patient based on the complaint information, and restores the human infrared thermal image selected by the view selection module into an RGB pseudo-color image based on the center temperature and temperature difference. The image enhancement module converts the RGB pseudo-color image into an HSV image, enhances the hue and saturation channels, and then converts it back into an RGB image. The image restoration module includes a grayscale image restorer, a human pose recognition model, a pre-trained second BERT model, and a pseudo-color image generation network. The complexing unit maps the temperature data from the selected human infrared thermogram to the RGB pixel space from 0 to 255 to generate a grayscale image. The human pose recognition model identifies key points of the human body in the grayscale image, including twenty parts: left upper arm aa, left forearm ab, left hand ac, right upper arm ba, right forearm bb, right hand bc, left thigh ca, left calf cb, left foot cc, right thigh da, right calf db, right foot dc, left chest ea, right chest eb, left abdomen ec, right abdomen ed, left buttock ee, right buttock ef, left brain fa, and right brain fb. The pre-trained second BERT model encodes the chief complaint information and generates a second embedding vector to guide the localization of the affected side. The pseudo-color image generation network generates an RGB pseudo-color image from the temperature data with the goal of maximizing the color difference between the affected and normal sides. The three-dimensional reconstruction module is used to construct a three-dimensional point cloud image based on human visible light images acquired from multiple angles. The disease diagnosis module is used to determine the diseased area based on the preprocessed human infrared thermogram and the chief complaint information; The mapping and matching module is used to map the diseased area to a three-dimensional point cloud image to obtain a three-dimensional infrared thermogram; The disease diagnosis module is also used to receive pain information from the patient based on the detection action collected by the voice module, thereby determining the pain area based on the pain information; and outputting the diagnosis result based on the pain area.

2. The infrared thermography-assisted diagnostic system as described in claim 1, characterized in that, The view selection module consists of a pre-trained first BERT model and a text classification model; the pre-trained first BERT model converts the main complaint information into a first embedding vector; the text classification model predicts the human infrared thermal image to be selected based on the first embedding vector.

3. The infrared thermography-assisted diagnostic system as described in claim 1, characterized in that, The pseudo-color image generation network is configured with two learnable variables: center temperature (CT) and temperature width (TW). During training, the pseudo-color image generation network adopts self-supervised learning to adjust the values ​​of center temperature and temperature width with the goal of maximizing the color difference between the affected side and the normal side.

4. The infrared thermography-assisted diagnostic system as described in claim 1, characterized in that, The image enhancement module includes: Set two learnable parameters, Hz, Ho and Sz, So, for hue and saturation respectively, where H represents hue, S represents saturation, z represents scaling factor, and o represents offset; input a set of RGB pseudo-color images, and convert the RGB pseudo-color images into HSV images using the following formula: In the formula, H' and S' represent the enhanced hue and saturation, respectively; Finally, the HSV image is restored to an RGB image.

5. The infrared thermography-assisted diagnostic system as described in any one of claims 1-4, characterized in that, The 3D reconstruction module includes a feature extraction network, a feature alignment module, a multi-view fusion network, and a PointNet decoder; The feature extraction network is used to extract 2D semantic features of human visible light images from each viewpoint and generate corresponding feature vectors; The feature alignment module is used to ensure that multi-view features remain consistent in space by minimizing the alignment loss, thereby achieving cross-view consistency of multi-view fused features. Multi-view fusion networks fuse image features from multiple perspectives into a unified 3D human body description through feature encoding. The PointNet decoder is used to decode the target human body mesh to obtain a 3D point cloud image that can represent a 3D human body model.

6. The infrared thermography-assisted diagnostic system according to any one of claims 1-4, characterized in that, The mapping and matching module includes a 3D-2D projection module, a pose alignment module, and a 2D-3D matching module; The 3D-2D projection module is used to project the vertices of a 3D point cloud image onto the view of an infrared acquisition instrument; The pose alignment module is used to calculate the pose transformation matrix to align the pose of the 3D point cloud image to the human pose space of the human infrared thermogram. The 2D-3D matching module is used to match 2D points of the diseased area to 3D projection points to obtain a three-dimensional infrared thermogram.

Images