Portable direct display intelligent AI diagnostic funduscope

Abstract

The application relates to the technical field of medical treatment, in particular to a portable intelligent AI diagnosis funduscope capable of direct display, which comprises a voice anti-interference interaction module, a closed-loop anti-shake control module and a fundus image splicing module. The voice anti-interference interaction module collects voice instructions of a user, carries out noise reduction processing on the voice instructions, analyzes the voice instructions into execution instructions of a control device, the closed-loop anti-shake control module monitors motion shaking data of the device in real time, generates reverse motion compensation through calculation, and offsets shaking caused by the handheld device, and the fundus image splicing module extracts key features of multiple local fundus photos, aligns and fuses the multiple local fundus photos into a complete panoramic fundus photo. The lesion contrast diagnosis module aligns two fundus photos before and after treatment. In the application, the voice anti-interference interaction module and the closed-loop anti-shake control module are arranged, the risk of respiratory tract droplet infection in traditional close-range examination is effectively avoided, and the extreme high definition and stability of the fundus image in the portable handheld state are ensured.

Description

Technical Field

[0001] This invention relates to the field of medical technology, and in particular to a portable, intelligent AI-powered diagnostic fundus endoscope with direct display capability. Background Technology

[0002] The commonly used direct ophthalmoscope in clinical practice mainly consists of basic optical and mechanical components such as a light source, lenses, grids, and diopter adjustment discs. During the examination, the doctor needs to observe the fundus structures illuminated by coaxial light in real time through the viewing port and manually adjust the parameters.

[0003] The examiner must hold the patient close to their hands during the examination, which not only increases the risk of respiratory infection, but also causes blurred images, a very small field of view and no panoramic view due to hand tremors. In addition, the equipment cannot accurately extract the differences in lesions before and after treatment and generate an objective and visual comparative description. Patients also cannot accurately refer their condition to the appropriate specialist, which hinders the implementation of hierarchical diagnosis and treatment and intelligent remote screening in ophthalmology. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a portable intelligent AI diagnostic fundus endoscope with direct display, which aims to improve the problems of easy cross-infection, narrow field of vision due to hand tremors and inability to retain panoramic digital data, as well as the inability to automatically match medical resources.

[0005] In a first aspect, the present invention provides the following technical solution: a portable intelligent AI diagnostic fundus endoscope system with direct display capability, comprising:

[0006] The voice anti-interference interaction module collects the user's voice commands, performs noise reduction processing on the voice commands, and parses them into execution commands for controlling the device;

[0007] The closed-loop anti-shake control module monitors the device's motion and vibration data in real time, and generates reverse motion compensation through calculation to counteract the shaking caused by handheld device;

[0008] The fundus image stitching module extracts key features from multiple local fundus photos, aligns and merges these photos into a complete panoramic fundus image.

[0009] The lesion comparison and diagnosis module aligns two fundus photos taken before and after treatment, calculates and extracts the lesions that differ between the two fundus photos, marks the lesions with different colors, and outputs a diagnostic description.

[0010] The medical resource matching module filters and matches corresponding ophthalmologists based on the diagnosis description and according to preset weight conditions.

[0011] Preferably, in the voice anti-interference interaction module, the step of collecting the user's voice commands and performing noise reduction processing on the voice commands includes:

[0012] It captures bone voiceprint signals transmitted by vocal cord vibration through a built-in vibration sensor;

[0013] Voice signals transmitted through the air are collected using a microphone;

[0014] The bone voiceprint signal is mixed with the speech signal for pickup and data fusion, filtering out environmental noise interference and preserving high-frequency details of the speech.

[0015] Preferably, in the voice anti-interference interaction module, the step of parsing it into execution instructions for the control device includes:

[0016] Analyze the energy parameters and zero-crossing rate parameters of the acquired speech signal to determine the start and end points of the speech.

[0017] Based on the start and end points, silent segments are removed to generate a clean signal and match it to generate the corresponding execution instructions.

[0018] Preferably, in the closed-loop anti-shake control module, the step of real-time monitoring of the device's motion jitter data and generating reverse motion compensation through calculation includes:

[0019] The device acquires real-time displacement and angular velocity data on the pitch, yaw, and roll axes using built-in gyroscopes and accelerometers.

[0020] A filtering algorithm is used to remove random noise from the displacement and angular velocity change data;

[0021] Based on the data after noise removal, the direction of jitter change is predicted, the required reverse compensation value is calculated, and the internal micro motor is driven to generate a precise reverse motion to counteract the jitter.

[0022] Preferably, in the fundus image stitching module, the step of extracting key features from multiple local fundus photos, aligning the multiple local fundus photos, and fusing them into a complete panoramic fundus photo includes:

[0023] Extract feature points and line features from the multiple local fundus images;

[0024] The geometric transformation relationship between the multiple local fundus images is established by feature matching, and the homography matrix is ​​calculated.

[0025] Based on the homography matrix, all local fundus images are transformed to the same coordinate system, and edge gradient fusion is performed by weighted averaging to eliminate stitching seams.

[0026] Preferably, in the lesion comparison and diagnosis module, the step of aligning the two fundus photographs before and after treatment includes:

[0027] Identify the optic disc and macular positions in two fundus photographs taken before and after treatment;

[0028] The position of the optic disc and the position of the macula are used as fixed reference anchor points;

[0029] By controlling the image to perform translation, scaling, and rotation operations, the position of the optic disc and the macula are completely aligned in the two fundus photos before and after treatment.

[0030] Preferably, in the lesion comparison diagnosis module, the step of marking differentially differentiated lesions with different colors includes:

[0031] By comparing two fundus photos taken before and after treatment with alignment, the areas of difference in structure were extracted;

[0032] Determine the type of lesion to which the discrepancy area belongs;

[0033] Hemorrhagic lesions and microaneurysms are marked in red, exudative lesions are marked in yellow, and pore lesions are marked in white.

[0034] Preferably, in the medical resource matching module, the step of filtering and matching corresponding ophthalmologists according to preset weight conditions includes:

[0035] Establish an information database containing information on ophthalmologists;

[0036] We obtain weighted values ​​for the doctor's geographical location, the relevance of the doctor's professional field, and the number of appointments made on the platform the doctor is on;

[0037] The weighted values ​​are summed and sorted, and the doctor information with the highest overall ranking is locked and output.

[0038] Preferably, in the fundus image stitching module, before extracting key features from multiple local fundus images, the following steps are included:

[0039] Image data is acquired using a 50-megapixel camera and a photosensor.

[0040] The image data is sent to the main control board for processing via a data transmission protocol.

[0041] The stitched panoramic fundus image is then displayed synchronously on the LCD screen.

[0042] Secondly, the present invention provides the following technical solution: a portable, directly displayable intelligent AI diagnostic fundus endoscope method.

[0043] Step S1: Collect the user's voice commands, perform noise reduction processing on the voice commands, and parse them into execution commands for controlling the device;

[0044] Step S2: Monitor the device's motion jitter data in real time and generate reverse motion compensation through calculation to counteract the jitter caused by the handheld device;

[0045] Step S3: Extract key features from multiple local fundus photos, align and fuse the multiple local fundus photos into a complete panoramic fundus photo;

[0046] Step S4: Align the two fundus photos before and after treatment, calculate and extract the differential lesions between the two fundus photos, mark the differential lesions with different colors, and output the diagnostic description.

[0047] Step S5: Based on the diagnosis description, filter and match the corresponding ophthalmologists according to preset weight conditions.

[0048] The present invention has the following beneficial effects:

[0049] 1. In this invention, by setting up a voice anti-interference interaction module and a closed-loop anti-shake control module, the examiner can control the operation of the device directly through voice commands with mixed sound pickup without having to manually adjust the knob at close range. Moreover, the internal micro motor can compensate for hand shake in real time based on sensor data, effectively avoiding the risk of respiratory droplet transmission in traditional close-range examinations, and also ensuring extremely high clarity and stability of fundus images in portable handheld mode.

[0050] 2. In this invention, multiple local photos are fused into a panoramic digital image by a fundus image stitching module, and the lesion comparison and diagnosis module is used to accurately align the photos before and after treatment with the optic disc and macula as fixed anchor points. This solves the problem of narrow field of view and inability to archive traditional fundus endoscopes, and can intuitively and quantitatively mark the differences in lesion evolution by different colors, eliminating the drawbacks of diagnosis that rely on doctors' subjective experience and memory.

[0051] 3. In this invention, after the device outputs a diagnostic description with pictures and text, it automatically combines multiple preset weights such as the doctor's geographical location, professional direction correspondence, and popularity on the registration platform to screen and lock the specialist doctor with the best comprehensive ranking for the patient. This makes the portable device improve the allocation efficiency of ophthalmological medical resources and the patient's medical experience. Attached Figure Description

[0052] Figure 1 This is a system module diagram of a portable, directly-displayed intelligent AI diagnostic fundus endoscope proposed in this invention;

[0053] Figure 2This is a flowchart of a portable, directly displayable intelligent AI diagnostic fundus endoscope method proposed in this invention. Detailed Implementation

[0054] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1

[0056] In a first embodiment of the present invention, the present invention provides a portable, directly displayable intelligent AI diagnostic fundus endoscope, such as... Figure 1 As shown, it includes the following steps: A portable, directly displayable intelligent AI diagnostic fundus endoscope system, comprising:

[0057] The voice anti-interference interaction module collects the user's voice commands, performs noise reduction processing on the voice commands, and parses them into execution commands for controlling the device;

[0058] In the voice anti-interference interaction module, the steps of collecting user voice commands and performing noise reduction processing on the voice commands include:

[0059] It captures bone voiceprint signals transmitted by vocal cord vibration through a built-in vibration sensor;

[0060] Voice signals transmitted through the air are collected using a microphone;

[0061] The bone voiceprint signal is mixed with the speech signal for pickup and data fusion, filtering out environmental noise interference and preserving high-frequency details of the speech.

[0062] In the voice anti-interference interaction module, the steps of parsing it into control device execution instructions include:

[0063] Analyze the energy parameters and zero-crossing rate parameters of the acquired speech signal to determine the start and end points of the speech.

[0064] Based on the start and end points, silent segments are removed to generate a clean signal and match it to generate the corresponding execution instructions.

[0065] Specifically, the voice anti-interference interaction module internally includes a vibration sensor and a microphone. The vibration sensor is located on the side of the device closest to the user's face, used to collect mechanical vibration signals generated by the vibration of the vocal cords transmitted through the skull when the user speaks, and converts the mechanical vibration signals into a first electrical signal, which is defined as a bone voiceprint signal. The microphone is located on the surface of the device's outer casing, used to collect speech sound waves propagating in the air, and converts the speech sound waves into a second electrical signal, which is defined as an airborne speech signal.

[0066] The voice anti-interference interaction module synchronously inputs the bone voiceprint signal and the air speech signal to the hybrid pickup and data fusion unit. The data fusion unit performs high-pass filtering on the air speech signal to extract high-frequency feature components, while preserving the low-frequency stability characteristics of the bone voiceprint signal. The two signals are then linearly weighted and fused according to preset fusion weights to obtain a denoised, clean speech signal. The fusion process satisfies the following formula:

[0067] ;

[0068] in, Indicates time The pure speech signal after fusion; Indicates time The collected bone voiceprint signals; Indicates time The collected airborne speech signal; This represents a high-pass filter function used to filter out low-frequency ambient noise in airborne speech signals; and Let represent the weighting coefficients of the bone voiceprint signal and the air speech signal, respectively, satisfying . .

[0069] The weighting coefficients are dynamically adjusted based on the real-time collected ambient background noise sound pressure level. When the background noise sound pressure level exceeds a preset threshold, the weighting coefficients are increased. The value and decrease accordingly. The value.

[0070] The voice anti-interference interaction module performs voice endpoint detection processing on the clean voice signal to determine the start and end points of the voice command. Specifically, the module divides the clean voice signal into frames according to a preset frame length and frame shift, and calculates the short-time energy and short-time zero-crossing rate for each frame.

[0071] The formula for calculating short-time energy is:

[0072] ;

[0073] in, Indicates the first The short-time energy of a frame speech signal; Represents a discrete speech signal sequence; Indicates the window function; Indicates the length of the window function.

[0074] The formula for calculating the short-time zero crossing rate is:

[0075] ;

[0076] in, Indicates the first Short-time zero-crossing rate of frame speech signals; This represents a sign function; the output is 1 when the input value is greater than or equal to zero, and -1 when the input value is less than zero.

[0077] The system presets short-time energy thresholds and zero-crossing rate thresholds. When the short-time energy and short-time zero-crossing rate of several consecutive frames simultaneously exceed the corresponding thresholds, that time point is determined as the start point of the speech; when the short-time energy and short-time zero-crossing rate of several consecutive frames simultaneously fall below the corresponding thresholds, that time point is determined as the end point of the speech. The system extracts the speech segment located between the start and end points of the speech, removes the silent segments before and after it, and generates a valid speech command signal.

[0078] The voice interference immunity interaction module has a pre-stored command feature acoustic model library. The module extracts the Mel-frequency cepstral coefficient feature vector from the valid voice command signal and calculates the distance between this feature vector and the template feature vector in the command feature acoustic model library. When the calculated Euclidean distance is less than a preset matching threshold, it outputs the control execution command corresponding to the template with the smallest distance, which drives the fundus microscope to complete the corresponding functional operation.

[0079] The closed-loop anti-shake control module monitors the device's motion and vibration data in real time, and generates reverse motion compensation through calculation to counteract the shaking caused by handheld device;

[0080] In the closed-loop anti-shake control module, the real-time monitoring of the device's motion vibration data and the calculation of reverse motion compensation steps include:

[0081] The device acquires real-time displacement and angular velocity data on the pitch, yaw, and roll axes using built-in gyroscopes and accelerometers.

[0082] A filtering algorithm is used to remove random noise from displacement and angular velocity change data.

[0083] Based on the data after noise removal, the direction of jitter change is predicted, the required reverse compensation value is calculated, and the internal micro motor is driven to generate precise reverse motion to counteract the jitter.

[0084] Specifically, the intelligent AI diagnostic fundus endoscope system includes a closed-loop image stabilization control module. This module monitors the device's motion and vibration data in real time when held by hand, and generates reverse motion compensation through internal calculations to counteract the shaking caused by holding the device through physical mechanical displacement.

[0085] The closed-loop anti-shake control module integrates a microelectromechanical system (MEMS) inertial measurement unit (IMU). The IMU includes a three-axis gyroscope and a three-axis accelerometer. The closed-loop anti-shake control module acquires real-time angular velocity changes along the pitch, yaw, and roll axes using the three-axis gyroscope, and simultaneously acquires real-time linear acceleration changes along these three spatial axes using the three-axis accelerometer. These angular velocity and linear acceleration changes together constitute the initial motion jitter data.

[0086] The closed-loop anti-shake control module receives initial motion jitter data and uses a complementary filtering algorithm to fuse displacement and angular velocity change data, eliminating random high-frequency noise and low-frequency drift errors from the sensor sampling process. The calculation formula for the complementary filtering algorithm is as follows:

[0087] ;

[0088] in, Indicates the first The actual attitude angle after filtering for each sampling period; Indicates the first The actual attitude angle for each sampling period; Indicates the first Each sampling period consists of the angular velocity value output by the three-axis gyroscope; This represents the discrete sampling time interval of the system; This represents the observed angle calculated from the linear acceleration output by the triaxial accelerometer within the same period; This represents the preset filtering time constant, whose value range is limited to 1. After complementary filtering, the system outputs smooth 3D attitude data.

[0089] The closed-loop anti-shake control module extracts noise-removed 3D attitude data, calculates the deviation between the preset absolutely stationary target attitude and the actual attitude angle, and predicts the direction of jitter change based on this deviation. The system uses a proportional-integral-derivative (PID) control algorithm to calculate the required reverse compensation value. The formula for calculating the reverse compensation value is as follows:

[0090] ;

[0091] in, Indicates the first The reverse compensation control quantity output in each calculation cycle; Indicates the first The attitude deviation over each cycle is the difference between the set target attitude angle and the actual attitude angle. The difference; Indicates the first Attitude deviation per cycle; This represents the proportional gain coefficient, used to output the base compensation amount based on the current deviation; This represents the integral gain coefficient, used to accumulate historical deviations to eliminate steady-state errors; This represents the differential gain coefficient, used to calculate the rate of change of the deviation, thereby predicting the jitter trend and providing damping.

[0092] The closed-loop image stabilization control module will calculate the reverse compensation value. The signal is converted into a corresponding pulse width modulation (PWM) signal and transmitted to a micromotor inside the fundus mirror. This micromotor consists of three independent brushless DC motors distributed along the pitch, yaw, and roll axes. Upon receiving the PWM signal, the micromotor drives the corresponding mechanical axis according to the signal's duty cycle, generating a precise reverse motion that is opposite in direction and equal in angle to the predicted jitter change. This keeps the optical lens assembly of the fundus mirror spatially fixed relative to the user's eyeball, counteracting image shift caused by hand shake.

[0093] The fundus image stitching module extracts key features from multiple local fundus photos, aligns and merges these photos into a complete panoramic fundus image.

[0094] In the fundus image stitching module, the steps of extracting key features from multiple local fundus images, aligning and fusing these images into a single complete panoramic fundus image include:

[0095] Extract feature points and line features from multiple local fundus images;

[0096] Geometric transformation relationships between multiple local fundus images were established through feature matching, and the homography matrix was calculated.

[0097] Based on the homography matrix, all local fundus images are transformed to the same coordinate system, and edge gradient fusion is performed by weighted averaging to eliminate stitching seams;

[0098] Before extracting key features from multiple local fundus images in the fundus image stitching module, the following steps are included:

[0099] Image data is acquired using a 50-megapixel camera and a photosensor.

[0100] The image data is sent to the main control board for processing via a data transmission protocol.

[0101] The stitched panoramic fundus image is simultaneously displayed on the LCD screen;

[0102] Specifically, the fundus image stitching module is used to extract key features from multiple local fundus photos, align the multiple local fundus photos in space, and perform pixel-level fusion in the overlapping areas to finally generate a complete panoramic fundus photo.

[0103] Before the fundus image stitching module performs key feature extraction, the system uses a 50-megapixel camera and a photosensor to acquire image data. The photosensor receives the light signals reflected from the fundus tissue and converts them into digital image matrix data. After acquiring the image data, the system sends the image data to the main control board for processing via a data transmission protocol.

[0104] The fundus image stitching module reads multiple local fundus images from the main control board. The module constructs a scale space for each local fundus image and extracts feature points and line features from multiple images. The module calculates the gradient magnitude and direction of image pixels, locating local extrema as feature points; simultaneously, it uses an edge detection operator to extract continuous pixel sets of retinal vessel edges as line features. Subsequently, the module generates corresponding feature descriptor vectors for each extracted feature point and line feature.

[0105] After feature extraction, the module establishes geometric transformation relationships between multiple local fundus images through feature matching. The module calculates the Euclidean distance between feature descriptor vectors in different local fundus images, filters feature pairs with distances less than a set threshold, and establishes an initial matching pair set. The module uses a random sampling consensus algorithm to remove mismatched points from the initial matching pair set, retaining the inlier set, and calculates the homography matrix between adjacent local fundus images based on the inliers. The coordinate transformation formula for the homography matrix is:

[0106] ;

[0107] in, This represents the pixel coordinates in the first partial fundus image to be stitched together; This represents the coordinates of the corresponding mapped pixel point in the reference target coordinate system (i.e., the second local fundus image coordinate system); to These represent the nine spatial transformation parameters of the homography matrix.

[0108] Based on the calculated homography matrix, the module performs spatial mapping on all local fundus images, transforming them to the same coordinate system to complete image alignment. After alignment, pixel grayscale differences exist in overlapping areas of the local fundus images. The module uses a weighted average method to perform edge gradient fusion to eliminate stitching seams. The weighted average fusion formula for pixels in the overlapping area is:

[0109] ;

[0110] in, This indicates the coordinates of the fused panoramic fundus image. The actual pixel grayscale value at that location; This indicates the coordinates of the first aligned partial fundus image in the overlapping region. The pixel grayscale value at that location; This indicates that the aligned second partial fundus images are at the same coordinate in the overlapping area. The pixel grayscale value at that location; and These represent the gradient weighting coefficients of the first and second local fundus images at this coordinate system. and The value range is between 0 and 1, and satisfies The magnitude of the weighting coefficient is determined by the current coordinates. The weight coefficient is calculated based on the physical pixel distance from the boundary of the overlapping images. The closer to the boundary, the smaller the corresponding weight coefficient.

[0111] After weighted average fusion calculation, the module stitches all the local fundus images into a continuous pixel matrix, i.e., a complete panoramic fundus image. Finally, the main control board synchronously displays the stitched panoramic fundus image on the LCD screen.

[0112] The lesion comparison and diagnosis module aligns two fundus photos taken before and after treatment, calculates and extracts the lesions that differ between the two fundus photos, marks the lesions with different colors, and outputs a diagnostic description.

[0113] In the lesion comparison and diagnosis module, the steps for aligning two fundus photographs before and after treatment include:

[0114] Identify the optic disc and macular positions in two fundus photographs taken before and after treatment;

[0115] Use the position of the optic disc and the position of the macula as fixed reference anchor points;

[0116] By controlling the image to perform translation, scaling, and rotation operations, the position of the optic disc and the macula in the two fundus photos before and after treatment are completely superimposed and aligned.

[0117] In the lesion comparison diagnosis module, the steps for marking differentially differentiated lesions with different colors include:

[0118] By comparing two fundus photos taken before and after treatment with alignment, the areas of difference in structure were extracted;

[0119] Determine the type of lesion to which the area of ​​difference belongs;

[0120] Hemorrhage lesions and microaneurysms are marked in red, exudative lesions are marked in yellow, and puncture lesions are marked in white.

[0121] Specifically, the lesion comparison diagnosis module is used to spatially align two fundus photos before and after treatment, calculate and extract the structural difference regions in the two fundus photos, identify the lesion type to which the difference regions belong, mark the difference lesions using a preset color table, and output diagnostic description text containing lesion type and coordinate region.

[0122] In the lesion comparison and diagnosis module, the system extracts the first fundus photograph acquired before treatment and the second fundus photograph acquired after treatment from the same patient. The module performs pixel traversal on the first and second fundus photographs to identify the optic disc center and macular center positions in the images. The module also extracts the optic disc center coordinates from the first fundus photograph. Coordinates of the macula center Simultaneously, the coordinates of the optic disc center were extracted from the second fundus photograph. Coordinates of the macula center .

[0123] The module uses the extracted optic disc center and macula center positions as fixed reference anchor points to calculate the spatial transformation parameters between the two images, including scaling, rotation angle, and translation. The formula for calculating the scaling is:

[0124] ;

[0125] in, This parameter represents the scaling ratio of the first fundus image relative to the second fundus image.

[0126] The formula for calculating the rotation angle is:

[0127] ;

[0128] in, This indicates the angle at which the first fundus photograph needs to be rotated; This represents the arctangent function.

[0129] The module calculates the scaling ratio. With rotation angle The first fundus photograph was scaled and rotated, and the horizontal and vertical translations were calculated to ensure that the optic disc and macula positions in the first fundus photograph were completely aligned with those in the second fundus photograph. The coordinate transformation formula is as follows:

[0130] ;

[0131] in, This represents the initial two-dimensional coordinates of the pixels in the first fundus photograph; This indicates the alignment coordinates of the pixel in the second fundus image coordinate system after alignment mapping; and These represent the translation amounts of the first fundus photograph on the horizontal and vertical axes, respectively.

[0132] After alignment, the module compares the aligned first and second fundus images to extract regions of structural change. The module calculates the pixel feature difference matrix between the two images at the same coordinate position.

[0133] ;

[0134] in, Representing coordinates The absolute value matrix of differences at each location; Indicates the coordinates of the second fundus photograph. Pixel feature value at; This indicates the first fundus photograph after alignment in the mapped coordinates. Corresponding to the same coordinates The pixel feature value at that location. The module sets the grayscale difference threshold. .when At that time, the system determines the coordinates. This belongs to the region of difference where the structure has changed.

[0135] The module extracts the complete boundaries of the differential regions and inputs the feature vectors of these regions into a pre-stored lesion classification model to determine the specific lesion type to which the differential region belongs. Subsequently, the module calls a pre-defined color channel mapping table to perform pixel coverage operations on the differential regions: regions identified as hemorrhagic lesions and microaneurysms are rewritten with channel assignments and marked in red; regions identified as exudative lesions are rewritten with channel assignments and marked in yellow; and regions identified as pore lesions are rewritten with channel assignments and marked in white. After color marking is completed, the module calculates the pixel area, lesion type classification result, and coordinate range data for each color-marked region, generates a structured diagnostic description text containing the above parameters, and outputs it to the system's external display terminal.

[0136] The medical resource matching module filters and matches corresponding ophthalmologists based on the diagnosis description and according to preset weight conditions.

[0137] In the medical resource matching module, the steps for filtering and matching corresponding ophthalmologists according to preset weight conditions include:

[0138] Establish an information database containing information on ophthalmologists;

[0139] We obtain weighted values ​​for the doctor's geographical location, the relevance of the doctor's professional field, and the number of appointments made on the platform the doctor is on;

[0140] The weighted values ​​are summed and sorted, and the doctor information with the highest overall ranking is locked and output.

[0141] Specifically, the medical resource matching module is used to filter and match the most suitable ophthalmologist for the patient based on the diagnostic description output by the lesion comparison and diagnosis module according to preset weight conditions, so as to realize the automated and precise allocation of medical resources.

[0142] In the medical resource matching module, the system first establishes an information database containing ophthalmologists in the storage unit of the main control board or a remote server. This information database stores structured attribute feature data of multiple candidate doctors, including but not limited to: the doctor's geographical coordinates, the set of specialization tags for fundus diseases they specialize in treating, and the real-time registration queue number on the doctor's medical platform.

[0143] The module parses the received diagnostic descriptions (such as specific lesion types like bleeding, exudation, microaneurysms, or punctures) and traverses the information database to obtain the weighted evaluation values ​​of each candidate doctor in three core evaluation dimensions: the weighted value of the doctor's geographical location, the weighted value of the doctor's professional direction correspondence, and the weighted value of the number of appointments made on the doctor's platform.

[0144] Specifically, the module calculates the spherical distance between the current device location and the candidate doctor's practice location, and maps it to a geographic location weight value according to a preset distance decay function; the closer the distance, the higher the value. The module extracts lesion keywords from the diagnostic description, calculates the text feature similarity between them and the candidate doctor's professional direction label set, and normalizes them to a professional direction correspondence weight value; the higher the matching degree, the higher the value. The module obtains the current number of appointments for the candidate doctor on the platform in real time through a data interface, and uses an inverse proportional mapping function to convert it into an appointment quantity weight value; the fewer the appointments in the queue (i.e., the shorter the waiting time), the higher the weight value.

[0145] After obtaining the numerical values ​​of the above features, the module uses a multi-criteria linear weighted summation algorithm to sum and sort the weight values. The formula for calculating the comprehensive matching score is:

[0146] ;

[0147] in, Indicates the first The overall matching score of each candidate doctor; and They represent the first The weighting values ​​for each candidate doctor are: geographical location, professional direction relevance, and number of appointments. , and These represent the global importance ratios set by the system for geographical location, professional field, and number of registrations, respectively, and satisfy the boundary conditions. + + =1.

[0148] The module iterates through the database to calculate the overall matching score for all candidate ophthalmologists. Then, the candidate list is sorted in descending order based on the score. The system locks the candidate with the highest overall score (i.e., the one ranked first). The module selects the doctor with the highest value (indicated by the highest value) as the final matched ophthalmologist. Finally, it outputs the doctor's information, including name, hospital, and appointment link, displayed on the LCD screen for patients to view and book appointments with a single click.

[0149] Example 2:

[0150] Existing portable fundus examination devices suffer from problems such as shaky and blurry images during handheld shooting, limited field of view in a single shot preventing panoramic views, difficulty in accurately quantifying and comparing lesions before and after treatment, and low efficiency in matching medical resources. To address these issues, this invention provides a portable, directly displayable, intelligent AI-based diagnostic fundus endoscope method, the structure of which is as follows: Figure 2 As shown. The specific implementation process of this method is as follows:

[0151] Step S1: Collect the user's voice commands, perform noise reduction processing on the voice commands, and parse them into execution commands for controlling the device;

[0152] Step S2: Monitor the device's motion jitter data in real time and generate reverse motion compensation through calculation to counteract the jitter caused by the handheld device;

[0153] Step S3: Extract key features from multiple local fundus photos, align and fuse the multiple local fundus photos into a complete panoramic fundus photo;

[0154] Step S4: Align the two fundus photos before and after treatment, calculate and extract the differential lesions between the two fundus photos, mark the differential lesions with different colors, and output the diagnostic description.

[0155] Step S5: Based on the diagnosis description, filter and match the corresponding ophthalmologists according to preset weight conditions.

[0156] Specifically, in step S1, the user's voice commands are acquired using the microphone built into the portable device, and the audio signal of the voice commands is extracted. An acoustic noise reduction filtering algorithm is applied to filter out ambient background noise in the audio signal, extracting clean voice features. The clean voice features are input into a pre-established speech recognition engine for semantic parsing, generating corresponding low-level control commands, and transmitting these control commands to the main control unit to trigger subsequent operations.

[0157] In step S2, the built-in inertial measurement sensor collects the device's acceleration and angular velocity data in three-dimensional space in real time. Based on the collected acceleration and angular velocity data, the displacement changes of the device's yaw, pitch, and roll angles in the suspended handheld state are calculated. These displacement changes are converted into drive compensation signals for the stepper motor, controlling the optical lens group to produce a reverse displacement to counteract the optical axis shift caused by physiological shaking of the handheld device, ensuring focused acquisition of clear fundus images.

[0158] In step S3, multiple local fundus images taken consecutively at different field of view are read. The vascular bifurcation points and optic disc edge contours in each local fundus image are extracted as key feature points. The relative positional relationship of key feature points between adjacent local fundus images is calculated to determine the spatial transformation relationship. Based on the spatial transformation relationship, pixel-level alignment is performed on the multiple local fundus images to eliminate image seams in overlapping areas, and finally, a smooth panoramic fundus image containing the complete retinal structure is generated.

[0159] In step S4, panoramic fundus images of the same patient before and after treatment are retrieved. The optic disc center and macula center positions in the two fundus images are identified and used as fixed reference anchor points. Based on these anchor points, the images are scaled, rotated, and translated to ensure complete overlap and alignment of the pre- and post-treatment fundus images. The pixel features of the aligned images are compared to extract regions of structural change. These regions are then input into a lesion classification model to determine the lesion type. Hemorrhage lesions and microaneurysms are marked using the red channel, exudative lesions using the yellow channel, and pitted lesions using the white channel. The pixel area and type of each color-marked region are statistically analyzed to generate a structured diagnostic description text.

[0160] In step S5, the lesion type is extracted from the diagnostic description text as a search keyword. The pre-established database of ophthalmology specialists' information is traversed. The geographical distance between the current device location and the doctor's practice location is calculated and converted into a geographical location weight value. The matching degree between the lesion type and the doctor's specialty is calculated and converted into a specialty weight value. The number of people queuing on the doctor's registration platform is obtained and converted into a registration quantity weight value. The above geographical location weight values, specialty weight values, and registration quantity weight values ​​are weighted and summed according to the system-set proportional coefficients to obtain a comprehensive ranking score for each doctor. All doctors are sorted in descending order of comprehensive ranking score, and the information of the doctor with the highest comprehensive ranking is extracted and displayed on the device's screen to complete the matching.

[0161] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A portable, directly displayable intelligent AI-based fundus endoscope system, characterized in that, include: The voice anti-interference interaction module collects the user's voice commands, performs noise reduction processing on the voice commands, and parses them into execution commands for controlling the device; The closed-loop anti-shake control module monitors the device's motion and vibration data in real time, and generates reverse motion compensation through calculation to counteract the shaking caused by handheld device; The fundus image stitching module extracts key features from multiple local fundus photos, aligns and merges these photos into a complete panoramic fundus image. The lesion comparison and diagnosis module aligns two fundus photos taken before and after treatment, calculates and extracts the lesions that differ between the two fundus photos, marks the lesions with different colors, and outputs a diagnostic description. The medical resource matching module filters and matches corresponding ophthalmologists based on the diagnosis description and according to preset weight conditions.

2. The portable intelligent AI diagnostic fundus endoscope system with direct display according to claim 1, characterized in that, In the voice anti-interference interaction module, the step of collecting the user's voice commands and performing noise reduction processing on the voice commands includes: It captures bone voiceprint signals transmitted by vocal cord vibration through a built-in vibration sensor; Voice signals transmitted through the air are collected using a microphone; The bone voiceprint signal is mixed with the speech signal for sound pickup and data fusion, filtering out environmental noise interference and preserving high-frequency details of the speech.

3. The portable intelligent AI diagnostic fundus endoscope system with direct display according to claim 1, characterized in that, In the voice anti-interference interaction module, the step of parsing it into execution instructions for the control device includes: Analyze the energy parameters and zero-crossing rate parameters of the acquired speech signal to determine the start and end points of the speech. Based on the start and end points, silent segments are removed to generate a clean signal and match it to generate the corresponding execution instructions.

4. The portable intelligent AI diagnostic fundus endoscope system with direct display according to claim 1, characterized in that, In the closed-loop anti-shake control module, the step of real-time monitoring of the device's motion jitter data and generating reverse motion compensation through calculation includes: The device acquires real-time displacement and angular velocity data on the pitch, yaw, and roll axes using built-in gyroscopes and accelerometers. A filtering algorithm is used to remove random noise from the displacement and angular velocity change data; Based on the data after noise removal, the direction of jitter change is predicted, the required reverse compensation value is calculated, and the internal micro motor is driven to generate a precise reverse motion to counteract the jitter.

5. A portable intelligent AI diagnostic fundus endoscope system with direct display as described in claim 1, characterized in that, In the fundus image stitching module, the step of extracting key features from multiple local fundus images, aligning the multiple local fundus images, and fusing them into a complete panoramic fundus image includes: Extract feature points and line features from the multiple local fundus images; The geometric transformation relationship between the multiple local fundus images is established by feature matching, and the homography matrix is ​​calculated. Based on the homography matrix, all local fundus images are transformed to the same coordinate system, and edge gradient fusion is performed by weighted averaging to eliminate stitching seams.

6. The portable intelligent AI diagnostic fundus endoscope system with direct display according to claim 1, characterized in that, In the lesion comparison and diagnosis module, the step of aligning the two fundus photographs before and after treatment includes: Identify the optic disc and macular positions in two fundus photographs taken before and after treatment; The position of the optic disc and the position of the macula are used as fixed reference anchor points; By controlling the image to perform translation, scaling, and rotation operations, the position of the optic disc and the macula are completely aligned in the two fundus photos before and after treatment.

7. A portable intelligent AI diagnostic fundus endoscope system with direct display as described in claim 1, characterized in that, In the lesion comparison diagnosis module, the step of marking differentially differentiated lesions with different colors includes: By comparing two fundus photos taken before and after treatment with alignment, the areas of difference in structure were extracted; Determine the type of lesion to which the discrepancy area belongs; Hemorrhagic lesions and microaneurysms are marked in red, exudative lesions are marked in yellow, and pore lesions are marked in white.

8. A portable intelligent AI diagnostic fundus endoscope system with direct display as described in claim 1, characterized in that, In the medical resource matching module, the step of filtering and matching corresponding ophthalmologists according to preset weight conditions includes: Establish an information database containing information on ophthalmologists; We obtain weighted values ​​for the doctor's geographical location, the relevance of the doctor's professional field, and the number of appointments made on the platform the doctor is on; The weighted values ​​are summed and sorted, and the doctor information with the highest overall ranking is locked and output.

9. A portable intelligent AI diagnostic fundus endoscope system with direct display capability according to claim 1, characterized in that, In the fundus image stitching module, before extracting key features from multiple local fundus images, the following steps are included: Image data is acquired using a 50-megapixel camera and a photosensor. The image data is sent to the main control board for processing via a data transmission protocol. The stitched panoramic fundus image is then displayed synchronously on the LCD screen.

10. A portable, directly displayable intelligent AI diagnostic fundus endoscope method, characterized in that, The method for a portable, directly-displaying intelligent AI diagnostic fundus endoscope according to any one of claims 1-9 includes: Step S1: Collect the user's voice commands, perform noise reduction processing on the voice commands, and parse them into execution commands for controlling the device; Step S2: Monitor the device's motion jitter data in real time and generate reverse motion compensation through calculation to counteract the jitter caused by the handheld device; Step S3: Extract key features from multiple local fundus photos, align and fuse the multiple local fundus photos into a complete panoramic fundus photo; Step S4: Align the two fundus photos before and after treatment, calculate and extract the lesions that differ between the two fundus photos, mark the lesions with different colors, and output the diagnostic description. Step S5: Based on the diagnosis description, filter and match the corresponding ophthalmologists according to preset weight conditions.

Images