Image processing method and device based on large model and electronic equipment

Abstract

The invention provides an image processing method and device based on a large model and electronic equipment, and relates to the technical field of image processing, in particular to the technical field of artificial intelligence and large models. According to the specific implementation scheme, a first image of a liquid container is obtained; according to the first image, determining a remaining capacity ratio in the liquid container, and performing three-dimensional reconstruction on the liquid container to obtain three-dimensional scale information of the liquid container; determining description information of the liquid container; the first residual capacity of the liquid container is predicted through the large model according to the first image, the three-dimensional scale information and the description information, and the residual capacity prediction accuracy is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to the field of artificial intelligence and large model technology, specifically to an image processing method, apparatus and electronic device based on a large model. Background Technology

[0002] The volume of fluid in the infusion container is crucial for patients receiving intravenous infusions. When the fluid in the container is depleted, the needle at the bottom of the infusion container must be promptly removed from the patient's body to prevent air from entering the patient's body and harming their health. Therefore, it is necessary to monitor the volume of fluid in the infusion container in real time. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, and electronic device based on a large model.

[0004] According to one aspect of this disclosure, an image processing method based on a large model is provided, comprising:

[0005] Acquire the first image of the liquid container;

[0006] Determine the percentage of remaining capacity in the liquid container based on the first image;

[0007] The liquid container is reconstructed in three dimensions to obtain its three-dimensional scale information.

[0008] Determine the description information of the liquid container;

[0009] The first remaining capacity of the liquid container is predicted using a large model based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information.

[0010] According to another aspect of this disclosure, an image processing apparatus based on a large model is provided, comprising:

[0011] The first acquisition module is used to acquire a first image of the liquid container;

[0012] The second acquisition module is used to determine the percentage of remaining capacity in the liquid container based on the first image;

[0013] The third acquisition module is used to perform three-dimensional reconstruction of the liquid container to obtain the three-dimensional scale information of the liquid container;

[0014] The fourth acquisition module is used to determine the description information of the liquid container;

[0015] The fifth acquisition module is used to predict the first remaining capacity of the liquid container using a large model based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the description information.

[0016] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect embodiment.

[0020] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect embodiment.

[0021] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when processed by a processor, implements the method described in the first aspect embodiment.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0024] Figure 1 This is a schematic diagram of an image processing method based on a large model provided in an embodiment of this disclosure;

[0025] Figure 2 This is a schematic diagram of another image processing method based on a large model provided in this disclosure embodiment;

[0026] Figure 3 This is a schematic diagram of another image processing method based on a large model provided in this disclosure embodiment;

[0027] Figure 4 This is a schematic diagram of another image processing method based on a large model provided in this disclosure embodiment;

[0028] Figure 5 This is a schematic diagram of a system for image processing based on large models provided in the embodiments of this disclosure;

[0029] Figure 6 This is a schematic diagram of an image processing apparatus based on a large model provided in an embodiment of this disclosure;

[0030] Figure 7 A schematic block diagram of an electronic device used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] Image processing is the technology of using computers to analyze images to achieve desired results. It generally refers to digital image processing. A digital image is a large two-dimensional array obtained by taking pictures with equipment such as industrial cameras, video cameras, or scanners. The elements of this array are called pixels, and their values ​​are called gray values. Image processing technology generally includes image compression, enhancement and restoration, matching, description, and recognition.

[0033] Artificial intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. It attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0034] A large model is a machine learning model with a large number of parameters and a complex structure. It can process massive amounts of data and complete various complex tasks, such as natural language processing, computer vision, and speech recognition. Large models are characterized by a large number of parameters, large training data, and large computing resources. They have the ability to solve general tasks, follow human instructions, and perform complex reasoning.

[0035] Figure 1 This is a schematic diagram of an image processing method based on a large model provided in an embodiment of this disclosure. Figure 1 As shown, the method includes:

[0036] S101, Obtain the first image of the liquid container.

[0037] Optionally, the first image can be obtained by taking a picture using the user's terminal device or by taking a picture using a surveillance camera in the current space; the first image includes at least the complete liquid container and the position of the liquid inside the liquid container.

[0038] In this embodiment, an infusion scenario is used as an example. In the infusion scenario, the liquid container is an infusion bag or infusion bottle, and the liquid to be monitored is the liquid in the infusion bag or infusion bottle, in order to determine whether the infusion bottle needs to be replaced or the infusion needs to be ended.

[0039] In some embodiments, the first image may also include information for identifying the location of the liquid container, such as a label indicating the location of the patient in the infusion room, so as to determine the location of the liquid container based on the label and the corresponding infusion patient based on the location.

[0040] S102, determine the percentage of remaining capacity in the liquid container based on the first image.

[0041] It is understandable that as the liquid in the container gradually decreases, the areas with liquid and the areas without liquid will have different imaging effects. Therefore, the first image can be binarized to distinguish between the liquid-containing and liquid-free parts of the container.

[0042] In some embodiments, the first image may be filtered before binarization to improve its clarity.

[0043] Furthermore, boundary extraction can be performed based on the binarized first image to obtain a set of edge lines in the first image. The set of edge lines includes at least the liquid plane in the liquid container, which is the edge line that distinguishes the liquid portion and the liquid-free portion of the liquid container. In some embodiments, the set of edge lines may also include the bottom edge line of the liquid container, and the area between the boundary line corresponding to the liquid plane and the bottom edge line is the region of remaining liquid.

[0044] Optionally, the percentage of remaining capacity in the liquid container can be determined based on the proportion of the remaining liquid area in the total liquid capacity.

[0045] S103, perform three-dimensional reconstruction of the liquid container to obtain the three-dimensional scale information of the liquid container.

[0046] In some embodiments, if the liquid container is a container with scale lines, the scale lines can be identified based on the captured first image, thereby obtaining the position of the scale lines and the three-dimensional coordinates of the scale lines in the liquid container.

[0047] Optionally, in this embodiment, a three-dimensional coordinate system is constructed with the lower left corner of the liquid container as the origin, and the three-dimensional coordinates of the scale line of the liquid container in the three-dimensional coordinate system are obtained to obtain the three-dimensional scale information of the liquid container.

[0048] In some embodiments, if there are no scale lines on the surface of the liquid container, three-dimensional reconstruction can be performed based on the geometry of the liquid container to obtain virtual scale lines of the liquid container, thereby determining three-dimensional scale information based on the position information of the virtual scale lines in three-dimensional coordinates.

[0049] Optionally, a dynamic-fusion algorithm can be used to analyze multiple consecutive images of the liquid container to extract its geometric and positional information. Then, based on the geometric and positional information, a three-dimensional reconstruction of the liquid container can be achieved to obtain three-dimensional scale information.

[0050] S104, Determine the description information of the liquid container.

[0051] In some embodiments, the descriptive information of the liquid container may include, but is not limited to, the length, width, position of the capacity line, and color of the liquid inside the liquid container.

[0052] S105, using a large model, predicts the first remaining capacity of the liquid container based on the first image, the percentage of remaining capacity, three-dimensional scale information, and descriptive information.

[0053] Optionally, a prompt word for the large model can be generated based on the first image, the percentage of remaining capacity, the three-dimensional scale information, and the descriptive information. The prompt word is then input into the large model, which is pre-trained to predict the first remaining capacity of the liquid container. This first remaining capacity is the specific percentage of the remaining capacity of the liquid in the liquid container, so as to determine whether there is a situation where the liquid in the liquid container is almost empty.

[0054] In this embodiment, the remaining capacity percentage in the liquid container is obtained through the first image. Obtaining the remaining capacity percentage directly from the imaging results is more efficient. The liquid container is then reconstructed in three dimensions to obtain the corresponding three-dimensional scale information. Furthermore, the descriptive information of the liquid container is obtained. The first image of the liquid container, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information are all input into a large model. The large model performs inference and prediction to obtain a more accurate prediction of the first remaining capacity of the liquid container. This eliminates the need for real-time monitoring by human resources, improves the safety of infusion in infusion scenarios, and reduces the occurrence of medical accidents.

[0055] Figure 2 This is a schematic diagram of another image processing method based on a large model provided in an embodiment of this disclosure. For example... Figure 2 As shown, the method includes:

[0056] S201, when the liquid container is first identified, the first image sequence is captured.

[0057] When the liquid container is first identified after the infusion begins, a first image sequence of the liquid container is acquired. The first image sequence includes multiple second images that are consecutive at different times, so as to obtain more accurate three-dimensional scale information and descriptive information of the liquid container based on the multiple second images.

[0058] S202, Perform three-dimensional reconstruction of the liquid container based on the first image sequence to obtain the three-dimensional scale information of the liquid container.

[0059] In some embodiments, a detection box recognition can be performed on the second image in the first image sequence to obtain a first detection box of the second image; alternatively, the YOLO (You Only Look Once) target detection algorithm can be used for detection box recognition. The YOLO algorithm can identify and locate multiple objects in an image or video in real time. Therefore, the detection box recognition can be performed on the second image based on the algorithm. The detection target is a liquid container area, and a first detection box of the second image is obtained. The first detection box contains the detected target liquid container.

[0060] Furthermore, a corresponding third image can be obtained from the second image based on the first detection box, and the third image is used to construct the second image sequence. Optionally, the first detection box in the second image can be enlarged to obtain the second detection box; the intersection operation of the second detection boxes can be performed to obtain the third detection box; the area where the third detection box is located can be cropped from the second image as the third image; that is, the intersection operation of all the second detection boxes is performed to form a new box as the third detection box, and the image is cropped from the second image based on the third detection box to obtain the third image corresponding to the third detection box. The second image sequence is constructed based on all the second images.

[0061] Furthermore, the second image sequence is processed based on a dynamic fusion algorithm. The dynamic fusion algorithm can reconstruct non-rigid deformation scenes through real-time data fusion. Therefore, after processing the second image sequence using the dynamic fusion algorithm, the geometric structure and spatial position of the liquid container are obtained. In this embodiment, the spatial position refers to three-dimensional coordinates, and the geometric structure is displayed using a mesh model, which includes a series of polygons and vertex information.

[0062] Based on the geometric structure and spatial location, the three-dimensional scale information of the liquid container is determined; alternatively, the geometric structure and spatial location can be input into a pre-trained model to uniformly divide the geometric structure of the liquid container, thereby generating virtual scale lines of the liquid container and the liquid capacity percentage corresponding to each scale line, thus obtaining the three-dimensional scale information of the liquid container.

[0063] In some embodiments, three-dimensional scale information can be stored, and when identifying the remaining liquid capacity in a liquid container, the three-dimensional scale information can be used as the basis for prediction to improve the accuracy of remaining capacity prediction.

[0064] It is understandable that if the liquid container has scale lines, the scale lines can be identified based on the second image in the first image sequence, thereby determining the three-dimensional scale information of the liquid container.

[0065] S203, determine the percentage of remaining capacity in the liquid container.

[0066] In this embodiment, the remaining capacity percentage is determined based on the image recognition acquired at the current moment. For example, during the first recognition, the remaining capacity percentage is obtained based on the second image in the first image sequence. During real-time recognition during the infusion process, the remaining capacity percentage is obtained based on the first image acquired.

[0067] In some embodiments, before obtaining the remaining capacity percentage, the type of the current liquid container can be identified, including regular infusion bottles or irregular infusion bags; alternatively, a pre-trained classification model can be used to identify the type of the liquid container based on the captured image to determine whether the current liquid container is a regular infusion bottle or an irregular infusion bag.

[0068] In some embodiments, if the current liquid container is a regular infusion bottle, the captured image can be filtered, binarized, and feature extracted sequentially to obtain edge feature points in the image. The remaining capacity line and bottom edge line in the infusion bottle are determined based on the edge feature points. The remaining height of the liquid capacity is determined based on the height between the remaining capacity line and the bottom edge line, thereby estimating the remaining capacity percentage and improving the efficiency of obtaining the remaining capacity percentage.

[0069] In some embodiments, if the current liquid container is an irregular infusion bag, the captured image is sequentially filtered, binarized, and feature extracted to obtain edge feature points in the image. The remaining capacity line and bottom edge line in the liquid container are determined based on the edge feature points. Further, the region formed by the remaining capacity line and the bottom edge line is segmented to obtain at least two local regions. The remaining capacity percentage in the liquid container is obtained for each local region, thereby reducing the error in estimating the remaining capacity percentage for irregular infusion bags.

[0070] Optionally, the capacity percentage of each local area can be obtained, and the remaining capacity percentage can be determined based on the capacity percentage of all local areas. For example, the capacity percentages of all local areas can be added together to obtain a more accurate remaining capacity percentage.

[0071] Optionally, the capacity percentage of each local area can be determined by obtaining the width percentage and area percentage of the local area in the liquid container, and then determining the capacity percentage of the local area based on the width percentage and area percentage.

[0072] For example, the remaining capacity line and the bottom edge line can be divided into multiple line segments, for instance, by identification and segmentation using a pre-trained model, resulting in multiple approximate trapezoidal regions as local regions. The area of ​​each trapezoidal region is calculated, and the ratio of the area of ​​the trapezoidal region to the total area of ​​the liquid container is used as the area percentage of the local region. Furthermore, the ratio of the width of each approximate trapezoidal region to the total width of the liquid container is used as the width percentage of the local region. The area percentage and width percentage of each local region are multiplied together to obtain the capacity percentage of the local region. The capacity percentages of all local regions are added together to obtain the estimated remaining capacity percentage, thus improving the accuracy of the remaining capacity percentage estimation.

[0073] In this embodiment of the disclosure, the method for implementing step S203 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0074] S204, Determine the description information of the liquid container.

[0075] In this embodiment of the disclosure, the method for implementing step S204 can be implemented in any of the various embodiments of this disclosure, and no limitation is made here, nor will it be described in detail.

[0076] S205 uses a large model to predict the first remaining capacity of a liquid container based on the first image, the percentage of remaining capacity, three-dimensional scale information, and descriptive information.

[0077] In this embodiment of the disclosure, the method for implementing step S205 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0078] In this embodiment, during the initial identification of the liquid container, multiple images of the liquid container are captured to obtain a first image sequence. Based on the first image sequence, the liquid container is reconstructed in three dimensions. If the liquid container has scale lines, the scale lines and the corresponding capacity percentages are identified to obtain the three-dimensional scale information of the liquid container. If the liquid container does not have scale lines, virtual scale lines and the corresponding capacity percentages are obtained based on the three-dimensional reconstruction, ensuring accurate acquisition of the three-dimensional scale information of the liquid container. The remaining capacity percentage of the liquid container is estimated based on the captured images. The method for obtaining the remaining capacity percentage is determined according to the different types of liquid containers, making resource utilization more rational. The acquired first image, remaining capacity percentage, three-dimensional scale information, and descriptive information are input into a large model to obtain the first remaining capacity. The large model performs inference analysis based on the original scale lines or the reconstructed virtual scale lines, combined with the descriptive information, to improve the accuracy of the large model's prediction of the first remaining capacity.

[0079] Figure 3 This is a schematic diagram of another image processing method based on a large model provided in an embodiment of this disclosure. For example... Figure 3 As shown, the method includes:

[0080] S301, Obtain the first image of the liquid container.

[0081] In this embodiment of the disclosure, the method for implementing step S301 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0082] S302, determine the percentage of remaining capacity in the liquid container based on the first image.

[0083] In this embodiment of the disclosure, the method for implementing step S302 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0084] S303, perform three-dimensional reconstruction of the liquid container to obtain the three-dimensional scale information of the liquid container.

[0085] In this embodiment of the disclosure, the method for implementing step S303 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0086] S304, Determine the description information of the liquid container.

[0087] In this embodiment of the disclosure, the method for implementing step S304 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0088] S305, based on the first image, remaining capacity percentage, three-dimensional scale information, and description information, generate prompt information for the large model.

[0089] Optionally, the first image, remaining capacity percentage, three-dimensional scale information, and descriptive information can be processed to generate prompt information for the large model, and the large model can be guided to predict the remaining liquid capacity based on the prompt information.

[0090] In some embodiments, prompt information can be generated by combining local regions and their corresponding area and width proportions to improve the prediction accuracy of large models.

[0091] S306, using the large model to understand the first image, remaining capacity percentage, three-dimensional scale information, and descriptive information based on the prompt information, determines the three-dimensional scale reached by the liquid container at the current moment.

[0092] Understandably, the large model can interpret and infer the actual three-dimensional scale of the remaining capacity percentage from the prompt information, identify the current three-dimensional scale reached by the remaining liquid, quantify the remaining liquid volume more accurately, and improve the accuracy of the remaining capacity prediction.

[0093] S307, Based on the three-dimensional scale and three-dimensional scale information reached at the current moment, determine the first remaining capacity of the liquid container at the current moment.

[0094] Optionally, the three-dimensional coordinate information may include a virtual scale and the liquid capacity percentage corresponding to the scale. Therefore, after determining the three-dimensional scale, the liquid capacity percentage corresponding to the three-dimensional scale is determined based on the three-dimensional scale information, and is used as the first remaining capacity at the current moment and output, thereby improving the efficiency and accuracy of obtaining the first remaining capacity.

[0095] In this embodiment, after determining the remaining capacity percentage, three-dimensional scale information, and descriptive information of the liquid container, a prompt message for the large model is generated based on the first image of the liquid container, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information. The large model understands the first image, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information based on the prompt message, determines the three-dimensional scale that the liquid container has reached at the current moment, more accurately quantifies the remaining liquid volume, improves the accuracy of the prediction of the remaining capacity, and intuitively determines the remaining liquid capacity in the liquid container based on the three-dimensional scale, thereby improving the inference efficiency of the large model.

[0096] Figure 4 This is a schematic diagram of another image processing method based on a large model provided in an embodiment of this disclosure. For example... Figure 4 As shown, the method includes:

[0097] S401, when performing the first identification of the liquid container, captures the first image sequence.

[0098] In this embodiment of the disclosure, the method for implementing step S401 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0099] S402, Perform three-dimensional reconstruction of the liquid container based on the first image sequence to obtain the three-dimensional scale information of the liquid container.

[0100] In this embodiment of the disclosure, the method for implementing step S402 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0101] S403, determine the percentage of remaining capacity in the liquid container.

[0102] In this embodiment of the disclosure, the method for implementing step S403 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0103] S404, Determine the description information of the liquid container.

[0104] In this embodiment of the disclosure, the method for implementing step S404 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0105] S405, based on the first image, remaining capacity percentage, three-dimensional scale information, and description information, generate prompt information for the large model.

[0106] In this embodiment of the disclosure, the method for implementing step S405 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0107] S406, using the large model to understand the first image, remaining capacity percentage, three-dimensional scale information, and descriptive information based on the prompt information, determines the three-dimensional scale reached by the liquid container at the current moment.

[0108] In this embodiment of the disclosure, the method for implementing step S406 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0109] S407, based on the three-dimensional scale and three-dimensional scale information reached at the current moment, determine the first remaining capacity of the liquid container at the current moment.

[0110] In some embodiments, after obtaining the first remaining capacity, it is determined whether the first remaining capacity meets the warning conditions, such as determining whether the first remaining capacity is lower than a preset capacity threshold, that is, the first remaining capacity is about to run out.

[0111] In response to the first remaining capacity meeting the warning condition, a prompt message is sent to one or more terminal devices associated with the liquid container to promptly remind the user that the liquid container is running low and to prevent medical accidents. Optionally, the location of the liquid container can be determined, and a prompt message can be sent to at least one of the management terminal associated with the location of the liquid container and the terminal of the user of the liquid container.

[0112] Optionally, the management terminal can be the terminal device corresponding to the personnel responsible for infusion management in the hospital or infusion room, and the user's terminal refers to the terminal device corresponding to the infusion patient. When the hospital monitors the fluid volume, it sends alarm information to the management terminal to remind the management personnel that the fluid in the current fluid container is about to run out. When the patient operates the terminal device to monitor the fluid container volume, alarm information is sent to the user's terminal to remind the patient that the fluid in the current fluid container is about to run out. At the same time, alarm information is also sent to the management terminal so that the infusion bottle can be replaced or the infusion can be stopped in time to avoid medical accidents.

[0113] In some embodiments, before a user starts infusion, a terminal device (e.g., a mobile phone) can scan a QR code corresponding to the user's location to bind the device, thereby binding the location with the user's terminal. When the first remaining capacity meets the warning conditions, a warning message is sent to the user's terminal device.

[0114] In some embodiments, the second remaining capacity predicted in the previous large model can also be determined; the capacity difference between the first remaining capacity and the second remaining capacity can be determined; and the real-time flow rate of the liquid container can be determined based on the capacity difference and the time interval between the two predictions; that is, the real-time flow rate of the liquid in the liquid container during the time interval can be obtained based on the time interval between the two predictions and the capacity difference.

[0115] Optionally, the next prediction time of the large model can be determined based on the real-time flow rate of the liquid container and the first remaining capacity; for example, half of the first remaining capacity can be used as the target capacity, and the ratio of the target capacity to the real-time flow rate can be used as the time interval between the next prediction time and the current time, thereby determining the next prediction time of the large model, making the time planning for liquid container capacity monitoring more reasonable.

[0116] In response to the arrival of the next forecast time, the remaining capacity is re-predicted until the remaining capacity meets the warning conditions and an early warning is issued.

[0117] In some embodiments, after determining the real-time flow rate of the liquid container, a reference flow rate can be determined based on the description information of the liquid container. The reference flow rate can be a flow rate determined based on the characteristics of the medication, such as some liquids being suitable for a slower flow rate input. Furthermore, the flow rate difference between the real-time flow rate and the reference flow rate can be obtained. In response to the flow rate difference being greater than a set value, a flow rate adjustment prompt is sent to one or more terminal devices associated with the liquid container, and the user corresponding to the terminal device adjusts the flow rate to avoid affecting the patient's infusion experience due to a large flow rate difference.

[0118] In some embodiments, when the flow rate difference is greater than a set value, flow rate adjustment information can be sent to the intelligent flow rate controller of the liquid container based on the flow rate difference. The intelligent flow rate controller adjusts the infusion flow rate according to the flow rate adjustment information to improve the patient's comfort during the infusion process.

[0119] In this embodiment, during the initial identification of the liquid container, multiple images of the liquid container are captured to obtain a first image sequence. Based on the first image sequence, a three-dimensional reconstruction of the liquid container is performed to obtain its three-dimensional scale information. The remaining capacity percentage of the liquid container is estimated based on the captured images. After determining the remaining capacity percentage, three-dimensional scale information, and descriptive information, a prompt message for the large model is generated based on the first image, remaining capacity percentage, three-dimensional scale information, and descriptive information. The large model interprets this prompt message to determine the current three-dimensional scale reached by the liquid container, thereby obtaining the first remaining capacity percentage of the liquid container at the current moment. The capacity, based on the original scale lines or reconstructed virtual scale lines, combined with descriptive information and the first image, is used for inference analysis to intuitively determine the remaining capacity of the liquid in the liquid container. This improves the inference efficiency of the large model and the accuracy of the prediction of the first remaining capacity. After determining the first remaining capacity, an alarm can be triggered in a timely manner based on whether the first remaining capacity meets the warning conditions to avoid medical accidents. At the same time, after obtaining the first remaining capacity, the real-time flow rate of the liquid can be determined based on the first remaining capacity. The time node of the next prediction and whether the flow rate needs to be adjusted can be determined based on the real-time flow rate to ensure infusion safety while improving the patient's infusion experience.

[0120] Figure 5 This is a schematic diagram of a system for image processing based on a large model provided in this embodiment of the present disclosure. It includes a camera unit and an alarm unit. The camera unit is used to acquire images of a liquid container at specific time intervals and call the large model to analyze and predict the first remaining capacity of the liquid container to determine whether the first remaining capacity calculated by the large model meets the warning conditions. When the warning conditions are met, the alarm unit performs warning processing.

[0121] Figure 6This is a schematic diagram of an image processing apparatus based on a large model provided in an embodiment of this disclosure. Figure 6 As shown, the large-model-based image processing apparatus 600 includes:

[0122] The first acquisition module 601 is used to acquire a first image of the liquid container;

[0123] The second acquisition module 602 is used to determine the percentage of remaining capacity in the liquid container based on the first image;

[0124] The third acquisition module 603 is used to perform three-dimensional reconstruction of the liquid container to obtain the three-dimensional scale information of the liquid container;

[0125] The fourth acquisition module 604 is used to determine the description information of the liquid container;

[0126] The fifth acquisition module 605 is used to predict the first remaining capacity of the liquid container based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information using a large model.

[0127] In some implementations, the third acquisition module 603 includes:

[0128] When the liquid container is first identified, a first image sequence is captured, wherein the first image sequence includes multiple second images that are consecutive in time;

[0129] The liquid container is reconstructed in three dimensions based on the first image sequence to obtain the three-dimensional scale information of the liquid container.

[0130] In some implementations, the third acquisition module 603 includes:

[0131] Perform bounding box recognition on the second image in the first image sequence to obtain the first bounding box of the second image;

[0132] Based on the first detection box, the corresponding third image is obtained from the second image, and the third image is used to form the second image sequence;

[0133] The second image sequence is processed based on a dynamic fusion algorithm to obtain the geometric structure and spatial location of the liquid container;

[0134] Based on the geometric structure and spatial location, determine the three-dimensional scale information of the liquid container.

[0135] In some implementations, the third acquisition module 603 includes:

[0136] The first detection box in the second image is magnified to obtain the second detection box;

[0137] Perform an intersection operation on the second detection box to obtain the third detection box;

[0138] The region containing the third detection box is cropped from the second image and used as the third image.

[0139] In some implementations, the second acquisition module 602 includes:

[0140] Feature extraction is performed on the first image to determine the edge feature points of the first image;

[0141] Based on the edge feature points, determine the remaining capacity line of the liquid container and the bottom edge line;

[0142] The percentage of remaining capacity is determined based on the remaining capacity line and the bottom edge line.

[0143] In some implementations, the second acquisition module 602 includes:

[0144] The region formed by the remaining capacity line and the bottom edge line is divided to obtain at least two local regions;

[0145] Obtain the capacity percentage of each of the local regions;

[0146] The remaining capacity percentage is determined based on the capacity percentage of all the aforementioned local regions.

[0147] In some implementations, the second acquisition module 602 includes:

[0148] For any given local area, obtain the width percentage of that local area within the liquid container;

[0149] Obtain the area percentage of the local region within the liquid container;

[0150] The capacity ratio of the local area is determined based on the width ratio and the area ratio.

[0151] In some implementations, the fifth acquisition module 605 includes:

[0152] Based on the first image, remaining capacity percentage, 3D scale information, and descriptive information, generate prompts for the large model;

[0153] Based on the prompts, the large model interprets the first image, remaining capacity percentage, three-dimensional scale information, and descriptive information to determine the current three-dimensional scale reached by the liquid container.

[0154] Based on the current three-dimensional scale reading and information, determine the first remaining capacity of the liquid container at the current moment.

[0155] In some implementations, device 600 also includes:

[0156] In response to the first remaining capacity meeting the warning conditions, a notification message is sent to one or more terminal devices associated with the liquid container.

[0157] In some implementations, device 600 further includes at least one of the following operations:

[0158] The location of the liquid container is determined, and a notification message is sent to at least one of the management terminal associated with the location of the liquid container and the terminal of the user of the liquid container.

[0159] In some implementations, device 600 also includes:

[0160] Determine the second remaining capacity predicted in the previous large model;

[0161] Determine the capacity difference between the first remaining capacity and the second remaining capacity;

[0162] The real-time flow rate of the liquid container is determined based on the capacity difference and the time interval between the two predictions.

[0163] The next prediction time for the large model is determined based on the real-time flow rate and the first remaining capacity of the liquid container.

[0164] In response to the arrival of the next forecast time, the remaining capacity is re-forecasted.

[0165] In some implementations, device 600 also includes:

[0166] Determine the reference flow rate of the liquid container based on its description information;

[0167] Obtain the flow rate difference between real-time flow rate and reference flow rate;

[0168] In response to a flow rate difference exceeding a set value, a flow rate adjustment prompt is sent to one or more terminal devices associated with the liquid container.

[0169] In some implementations, device 600 also includes:

[0170] In response to a flow rate difference exceeding a set value, flow rate adjustment information is sent to the intelligent flow rate controller of the liquid container based on the flow rate difference.

[0171] In this embodiment, during the initial identification of the liquid container, multiple images of the liquid container are captured to obtain a first image sequence. Based on the first image sequence, a three-dimensional reconstruction of the liquid container is performed to obtain its three-dimensional scale information. The remaining capacity percentage of the liquid container is estimated based on the captured images. After determining the remaining capacity percentage, three-dimensional scale information, and descriptive information, a prompt message for the large model is generated based on the first image, remaining capacity percentage, three-dimensional scale information, and descriptive information. The large model interprets this prompt message to determine the current three-dimensional scale reached by the liquid container, thereby obtaining the first remaining capacity percentage of the liquid container at the current moment. The remaining capacity, based on the original scale lines or reconstructed virtual scale lines, combined with descriptive information and the first image, is used for inference analysis to intuitively determine the remaining capacity of the liquid in the liquid container. This improves the inference efficiency of the large model and the accuracy of the prediction of the first remaining capacity. After determining the first remaining capacity, an alarm can be triggered in a timely manner based on whether the first remaining capacity meets the warning conditions to avoid medical accidents. At the same time, after obtaining the first remaining capacity, the real-time flow rate of the liquid can also be determined based on the first remaining capacity. The time node of the next prediction and whether the flow rate needs to be adjusted can be determined based on the real-time flow rate to ensure infusion safety while improving the patient's infusion experience.

[0172] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0173] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0174] Figure 7 A schematic block diagram of an electronic device for implementing embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0175] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0176] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as large-model-based image processing methods. For example, in some embodiments, the large-model-based image processing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the large-model-based image processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform large-model-based image processing methods by any other suitable means (e.g., by means of firmware).

[0178] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0183] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0184] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0185] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method based on a large model, comprising: Acquire the first image of the liquid container; Determine the percentage of remaining capacity in the liquid container based on the first image; The liquid container is reconstructed in three dimensions to obtain its three-dimensional scale information. Determine the description information of the liquid container; The first remaining capacity of the liquid container is predicted using a large model based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information.

2. The method according to claim 1, wherein, The step of performing three-dimensional reconstruction of the liquid container to obtain the three-dimensional scale information of the liquid container includes: When the liquid container is first identified, a first image sequence is captured, wherein the first image sequence includes a plurality of second images that are consecutive in time; The liquid container is reconstructed in three dimensions based on the first image sequence to obtain the three-dimensional scale information of the liquid container.

3. The method according to claim 2, wherein, The step of performing three-dimensional reconstruction of the liquid container based on the first image sequence to obtain the three-dimensional scale information of the liquid container includes: Perform bounding box recognition on the second image in the first image sequence to obtain the first bounding box of the second image; Based on the first detection box, a corresponding third image is obtained from the second image, and the third image is used to form a second image sequence; The second image sequence is processed based on a dynamic fusion algorithm to obtain the geometric structure and spatial location of the liquid container; Based on the geometric structure and spatial location, the three-dimensional scale information of the liquid container is determined.

4. The method according to claim 3, wherein, The step of obtaining the corresponding third image from the second image based on the first detection box includes: The first detection box in the second image is magnified to obtain the second detection box; Perform an intersection operation on the second detection box to obtain the third detection box; The area containing the third detection box is cropped from the second image and used as the third image.

5. The method according to claim 1, wherein, Determining the percentage of remaining capacity in the liquid container based on the first image includes: Feature extraction is performed on the first image to determine the edge feature points of the first image; Based on the edge feature points, determine the remaining capacity line of the liquid container and the bottom edge line; The percentage of remaining capacity is determined based on the remaining capacity line and the bottom edge line.

6. The method according to claim 5, wherein, Determining the remaining capacity percentage based on the remaining capacity line and the bottom edge line includes: The region formed by the remaining capacity line and the bottom edge line is divided to obtain at least two local regions; Obtain the capacity percentage of each of the local regions; The remaining capacity percentage is determined based on the capacity percentage of all the aforementioned local regions.

7. The method according to claim 6, wherein, The step of obtaining the capacity percentage of each local region includes: For any given local area, obtain the width percentage of that local area within the liquid container; Obtain the area percentage of the local region within the liquid container; The capacity ratio of the local area is determined based on the width ratio and the area ratio.

8. The method according to claim 1, wherein, The step of predicting the first remaining capacity of the liquid container using a large model based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information includes: Based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the description information, generate the prompt information for the large model; The large model interprets the first image, the remaining capacity percentage, the three-dimensional scale information, and the descriptive information based on the prompt information, and determines the three-dimensional scale that the liquid container has reached at the current moment. Based on the three-dimensional scale reached at the current moment and the three-dimensional scale information, the first remaining capacity of the liquid container at the current moment is determined.

9. The method according to any one of claims 1-8, wherein, After predicting the first remaining capacity of the liquid container, the method further includes: In response to the first remaining capacity meeting the warning conditions, a prompt message is sent to one or more terminal devices associated with the liquid container.

10. The method according to claim 9, wherein, Sending a notification message to one or more terminal devices associated with the liquid container further includes at least one of the following operations: The location of the liquid container is determined, and the prompt message is sent to at least one of the management terminal associated with the location of the liquid container and the terminal of the user of the liquid container.

11. The method according to any one of claims 1-8, wherein, After predicting the first remaining capacity of the liquid container, the method further includes: Determine the second remaining capacity predicted in the previous large model; Determine the capacity difference between the first remaining capacity and the second remaining capacity; The real-time flow rate of the liquid container is determined based on the capacity difference and the time interval between the two predictions. The next prediction time for the large model is determined based on the real-time flow rate of the liquid container and the first remaining capacity. In response to the arrival of the next prediction time, the remaining capacity is predicted again.

12. The method according to claim 11, wherein, After determining the real-time flow rate of the liquid container, the method further includes: Based on the description information of the liquid container, determine the reference flow rate of the liquid container; Obtain the flow rate difference between the real-time flow rate and the reference flow rate; In response to the flow rate difference being greater than a set value, a flow rate adjustment prompt is sent to one or more terminal devices associated with the liquid container.

13. The method according to claim 12, wherein, The method further includes: In response to the flow rate difference being greater than a set value, flow rate adjustment information is sent to the intelligent flow rate controller of the liquid container based on the flow rate difference.

14. An image processing apparatus based on a large model, comprising: The first acquisition module is used to acquire a first image of the liquid container; The second acquisition module is used to determine the percentage of remaining capacity in the liquid container based on the first image; The third acquisition module is used to perform three-dimensional reconstruction of the liquid container to obtain the three-dimensional scale information of the liquid container; The fourth acquisition module is used to determine the description information of the liquid container; The fifth acquisition module is used to predict the first remaining capacity of the liquid container using a large model based on the first image, the remaining capacity percentage, the three-dimensional scale information, and the description information.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-13.

Images