Image segmentation method and device based on large language model, and electronic equipment

By acquiring image and text features, determining the center point and ray endpoint, and performing adaptive image segmentation, the problem of low segmentation accuracy and insufficient shape information retention in image segmentation is solved, and higher segmentation accuracy and shape information are achieved reserve.

CN118941586AActive Publication Date: 2024-11-12INST OF AUTOMATION CHINESE ACAD OF SCI

Patent Information

Application Number
CN202410845624.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-12
Estimated Expiration
2044-06-27

AI Technical Summary

Technical Problem

Existing large language models are difficult to achieve accurate text and image format uniformity in image segmentation tasks, resulting in low segmentation accuracy and failure to effectively retain the outline shape information of the image.

Method used

通过获取待分割图像和问题文本,分别进行特征提取处理以获得视觉嵌入向量和文本嵌入向量,确定中心点和射线以指示轮廓,采用自适应的射线终点选择方法进行图像分割,确保分割精度。

Benefits of technology

It improves the segmentation accuracy of image segmentation, can accurately ensure the contour shape information of the image to be segmented, and enhances the fine-grained perception and positioning capabilities of large language models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941586A_ABST
    Figure CN118941586A_ABST
Patent Text Reader

Abstract

The invention provides an image segmentation method and device based on a large language model, and electronic equipment, and the method comprises the steps: obtaining a to-be-segmented image and a question text which is used for indicating a to-be-segmented target in the to-be-segmented image; respectively carrying out feature extraction processing on the to-be-segmented image and the problem text to obtain a visual embedding vector and a text embedding vector; the central point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, a preset number of rays with the central point as a starting point are determined, and the end points of the rays are used for indicating the contour of the target to be segmented; and performing segmentation processing on the to-be-segmented image based on the preset number of rays to obtain the to-be-segmented target. The contour of the to-be-segmented target is adaptively determined through the central point and the preset number of rays with the central point as the starting point, contour shape information of the to-be-segmented image can be accurately reserved, and the segmentation precision during image segmentation is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to an image segmentation method, device, and electronic equipment based on a large language model. Background Art

[0002] Large Vision Language Models (LVLMs) have demonstrated impressive language processing capabilities, effectively achieving alignment between image and text tasks. While tasks in the natural language domain have been unified through techniques such as hint engineering or instruction tuning, a comprehensive unified representation is still lacking in vision tasks. This gap is particularly evident in object detection and image segmentation, the latter of which poses a significant challenge for unifying vision tasks. Some methods have managed to convert object detection frameworks into text representations, improving the ability of LVLMs to locate objects. However, these methods do not excel in the same text with images. Therefore, effectively converting image segmentation into text representations is crucial to achieve a unified approach for vision tasks.

[0003] In the field of image segmentation, the challenges associated with dealing with image segmentation masks are complex and diverse. Different from traditional methods, some studies such as Lisa, GLaMM, and PixelLM attempt to enable language models to learn mask tokens and generate segmentation predictions through specific decoder structures. However, these methods do not achieve true unification of text and image formats, add additional burdens to large language models (LLMs), and fail to maintain a unified structure with large visual language models. Therefore, the best solution lies in fundamentally enhancing the fine-grained perception and localization capabilities of LVLMs rather than relying on external help. There are many challenges in the process of unified representation methods for image segmentation: first, a more optimized and simplified visual representation method is needed; second, a unified visual-text format needs to be constructed; in addition, spatial shape information needs to be accurately preserved. Summary of the invention

[0004] The present invention provides an image segmentation method, device, and electronic device based on a large language model, which are used to solve the defect of low accuracy when segmenting images based on a large language model in the prior art, and achieve accurate retention of contour shape information of the image to be segmented, effectively improving the segmentation accuracy during image segmentation.

[0005] The present invention provides an image segmentation method based on a large language model, comprising the following steps.

[0006] Acquire an image to be segmented and a question text, wherein the question text is used to indicate an object to be segmented in the image to be segmented; Performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; Determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point, wherein the end points of the rays are used to indicate the contour of the target to be segmented; The image to be segmented is segmented based on the preset number of rays to obtain the object to be segmented.

[0007] According to an image segmentation method based on a large language model provided by the present invention, the central point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, and a preset number of rays starting from the central point are determined, including: Determine the center point of the object to be segmented based on the visual embedding vector and the text embedding vector; Taking the center point as the starting point, the rays are sequentially determined in the following manner: the direction of the first ray is set, and for each subsequent ray, the angle range between the ray and the previous ray is set; The rays to be determined that meet the preset conditions within the angle range are taken as the rays, wherein the rays to be determined are a plurality of rays within the angle range that start from the center point and end at the contour point of the target to be segmented.

[0008] According to an image segmentation method based on a large language model provided by the present invention, taking the ray to be determined that meets a preset condition within the angle range as the ray comprises: Calculating the length of the ray to be determined with the maximum angle and the length of the ray to be determined with the maximum length within the angle range respectively; determining the difference between the length of the ray to be determined with the maximum angle and the length of the ray to be determined with the maximum length; When the difference is less than or equal to a preset threshold, taking the ray to be determined with the maximum angle as the ray; When the difference is greater than the preset threshold, the ray to be determined with the maximum length is used as the ray.

[0009] According to an image segmentation method based on a large language model provided by the present invention, segmenting the image to be segmented based on the preset number of rays to obtain the target to be segmented includes: Determine the end points corresponding to the preset number of rays respectively, and determine the coordinates of the end points in the image to be segmented; and segment the image to be segmented based on the coordinates of the end points to obtain the target to be segmented.

[0010] According to the image segmentation method based on a large language model provided by the present invention, before performing feature extraction processing on the image to be segmented and the question text respectively, the method further includes: Performing image interpolation processing on the image to be segmented to obtain an image to be segmented with a preset resolution.

[0011] According to an image segmentation method based on a large language model provided by the present invention, the feature extraction process is performed on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector, including: Performing visual coding processing on the image to be segmented with the preset resolution to obtain image coding features; Performing projection processing on the image encoding feature to obtain the visual embedding vector; Perform text feature extraction processing on the question text to obtain the text embedding vector.

[0012] The present invention also provides an image segmentation device based on a large language model, comprising the following modules: An acquisition module, used for acquiring an image to be segmented and a question text, wherein the question text is used for indicating an object to be segmented in the image to be segmented; A feature extraction module is used to perform feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; A determination module, configured to determine a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point, wherein the end points of the rays are used to indicate the contour of the target to be segmented; The image segmentation module is used to perform segmentation processing on the image to be segmented based on the preset number of rays to obtain the object to be segmented.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image segmentation method based on a large language model as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described image segmentation methods based on a large language model.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image segmentation method based on any one of the large language models described above is implemented.

[0016] The image segmentation method based on a large language model provided by the present invention obtains an image to be segmented and a question text, and then performs feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector, and then determines the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determines a preset number of rays starting from the center point, and indicates the contour of the target to be segmented with the end point of the ray, thereby performing image segmentation on the image to be segmented, and can accurately retain the contour shape information of the image to be segmented, effectively improving the segmentation accuracy during image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the image segmentation method based on a large language model provided by the present invention.

[0019] Figure 2 It is one of the overall structural schematic diagrams of the image segmentation method based on the large language model provided by the present invention.

[0020] Figure 3 This is the second overall structural diagram of the image segmentation method based on the large language model provided by the present invention.

[0021] Figure 4 It is a schematic diagram for comparing segmentation results of the image segmentation method based on a large language model provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of an image segmentation device based on a large language model provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Figure 1is a flow chart of the image segmentation method based on a large language model provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104: Step 101: Obtain the image to be segmented and the question text.

[0026] The question text is used to indicate the target to be segmented in the image to be segmented. The image to be segmented includes at least one target to be segmented, and the target to be segmented can be a specific entity such as an animal or a plant.

[0027] Specifically, see Figure 2 , Figure 2 This is one of the overall structural diagrams of the image segmentation method based on a large language model provided by the present invention. Figure 2 As shown, the input image to be segmented X v is an image containing a gray cat lying on a table, the question text X ins for "Where is the cat located in this picture?"

[0028] In other embodiments, see Figure 3 , Figure 3 This is the second overall structural diagram of the image segmentation method based on the large language model provided by the present invention. Figure 3 As shown, the input image to be segmented X v For an image containing a koala lying on a branch, the question text X ins for "Where is the koala located in this picture?"

[0029] Step 102: perform feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector.

[0030] In some embodiments, after obtaining the image to be segmented and the question text, feature extraction processing needs to be performed on the image to be segmented and the question text respectively, so as to facilitate the large language model to further predict the answer.

[0031] Before step 102, the method further includes the following steps: performing image interpolation processing on the image to be segmented to obtain an image to be segmented with a preset resolution.

[0032] Specifically, the large language model SegLLM used in the present invention aims to unify pixel-level visual tasks in text form. The model converts the image segmentation task into a sequence-to-sequence format that follows instructions, thus achieving unified processing of the task. Therefore, for a given image to be segmented X v, it is necessary to first use the CLIP-L / 14@224 model to interpolate it, convert its resolution to a preset resolution, and obtain an image to be segmented with a preset resolution, so as to reduce the computational cost in subsequent model processing. For example, the interpolation algorithm can use nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., and the preset resolution can be 448*448.

[0033] In some embodiments, step 102 can be implemented by: performing visual encoding processing on the image to be segmented with a preset resolution to obtain image encoding features; performing projection processing on the image encoding features to obtain the visual embedding vector; performing text feature extraction processing on the question text to obtain the text embedding vector.

[0034] Continue to see Figure 2 ,exist Figure 2 In the example, for the image to be segmented X v , first perform the above-mentioned interpolation processing on it, then input the image to be segmented with a preset resolution into the visual encoding layer (Visual Encoder) for visual encoding processing, obtain image encoding features, and then input the visual encoding features into the projection layer (Projection Layer) for projection processing to obtain a visual embedding vector. For the question text, the text feature extraction processing is directly performed through the Tokenizer to obtain a text embedding vector. It should be noted that the above-mentioned visual embedding vector and text embedding vector are both features of the text dimension, so the above-mentioned visual embedding vector and text embedding vector can be directly input into the large language model for processing.

[0035] In some other embodiments, see Figure 3 ,exist Figure 3 In the example, for the image to be segmented X v , first perform the above interpolation processing on it, then input the image to be segmented with a preset resolution into the visual encoding layer (Visual Encoder) for visual encoding processing to obtain image encoding features, and then input the visual encoding features into the projection layer (Projection Layer) for projection processing to obtain a visual embedding vector. For the question text, the text feature extraction processing is directly performed through the Tokenizer to obtain a text embedding vector, and then the above visual embedding vector and text embedding vector are input into the large language model for processing.

[0036] Step 103: determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point.

[0037] The end point of the ray is used to indicate the contour of the target to be segmented, that is, the end point of the ray is a point on the contour of the target to be segmented.

[0038] In some embodiments, step 103 can be implemented in the following manner: determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector; take the center point as the starting point, and determine the rays in the following order: set the direction of the first ray, and for each subsequent ray, set the angle range between the ray and the previous ray; take the ray to be determined that meets the preset conditions within the angle range as the ray, wherein the ray to be determined is a plurality of rays within the angle range that take the center point as the starting point and end at the contour point of the target to be segmented.

[0039] Specifically, see Figure 3 Below, for the image to be segmented (Image Input), the polar coordinates (PolarPoints) are calculated, that is, the end point of the ray is calculated. First, the center point of the target to be segmented needs to be determined based on the visual embedding vector and the text embedding vector. Here, the center point can be the center of the coordinate box corresponding to the target to be segmented, or the center of mass of the target to be segmented. Then, starting from the center point, a ray is emitted in the direction of the contour of the target to be segmented. The direction of the first ray is 0 degrees, and then the subsequent rays are determined in a clockwise or counterclockwise order. Here, taking the counterclockwise direction as an example, the number of rays is set to 36, and the angle range between the two interval rays is δ (δ=θ±σ, θ=10°, σ=5°). For the second ray, within this angle range, all rays composed of points from the center point to the contour are regarded as rays to be determined, and then the rays to be determined that meet the preset conditions are used as the second rays.

[0040] In some embodiments, taking the ray to be determined that meets the preset conditions within the angle range as the ray includes: respectively calculating the length of the ray to be determined with the maximum angle within the angle range, and the length of the ray to be determined with the maximum length; determining the difference between the length of the ray to be determined with the maximum angle and the length of the ray to be determined with the maximum length; when the difference is less than or equal to a preset threshold, taking the ray to be determined with the maximum angle as the ray; when the difference is greater than the preset threshold, taking the ray to be determined with the maximum length as the ray.

[0041] Specifically, see Figure 3 Next, continuing with the second ray as an example, the Vistall algorithm can be used to calculate and sort the angles between the undetermined ray and the first ray corresponding to all contour points within the above angle range, and then determine the end point corresponding to the undetermined ray with the largest angle, denoted as p c ; At the same time, calculate the lengths of all the rays to be determined within the above angle range and sort them, and determine the end point corresponding to the ray with the largest length to be determined, recorded as p fThen, we locate p by the coordinate system c and p f At the corresponding position in the segmented image, and calculate p c Distance to the center point d c and p f The distance to the center point df. Point p c It is used to represent the local details of the target to be segmented within a given angle range, and p f Represents the global structure of the target to be segmented. Then set a threshold τ and calculate d c and d f If the difference is less than or equal to τ, then select p c As the polar coordinate point d i , that is, the ray with the maximum angle to be determined is used as the second ray to retain the local details of the target to be segmented; on the contrary, if d f and d c If the difference between them is greater than τ, then choose p f As the polar coordinate point d i , that is, the ray with the maximum length to be determined is used as the second ray to retain the global contour of the target to be segmented. The specific calculation method is shown in the following formula.

[0042]

[0043] Similarly, the third, fourth, ..., and thirty-sixth rays are all determined in the above manner until the pole coordinates corresponding to all thirty-six rays (ie, the coordinates of the end points of the rays) are determined.

[0044] In the embodiment of the present invention, by adaptively adjusting the position of each pole, the selection of each pole is more flexible when facing an image with disorder and complexity, so that local details can be more effectively highlighted, thereby greatly improving the accuracy of the segmentation result of the target to be segmented.

[0045] Step 104: Segment the image to be segmented based on the preset number of rays to obtain the object to be segmented.

[0046] In some embodiments, step 104 can be specifically implemented in the following manner: determining the end points corresponding to the preset number of rays, and determining the coordinates of the end points in the image to be segmented; segmenting the image to be segmented based on the coordinates of the end points to obtain the target to be segmented.

[0047] Specifically, see Figure 2 After obtaining a preset number of rays through the large language model, the end points corresponding to the preset number of rays are determined, and the coordinates of the end points in the image to be segmented are determined, for example Figure 2The coordinates of all endpoints shown in [<bin_742><bin_439><bin_737><bin_449> …<bin_737><bin_426> ]",in"<bin_742><bin_439> " represents the center point of the target to be segmented,"<bin_737> " indicates the distance from the first end point to the center point,"<bin_449> " indicates the distance from the second end point to the center point,"<bin_737> " indicates the distance from the second last end point to the center point,"<bin_426> ” indicates the distance from the last end point to the center point.

[0048] Another example Figure 3 The coordinates of all the end points shown in are “[742, 439, 737, 449…, 737, 426]”, where “742, 439” represents the center point of the target to be segmented, “737” represents the distance from the first end point to the center point, “449” represents the distance from the second end point to the center point, “737” represents the distance from the second to last end point to the center point, and “426” represents the distance from the last end point to the center point.

[0049] It should be noted that the end point, pole point and polar coordinate point in the present invention all have the same meaning.

[0050] Then, the image to be segmented is segmented based on the coordinates of the end point to obtain the target to be segmented, that is, according to the coordinates of the poles arranged in sequence, all the poles are connected in a preset clockwise or counterclockwise order to obtain the outline of the target to be segmented, and then the image to be segmented is segmented according to the outline to obtain the target to be segmented.

[0051] In some embodiments, see Figure 4 , Figure 4 is a schematic diagram comparing the segmentation results of the image segmentation method based on a large language model provided by the present invention, Figure 4 The left side shows the image segmentation result based on VistaLLM's large language model. It can be seen that the head of the person in the image is not segmented, and the segmentation effect is poor. Figure 4 The middle one is the result of image segmentation based on PolarMask. Although the segmentation is performed by selecting a preset number of poles on the contour of the target to be segmented, the angle between the rays corresponding to each pole is fixed, and the method of adaptively selecting poles of the present invention is not adopted. Therefore, the segmentation effect on some local details is not ideal. For example, some details of the face and hands of the person to be segmented are not completely segmented. Figure 4On the right is the effect of segmentation based on the SegLLM large language model of this application. It can be seen that the entire person to be segmented is segmented, and even some details of the hands and face are completely segmented. Therefore, it can be seen that the segmentation accuracy of the present invention is greatly improved.

[0052] The image segmentation device based on a large language model provided by the present invention is described below. The image segmentation device based on a large language model described below and the image segmentation method based on a large language model described above can be referenced to each other.

[0053] See also Figure 5 , Figure 5 is a schematic diagram of the structure of the image segmentation device based on the large language model provided by the present invention. Figure 5 As shown, the image segmentation device 500 based on the large language model provided by the present invention includes an acquisition module 501, a feature extraction module 502, a determination module 503 and an image segmentation module 504. The acquisition module 501 is used to acquire the image to be segmented and the question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; the feature extraction module 502 is used to perform feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; the determination module 503 is used to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; the image segmentation module 504 is used to perform segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0054] In some embodiments, the determination module 503 is also used to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector; taking the center point as the starting point, determining the rays in sequence in the following manner: setting the direction of the first ray, and for each subsequent ray, setting the angle range between the ray and the previous ray; taking the ray to be determined that meets the preset conditions within the angle range as the ray, wherein the ray to be determined is a plurality of rays within the angle range that take the center point as the starting point and end at the contour point of the target to be segmented.

[0055] In some embodiments, the determination module 503 is also used to respectively calculate the length of the ray to be determined at the maximum angle within the angle range, and the length of the ray to be determined at the maximum length; determine the difference between the length of the ray to be determined at the maximum angle and the length of the ray to be determined at the maximum length; when the difference is less than or equal to a preset threshold, the ray to be determined at the maximum angle is used as the ray; when the difference is greater than the preset threshold, the ray to be determined with the maximum length is used as the ray.

[0056] In some embodiments, the image segmentation module 504 is further used to determine the end points corresponding to the preset number of rays, and determine the coordinates of the end points in the image to be segmented; and segment the image to be segmented based on the coordinates of the end points to obtain the target to be segmented.

[0057] In some embodiments, the feature extraction module 502 is further configured to perform image interpolation processing on the image to be segmented to obtain an image to be segmented with a preset resolution.

[0058] In some embodiments, the feature extraction module 502 is also used to perform visual encoding processing on the image to be segmented with a preset resolution to obtain image encoding features; perform projection processing on the image encoding features to obtain the visual embedding vector; and perform text feature extraction processing on the question text to obtain the text embedding vector.

[0059] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute an image segmentation method based on a large language model, the method comprising: obtaining an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays starting from the center point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0060] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0061] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image segmentation method based on a large language model provided by the above methods, and the method includes: obtaining an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays with the center point as the starting point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0062] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image segmentation method based on a large language model provided by the above-mentioned methods, the method comprising: obtaining an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays starting from the center point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0063] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image segmentation method based on a large language model, It is characterized in that include: Acquire an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; perform feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; Determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point, wherein the end points of the rays are used to indicate the contour of the target to be segmented; The image to be segmented is segmented based on the preset number of rays to obtain the object to be segmented.

2. The image segmentation method based on a large language model according to claim 1, It is characterized in that The determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays starting from the center point, includes: Determine the center point of the object to be segmented based on the visual embedding vector and the text embedding vector; Taking the center point as the starting point, the rays are sequentially determined in the following manner: the direction of the first ray is set, and for each subsequent ray, the angle range between the ray and the previous ray is set; The rays to be determined that meet the preset conditions within the angle range are taken as the rays, wherein the rays to be determined are a plurality of rays within the angle range that start from the center point and end at the contour point of the target to be segmented.

3. The image segmentation method based on a large language model according to claim 2, It is characterized in that The taking the to-be-determined ray that satisfies a preset condition within the angle range as the ray comprises: Calculating the length of the ray to be determined with the maximum angle and the length of the ray to be determined with the maximum length within the angle range respectively; determining the difference between the length of the ray to be determined with the maximum angle and the length of the ray to be determined with the maximum length; When the difference is less than or equal to a preset threshold, taking the ray to be determined with the maximum angle as the ray; When the difference is greater than the preset threshold, the ray to be determined with the maximum length is used as the ray.

4. The image segmentation method based on a large language model according to claim 1, It is characterized in that The step of performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented includes: Determine the end points corresponding to the preset number of rays respectively, and determine the coordinates of the end points in the image to be segmented; The image to be segmented is segmented based on the coordinates of the end point to obtain the object to be segmented.

5. The image segmentation method based on a large language model according to claim 1, It is characterized in that Before performing feature extraction processing on the image to be segmented and the question text respectively, the method further includes: Performing image interpolation processing on the image to be segmented to obtain an image to be segmented with a preset resolution.

6. The image segmentation method based on a large language model according to claim 5, It is characterized in that The step of performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector comprises: performing visual coding processing on the image to be segmented of the preset resolution to obtain image coding features; Performing projection processing on the image encoding feature to obtain the visual embedding vector; Perform text feature extraction processing on the question text to obtain the text embedding vector.

7. An image segmentation device based on a large language model, It is characterized in that include: An acquisition module, used for acquiring an image to be segmented and a question text, wherein the question text is used for indicating an object to be segmented in the image to be segmented; A feature extraction module is used to perform feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; A determination module, configured to determine a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point, wherein the end points of the rays are used to indicate the contour of the target to be segmented; The image segmentation module is used to perform segmentation processing on the image to be segmented based on the preset number of rays to obtain the object to be segmented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the image segmentation method based on the large language model as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the image segmentation method based on a large language model as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the image segmentation method based on a large language model as claimed in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Tissue pathology image cell nucleus segmentation method and system based on polar coordinate representation

    CN112330701A

  • Image segmentation method and system based on multi-modal dialogue language model

    CN117036706A

  • Image editing method and device, equipment, storage medium and program product

    CN117611709A

  • Visual prompt generation method and device

    CN118229835A

  • Object-centric neural decomposition for image re-rendering

    US20230215085A1

Cited By

  • Insight-based research synthesis and processing platform

    GB2705378A