Image segmentation method and apparatus based on large language model, and electronic device

Abstract

The present application provides an image segmentation method and apparatus based on a large language model, and an electronic device. The method comprises: obtaining an image to be segmented and a question text, wherein the question text is used for indicating a target to be segmented in said image; respectively performing feature extraction processing on said image and the question text to obtain a visual embedding vector and a text embedding vector; determining the central point of said target on the basis of the visual embedding vector and the text embedding vector, and determining a preset number of rays with the central point as a starting point, wherein the end points of the rays are used for indicating the contour of said target; and performing segmentation processing on said image on the basis of the preset number of rays to obtain said target. The contour of said target is adaptively determined by means of the central point and the preset number of rays with the central point as the starting point, such that contour shape information of said image can be accurately reserved, effectively improving the segmentation precision during image segmentation.

Description

Image segmentation method and device based on large language model, and electronic device

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 2024108456247, filed on June 27, 2024, and entitled “Image segmentation method and device based on large language model, and electronic device”, which is incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computer image processing, and in particular to an image segmentation method and device based on a large language model, and an electronic device. BACKGROUND

[0004] Large vision language models (LVLMs) have demonstrated impressive language processing capabilities, effectively enabling alignment between image and text tasks. While tasks in the natural language domain have been unified through techniques such as prompt engineering or instruction tuning, there is still a lack of a comprehensive unified representation in visual tasks. This gap is particularly evident in object detection and image segmentation, which pose significant challenges to unified visual tasks. Some methods have managed to convert object detection frameworks into textual representations, improving the ability of LVLMs to locate objects. However, these methods are not prominent in terms of the same text and image. Therefore, effectively converting image segmentation into a textual representation is crucial for implementing a unified approach to visual tasks.

[0005] In the field of image segmentation, the challenges related to processing image segmentation masks are complex and diverse. Unlike traditional methods, some studies such as Lisa, GLaMM, and PixelLM attempt to have language models learn mask tokens and generate segmentation predictions through specific decoder structures. However, these methods do not achieve true unification of text and image formats, add additional burdens to large language models (LLMs), and fail to maintain a unified structure with large vision language models. Therefore, the best solution lies in fundamentally enhancing the fine-grained perception and localization capabilities of LVLMs rather than relying on external assistance. In the process of unified representation of image segmentation, there are many challenges: first, a more optimized and simplified visual representation method is needed; second, a unified visual-text format needs to be constructed; in addition, accurate spatial shape information needs to be preserved.

[0006] SUMMARY

[0007] The application provides a large language model-based image segmentation method and device and electronic equipment to solve the low precision of image segmentation based on a large language model in the prior art, accurately preserves the contour shape information of the image to be segmented, and effectively improves the segmentation accuracy.

[0008] The application provides a large language model-based image segmentation method, including the following steps.

[0009] Obtain an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented;

[0010] Respectively perform feature extraction processing on the image to be segmented and the question text to obtain a visual embedding vector and a text embedding vector;

[0011] Determine a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays with the center point as the starting point, wherein the end point of the ray is used to indicate the contour of the target to be segmented;

[0012] Segment the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0013] According to the large language model-based image segmentation method provided by the application, the center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, and a preset number of rays with the center point as the starting point are determined, including:

[0014] Determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector;

[0015] With the center point as the starting point, the rays are sequentially determined in the following manner: set the direction of the first ray, and for each subsequent ray, set the angle range between the ray and the previous ray;

[0016] The to-be-determined rays in the angle range that meet the preset condition are taken as the rays, wherein the to-be-determined rays are a plurality of rays in the angle range with the center point as the starting point and the contour point of the target to be segmented as the end point.

[0017] According to the large language model-based image segmentation method provided by the application, the to-be-determined rays in the angle range that meet the preset condition are taken as the rays, including:

[0018] Respectively calculate the length of the to-be-determined ray with the maximum angle in the angle range and the length of the to-be-determined ray with the maximum length;

[0019] determining a difference between a length of the ray with the maximum angle and a length of the ray with the maximum length;

[0020] when the difference is less than or equal to a preset threshold, taking the ray with the maximum angle as the ray;

[0021] when the difference is greater than the preset threshold, taking the ray with the maximum length as the ray.

[0022] According to the image segmentation method based on a large language model provided in the present application, the preset number of rays are used to perform segmentation processing on the image to be segmented, so as to obtain the target to be segmented, which comprises the following steps:

[0023] determining the end points corresponding to the preset number of rays respectively, and determining the coordinates of the end points in the image to be segmented;

[0024] performing segmentation processing on the image to be segmented based on the coordinates of the end points, so as to obtain the target to be segmented.

[0025] According to the image segmentation method based on a large language model provided in the present application, before the feature extraction processing is performed on the image to be segmented and the question text respectively, the method further comprises the following steps:

[0026] performing image interpolation processing on the image to be segmented, so as to obtain an image to be segmented with a preset resolution.

[0027] According to the image segmentation method based on a large language model provided in the present application, the feature extraction processing is performed on the image to be segmented and the question text respectively, so as to obtain a visual embedding vector and a text embedding vector, which comprises the following steps:

[0028] performing visual coding processing on the image to be segmented with the preset resolution, so as to obtain an image coding feature;

[0029] performing projection processing on the image coding feature, so as to obtain the visual embedding vector;

[0030] performing text feature extraction processing on the question text, so as to obtain the text embedding vector.

[0031] The present application further provides an image segmentation device based on a large language model, which comprises the following modules:

[0032] an acquisition module, which is used to acquire an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented;

[0033] a feature extraction module, which is used to perform feature extraction processing on the image to be segmented and the question text respectively, so as to obtain a visual embedding vector and a text embedding vector.

[0034] determining a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays with the center point as a starting point, wherein endpoints of the rays are used to indicate an outline of the target to be segmented;

[0035] an image segmentation module configured to perform segmentation processing on the image to be segmented based on the preset number of rays, to obtain the target to be segmented.

[0036] The application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image segmentation method based on a large language model according to any one of the above when executing the program.

[0037] The application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the image segmentation method based on a large language model according to any one of the above.

[0038] The application also provides a computer program product including a computer program, wherein the computer program is executable by a processor to implement the image segmentation method based on a large language model according to any one of the above.

[0039] The image segmentation method based on a large language model provided by the application can obtain an image to be segmented and a question text, then perform feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector, determine a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays with the center point as a starting point, wherein endpoints of the rays are used to indicate an outline of the target to be segmented, so as to perform image segmentation on the image to be segmented, which can accurately retain the outline shape information of the image to be segmented and effectively improve the segmentation accuracy during image segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0041] FIG. 1 is a flowchart of the image segmentation method based on a large language model provided by the application;

[0042] FIG. 2 is a schematic diagram of the overall structure of the image segmentation method based on a large language model provided by the application;

[0043] FIG. 3 is a schematic diagram of the overall structure of the image segmentation method based on a large language model according to the present application;

[0044] FIG. 4 is a schematic diagram of the segmentation results of the image segmentation method based on a large language model according to the present application;

[0045] FIG. 5 is a schematic diagram of the structure of the image segmentation device based on a large language model according to the present application;

[0046] FIG. 6 is a schematic diagram of the structure of the electronic device according to the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0048] FIG. 1 is a schematic diagram of the flow of the image segmentation method based on a large language model according to the present application, as shown in FIG. 1, the method comprises the following steps 101 to 104.

[0049] Step 101, obtaining an image to be segmented and a question text.

[0050] The question text is used to indicate a target to be segmented in the image to be segmented. The image to be segmented includes at least one target to be segmented, which can be a specific entity such as an animal or a plant.

[0051] Specifically, referring to FIG. 2, which is a schematic diagram of the overall structure of the image segmentation method based on a large language model according to the present application, as shown in FIG. 2, the input image to be segmented X v is an image containing a gray cat lying on a table, the question text X ins is "Where is the cat in this picture?".

[0052] In other embodiments, referring to FIG. 3, which is a schematic diagram of the overall structure of the image segmentation method based on a large language model according to the present application, as shown in FIG. 3, the input image to be segmented X v is an image containing a koala lying on a branch, the question text X ins is "Where is the koala in this picture?".

[0053] Step 102, respectively, the feature extraction processing is carried out to the image to be segmented and the question text, and the visual embedding vector and the text embedding vector are obtained.

[0054] In some embodiments, after obtaining the image to be segmented and the question text, feature extraction processing needs to be carried out to the image to be segmented and the question text respectively, so as to facilitate the large language model to further predict the answer.

[0055] Before step 102, the following step is further included: image interpolation processing is carried out to the image to be segmented, and the image to be segmented with a preset resolution is obtained.

[0056] Specifically, the large language model SegLLM used in the present application aims to unify the pixel-level visual task in the form of text. The model converts the image segmentation task into a sequence-to-sequence format following the instruction, and realizes the unified processing of the task. Therefore, for a given image to be segmented X v , it is necessary to use the CLIP-L / 14@224 model to perform interpolation processing on it first, convert its resolution to a preset resolution, and obtain the image to be segmented with a preset resolution, so as to reduce the calculation cost when the subsequent model processing. Illustratively, the interpolation algorithm can use nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., and the preset resolution can be 448*448.

[0057] In some embodiments, step 102 can be implemented in the following manner: the image to be segmented with a preset resolution is subjected to visual coding processing to obtain image coding features; the image coding features are subjected to projection processing to obtain the visual embedding vector; the question text is subjected to text feature extraction processing to obtain the text embedding vector.

[0058] Continuing to refer to FIG. 2, in FIG. 2, for the image to be segmented X v , the interpolation processing described above is first performed, and then the image to be segmented with a preset resolution is input into the visual coding layer (Visual Encoder) for visual coding processing to obtain image coding features, and the visual coding features are input into the projection layer (Projection Layer) for projection processing to obtain the visual embedding vector. For the question text, text feature extraction processing is directly performed through the Tokenizer to obtain the text embedding vector. It should be noted that the visual embedding vector and the text embedding vector described above are both features in the text dimension, and therefore the visual embedding vector and the text embedding vector described above can be directly input into the large language model for processing.

[0059] In other embodiments, continuing to refer to FIG. 3, in FIG. 3, for the image to be segmented X vThe image to be segmented of the preset resolution is input into a visual encoding layer (Visual Encoder) for visual encoding processing after the interpolation processing, to obtain image encoding features, and then the visual encoding features are input into a projection layer (Projection Layer) for projection processing to obtain a visual embedding vector. For the problem text, the text features are extracted by the Tokenizer to obtain a text embedding vector, and then the visual embedding vector and the text embedding vector are input into the large language model for processing.

[0060] In step 103, the center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, and a preset number of rays starting from the center point are determined.

[0061] The end point of the ray is used to indicate the contour of the target to be segmented, that is, the end point of the ray is a point on the contour of the target to be segmented.

[0062] In some embodiments, step 103 can be implemented in the following manner: the center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector; the rays are sequentially determined starting from the center point in the following manner: the direction of the first ray is set, and for each subsequent ray, the angle range between the ray and the previous ray is set; the to-be-determined ray in the angle range that meets the preset condition is taken as the ray, wherein the to-be-determined ray is a plurality of rays in the angle range starting from the center point and ending at the contour point of the target to be segmented.

[0063] Specifically, referring to the lower part of FIG. 3, the polar point coordinates (Polar Points) of the image to be segmented (Image Input) are calculated, that is, the end points of the rays are calculated. First, the center point of the target to be segmented is determined according to the visual embedding vector and the text embedding vector. Here, the center point can be the center of the coordinate frame corresponding to the target to be segmented, or the center of mass of the target to be segmented. Then, a ray is emitted from the center point to the contour direction of the target to be segmented. The direction of the first ray is 0 degrees, and then the subsequent rays are sequentially determined in the clockwise or counterclockwise direction. Here, the counterclockwise direction is taken as an example, and the number of rays is set to 36. The angle range between the two rays is δ (δ = θ ± σ, θ = 10°, σ = 5°). For the second ray, all rays from the center point to the points on the contour within the angle range are regarded as to-be-determined rays, and then the to-be-determined ray that meets the preset condition is taken as the second ray.

[0064] In some embodiments, the determining the ray from the rays to be determined in the angle range satisfying the preset condition comprises: calculating the length of the ray to be determined with the maximum angle in the angle range and the length of the ray to be determined with the maximum length respectively; determining the difference between the length of the ray to be determined with the maximum angle and the length of the ray to be determined with the maximum length; when the difference is less than or equal to a preset threshold, the ray to be determined with the maximum angle is determined as the ray; when the difference is greater than the preset threshold, the ray to be determined with the maximum length is determined as the ray.

[0065] Specifically, continuing to refer to FIG. 3 below, and continuing to take the second ray as an example, the vistall algorithm can be used to calculate and sort the angles between the rays to be determined corresponding to all the contour points in the angle range and the first ray, and then the end point corresponding to the ray to be determined with the maximum angle is determined, denoted as p c ; meanwhile, the lengths of all the rays to be determined in the angle range are calculated and sorted, and the end point corresponding to the ray to be determined with the maximum length is determined, denoted as p f . Then, p c and p f are located in the corresponding positions in the segmented image through the coordinate system, and the distance d c from p c to the center point and the distance d f from p f to the center point are calculated. p c is used to represent the local details of the target to be segmented in a given angle range, and p f represents the global structure of the target to be segmented. Then, a threshold τ is set, and the difference between d c and d f is calculated. If the difference is less than or equal to τ, p c is selected as the polar coordinate point d i , that is, the ray to be determined with the maximum angle is taken as the second ray to retain the local details of the target to be segmented; on the contrary, if the difference between d f and d c is greater than τ, p f is selected as the polar coordinate point d i , that is, the ray to be determined with the maximum length is taken as the second ray to retain the global contour of the target to be segmented. The specific calculation method is shown in the following formula.

[0066] Similarly, the third, fourth,..., and thirty-sixth rays are determined through the above method, until the polar coordinate points (that is, the coordinates of the end points of the rays) corresponding to all the thirty-six rays are determined.

[0067] In the embodiments of the present application, by adaptively adjusting the position of each pole, the selection of each pole is more flexible when facing images with disorder and complexity, and thus the local details can be highlighted more effectively, thereby greatly improving the accuracy of the segmentation result of the target to be segmented.

[0068] In step 104, the image to be segmented is segmented based on the preset number of rays to obtain the target to be segmented.

[0069] In some embodiments, step 104 can be specifically implemented by determining the end points corresponding to the preset number of rays respectively, and determining the coordinates of the end points in the image to be segmented; and segmenting the image to be segmented based on the coordinates of the end points to obtain the target to be segmented.

[0070] Specifically, continuing to refer to FIG. 2, after obtaining the preset number of rays by processing through the large language model, the end points corresponding to the preset number of rays are determined, and the coordinates of the end points in the image to be segmented are determined, for example, the coordinates of all end points shown in FIG. 2 are “[<bin_742><bin_439><bin_737><bin_449>…<bin_737><bin_426>]”, wherein “<bin_742><bin_439>” represents the center point of the target to be segmented, “<bin_737>” represents the distance from the first end point to the center point, “<bin_449>” represents the distance from the second end point to the center point, “<bin_737>” represents the distance from the second-to-last end point to the center point, and “<bin_426>” represents the distance from the last end point to the center point.

[0071] For another example, the coordinates of all end points shown in FIG. 3 are “[742, 439, 737, 449…, 737, 426]”, wherein “742, 439” represents the center point of the target to be segmented, “737” represents the distance from the first end point to the center point, “449” represents the distance from the second end point to the center point, “737” represents the distance from the second-to-last end point to the center point, and “426” represents the distance from the last end point to the center point.

[0072] It should be noted that the end point, the pole and the polar coordinate point in the present application all have the same meaning.

[0073] Then, the image to be segmented is segmented based on the coordinates of the end points to obtain the target to be segmented, that is, according to the coordinates of the poles arranged in order, all the poles are connected in the preset clockwise or counterclockwise order, and thus the contour of the target to be segmented is obtained, and then the image to be segmented is segmented according to the contour, and thus the target to be segmented is obtained.

[0074] In some embodiments, referring to FIG. 4, which is a comparison diagram of segmentation results of the large language model-based image segmentation method provided by the present application, the left side of FIG. 4 is the result of image segmentation based on the large language model of VistaLLM. It can be seen that the head of the person in the image is not segmented out, and the segmentation effect is poor. The middle of FIG. 4 is the result of image segmentation based on PolarMask. Although it is also segmented by selecting a preset number of poles on the contour of the target to be segmented, the angle between the rays corresponding to each pole is fixed, and the adaptive selection method of the poles provided by the present application is not used. Therefore, the segmentation effect in some local details is not ideal, for example, some details of the face and hands of the target to be segmented are not completely segmented out. The right side of FIG. 4 is the segmentation effect based on the large language model of SegLLM provided by the present application. It can be seen that the whole target to be segmented is segmented out, even some detailed parts of the hands and face are completely segmented out. Therefore, it can be seen that the segmentation accuracy provided by the present application is greatly improved.

[0075] The large language model-based image segmentation device provided by the present application is described below. The large language model-based image segmentation device described below can be correspondingly referred to the large language model-based image segmentation method described above.

[0076] Referring to FIG. 5, which is a structural diagram of the large language model-based image segmentation device provided by the present application, as shown in FIG. 5, the large language model-based image segmentation device 500 provided by the present application includes an acquisition module 501, a feature extraction module 502, a determination module 503, and an image segmentation module 504. The acquisition module 501 is configured to acquire an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented. The feature extraction module 502 is configured to perform feature extraction processing on the image to be segmented and the question text respectively, to obtain a visual embedding vector and a text embedding vector. The determination module 503 is configured to determine a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays with the center point as the starting point, wherein the end point of the ray is used to indicate the contour of the target to be segmented. The image segmentation module 504 is configured to perform segmentation processing on the image to be segmented based on the preset number of rays, to obtain the target to be segmented.

[0077] In some embodiments, the determining module 503 is further configured to determine a center point of the target to be segmented based on the visual embedding vector and the text embedding vector; and sequentially determine the rays starting from the center point in the following manner: set a direction of a first ray, and for each subsequent ray, set an angle range between the ray and a previous ray; and determine, as the ray, a to-be-determined ray in the angle range that satisfies a preset condition, wherein the to-be-determined ray is a ray in the angle range that starts from the center point and ends at a contour point of the target to be segmented.

[0078] In some embodiments, the determining module 503 is further configured to calculate a length of a to-be-determined ray of a maximum angle in the angle range and a length of a to-be-determined ray of a maximum length, respectively; determine a difference between the length of the to-be-determined ray of the maximum angle and the length of the to-be-determined ray of the maximum length; when the difference is less than or equal to a preset threshold, determine the to-be-determined ray of the maximum angle as the ray; and when the difference is greater than the preset threshold, determine the to-be-determined ray of the maximum length as the ray.

[0079] In some embodiments, the image segmentation module 504 is further configured to determine end points corresponding to the preset number of rays, respectively, and determine coordinates of the end points in the image to be segmented; and perform segmentation processing on the image to be segmented based on the coordinates of the end points to obtain the target to be segmented.

[0080] In some embodiments, the feature extraction module 502 is further configured to perform image interpolation processing on the image to be segmented to obtain an image to be segmented at a preset resolution.

[0081] In some embodiments, the feature extraction module 502 is further configured to perform visual encoding processing on the image to be segmented at the preset resolution to obtain image encoding features; perform projection processing on the image encoding features to obtain the visual embedding vector; and perform text feature extraction processing on the question text to obtain the text embedding vector.

[0082] Fig. 6 illustrates an entity structure diagram of an electronic device, as shown in Fig. 6, the electronic device can include: a processor 610, a communications interface 620, a memory 630 and a communications bus 640, wherein the processor 610, the communications interface 620, the memory 630 complete the communication among each other through the communications bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the large language model-based image segmentation method, the method comprising: obtaining an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; respectively performing feature extraction processing on the image to be segmented and the question text to obtain a visual embedding vector and a text embedding vector; determining a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays with the center point as the starting point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0083] In addition, the logic instructions in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program code storage media.

[0084] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program is executable by a processor to enable the computer to perform the large language model-based image segmentation method provided by the above method, which comprises: obtaining an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays with the center point as the starting point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; and performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0085] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is executable by a processor to implement the large language model-based image segmentation method provided by the above method, which comprises: obtaining an image to be segmented and a question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining a center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays with the center point as the starting point, wherein the end point of the ray is used to indicate the contour of the target to be segmented; and performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0086] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0087] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the various embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that contributes to the technical solutions can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image segmentation method based on a large language model, comprising: Obtain the image to be segmented and the question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; Feature extraction is performed on the image to be segmented and the question text respectively to obtain visual embedding vectors and text embedding vectors; The center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, and a preset number of rays starting from the center point are determined, wherein the endpoint of the ray is used to indicate the outline of the target to be segmented. The image to be segmented is segmented based on the preset number of rays to obtain the target to be segmented.

2. The image segmentation method based on a large language model according to claim 1, wherein, The step of determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays originating from the center point, includes: The center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector; Starting from the center point, the rays are determined sequentially in the following manner: the direction of the first ray is set, and for each subsequent ray, the angular range between the ray and the previous ray is set; The rays to be determined that meet the preset conditions within the angle range are taken as the rays, wherein the rays to be determined are a number of rays within the angle range that start from the center point and end at the contour point of the target to be segmented.

3. The image segmentation method based on a large language model according to claim 2, wherein, The step of selecting the ray to be determined that meets the preset conditions within the angle range as the ray includes: Calculate the length of the ray to be determined at the maximum angle within the stated angle range, and the length of the ray to be determined at the maximum length; Determine the difference between the length of the ray to be determined at the maximum angle and the length of the ray to be determined at the maximum length; When the difference is less than or equal to a preset threshold, the ray with the largest angle to be determined is taken as the ray; When the difference is greater than the preset threshold, the ray with the longest length to be determined is taken as the ray.

4. The image segmentation method based on a large language model according to claim 1, wherein, The image to be segmented is segmented based on the preset number of rays to obtain the target to be segmented, including: Determine the endpoints corresponding to the preset number of rays, and determine the coordinates of the endpoints in the image to be segmented; The image to be segmented is segmented based on the coordinates of the endpoint to obtain the target to be segmented.

5. The image segmentation method based on a large language model according to claim 1, wherein, Before performing feature extraction processing on the image to be segmented and the question text respectively, the method further includes: Image interpolation processing is performed on the image to be segmented to obtain an image to be segmented at a preset resolution.

6. The image segmentation method based on a large language model according to claim 5, wherein, The step of performing feature extraction processing on the image to be segmented and the question text respectively to obtain visual embedding vectors and text embedding vectors includes: The image to be segmented at the preset resolution is subjected to visual encoding processing to obtain image encoding features; The image encoding features are projected to obtain the visual embedding vector. The text feature extraction process is performed on the question text to obtain the text embedding vector.

7. An image segmentation device based on a large language model, comprising: An acquisition module is used to acquire an image to be segmented and question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; The feature extraction module is used to perform feature extraction processing on the image to be segmented and the question text respectively to obtain visual embedding vector and text embedding vector; The determination module is used to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and to determine a preset number of rays starting from the center point, wherein the endpoint of the ray is used to indicate the outline of the target to be segmented; The image segmentation module is used to segment the image to be segmented based on the preset number of rays to obtain the target to be segmented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it implements the image segmentation method based on a large language model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the image segmentation method based on a large language model as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the image segmentation method based on a large language model as described in any one of claims 1 to 6.

Images