Image segmentation methods, devices, and electronic equipment based on large language models

Abstract

This invention provides an image segmentation method, apparatus, and electronic device based on a large language model, comprising: acquiring an image to be segmented and question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays starting from the center point, wherein the endpoints of the rays are used to indicate the contour of the target to be segmented; and performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented. By adaptively determining the contour of the target to be segmented by the center point and the preset number of rays starting from the center point, the contour shape information of the image to be segmented can be accurately preserved, effectively improving the segmentation accuracy during image segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to an image segmentation method, apparatus, and electronic device based on a large language model. Background Technology

[0002] Large Vision Language Models (LVLMs) have demonstrated impressive language processing capabilities, effectively aligning image and text tasks. While tasks in the natural language domain have achieved unification through techniques such as cue engineering or instruction tuning, a comprehensive unified representation remains lacking in vision tasks. This gap is particularly pronounced in object detection and image segmentation, the latter posing significant challenges to unifying vision tasks. Some methods have managed to convert object detection frameworks into text representations, improving the object localization capabilities of LVLMs. However, these methods are not particularly outstanding in terms of both text and image representation. Therefore, effectively converting image segmentation into text representations is crucial for achieving a unified approach to vision tasks.

[0003] In the field of image segmentation, the challenges related to image segmentation masks are complex and diverse. Unlike traditional methods, some studies, such as Lisa, GLaMM, and PixelLM, attempt to enable language models to learn mask tokens and generate segmentation predictions through specific decoder structures. However, these methods have not achieved true unification of text and image formats, adding an extra burden to large language models (LLMs) and failing to maintain a unified structure with large visual language models. Therefore, the best solution lies in fundamentally enhancing the fine-grained perception and localization capabilities of LVLMs, rather than relying on external assistance. Several challenges exist in the process of developing a unified representation method for image segmentation: first, more optimized and simplified visual representation methods are needed; second, a unified visual-text format needs to be constructed; and third, spatial shape information needs to be accurately preserved. Summary of the Invention

[0004] This invention provides an image segmentation method, apparatus, and electronic device based on a large language model, which addresses the shortcomings of low accuracy in image segmentation based on large language models in the prior art. It achieves accurate preservation of the contour shape information of the image to be segmented, effectively improving the segmentation accuracy during image segmentation.

[0005] This invention provides an image segmentation method based on a large language model, comprising the following steps.

[0006] Obtain the image to be segmented and the question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; Feature extraction is performed on the image to be segmented and the question text respectively to obtain visual embedding vectors and text embedding vectors; The center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, and a preset number of rays starting from the center point are determined, wherein the endpoint of the ray is used to indicate the outline of the target to be segmented. The image to be segmented is segmented based on the preset number of rays to obtain the target to be segmented.

[0007] According to an image segmentation method based on a large language model provided by the present invention, the step of determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays originating from the center point, includes: The center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector; Starting from the center point, the rays are determined sequentially in the following manner: the direction of the first ray is set, and for each subsequent ray, the angular range between the ray and the previous ray is set; The rays to be determined that meet the preset conditions within the angle range are taken as the rays, wherein the rays to be determined are a number of rays within the angle range that start from the center point and end at the contour point of the target to be segmented.

[0008] According to the image segmentation method based on a large language model provided by the present invention, the step of selecting the ray to be determined that satisfies the preset conditions within the angular range as the ray includes: Calculate the length of the ray to be determined at the maximum angle within the angular range, and the length of the ray to be determined at the maximum length; determine the difference between the length of the ray to be determined at the maximum angle and the length of the ray to be determined at the maximum length. When the difference is less than or equal to a preset threshold, the ray with the largest angle to be determined is taken as the ray; When the difference is greater than the preset threshold, the ray with the longest length to be determined is taken as the ray.

[0009] According to the present invention, an image segmentation method based on a large language model is provided, which segments the image to be segmented based on a preset number of rays to obtain the target to be segmented, including: The endpoints corresponding to the preset number of rays are determined, and the coordinates of the endpoints in the image to be segmented are determined; the image to be segmented is segmented based on the coordinates of the endpoints to obtain the target to be segmented.

[0010] According to the image segmentation method based on a large language model provided by the present invention, before performing feature extraction processing on the image to be segmented and the question text respectively, the method further includes: Image interpolation processing is performed on the image to be segmented to obtain an image to be segmented at a preset resolution.

[0011] According to the image segmentation method based on a large language model provided by the present invention, the step of performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector includes: The image to be segmented at the preset resolution is subjected to visual encoding processing to obtain image encoding features; The image encoding features are projected to obtain the visual embedding vector. The text feature extraction process is performed on the question text to obtain the text embedding vector.

[0012] The present invention also provides an image segmentation device based on a large language model, comprising the following modules: An acquisition module is used to acquire an image to be segmented and question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; The feature extraction module is used to perform feature extraction processing on the image to be segmented and the question text respectively to obtain visual embedding vector and text embedding vector; The determination module is used to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and to determine a preset number of rays starting from the center point, wherein the endpoint of the ray is used to indicate the outline of the target to be segmented; The image segmentation module is used to segment the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image segmentation method based on a large language model as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image segmentation method based on a large language model as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image segmentation method based on a large language model as described above.

[0016] The image segmentation method based on a large language model provided by this invention acquires the image to be segmented and the question text, and then performs feature extraction processing on the image to be segmented and the question text respectively to obtain visual embedding vectors and text embedding vectors. Based on the visual embedding vectors and text embedding vectors, the center point of the target to be segmented is determined, and a preset number of rays starting from the center point are determined. The endpoints of the rays indicate the contour of the target to be segmented, thereby performing image segmentation on the image to be segmented. This method can accurately preserve the contour shape information of the image to be segmented and effectively improve the segmentation accuracy during image segmentation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the image segmentation method based on a large language model provided by the present invention.

[0019] Figure 2 This is one of the overall structural diagrams of the image segmentation method based on a large language model provided by the present invention.

[0020] Figure 3 This is the second schematic diagram of the overall structure of the image segmentation method based on a large language model provided by the present invention.

[0021] Figure 4 This is a schematic diagram comparing the segmentation results of the image segmentation method based on a large language model provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the image segmentation device based on a large language model provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Figure 1This is a flowchart illustrating the image segmentation method based on a large language model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101 to 104: Step 101: Obtain the image to be segmented and the question text.

[0026] The question text is used to indicate the target to be segmented in the image to be segmented. The image to be segmented includes at least one target, which can be a specific entity such as an animal or plant.

[0027] Specifically, see Figure 2 , Figure 2 This is one of the overall structural diagrams of the image segmentation method based on a large language model provided by the present invention, such as... Figure 2 As shown, the input image X to be segmented v The question text X contains an image of a gray cat lying on a table. ins The question is: "Where is the cat located in this picture?"

[0028] In other embodiments, see Figure 3 , Figure 3 This is the second schematic diagram of the overall structure of the image segmentation method based on a large language model provided by the present invention, as shown below. Figure 3 As shown, the input image X to be segmented v The question text X contains an image of a koala perched on a tree branch. ins The question is: "Where is the koala located in this picture?"

[0029] Step 102: Perform feature extraction processing on the image to be segmented and the problem text respectively to obtain visual embedding vector and text embedding vector.

[0030] In some embodiments, after obtaining the image to be segmented and the question text, feature extraction processing needs to be performed on the image to be segmented and the question text separately, so as to facilitate the large language model to further predict the answer.

[0031] Before step 102, the method further includes the following steps: performing image interpolation processing on the image to be segmented to obtain an image to be segmented at a preset resolution.

[0032] Specifically, the large language model SegLLM used in this invention aims to unify pixel-level visual tasks in text form. This model transforms image segmentation tasks into a sequence-to-sequence format that follows instructions, achieving unified task processing. Therefore, for a given image X to be segmented... vFirst, the image needs to be interpolated using the CLIP-L / 14@224 model to convert its resolution to a preset resolution, thus obtaining the image to be segmented at the preset resolution. This reduces computational costs during subsequent model processing. For example, the interpolation algorithm can use nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., and the preset resolution can be 448*448.

[0033] In some embodiments, step 102 can be implemented by: performing visual encoding processing on the image to be segmented at the preset resolution to obtain image encoding features; performing projection processing on the image encoding features to obtain the visual embedding vector; and performing text feature extraction processing on the question text to obtain the text embedding vector.

[0034] See also Figure 2 ,exist Figure 2 In the process, for the image X to be segmented v First, the image to be segmented is subjected to the interpolation process described above. Then, the image to be segmented at a preset resolution is input into the Visual Encoder layer for visual encoding to obtain image encoded features. These visual encoded features are then input into the Projection Layer for projection to obtain visual embedding vectors. For the problem text, text features are directly extracted using a Tokenizer to obtain text embedding vectors. It should be noted that both the visual embedding vectors and text embedding vectors are text-dimensional features; therefore, they can be directly input into a large language model for processing.

[0035] In other embodiments, see also Figure 3 ,exist Figure 3 In the process, for the image X to be segmented v First, the image to be segmented is subjected to the interpolation process described above. Then, the image to be segmented at a preset resolution is input into the Visual Encoder layer for visual encoding to obtain image encoded features. These visual encoded features are then input into the Projection Layer for projection to obtain visual embedding vectors. For the problem text, text features are directly extracted using a Tokenizer to obtain text embedding vectors. Finally, the visual embedding vectors and text embedding vectors are input into the large language model for further processing.

[0036] Step 103: Determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determine a preset number of rays starting from the center point.

[0037] The endpoint of the ray is used to indicate the contour of the target to be segmented; that is, the endpoint of the ray is a point on the contour of the target to be segmented.

[0038] In some embodiments, step 103 can be implemented as follows: determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector; starting from the center point, determining the rays in the following order: setting the direction of the first ray, and for each subsequent ray, setting the angle range between the ray and the previous ray; taking the rays to be determined that meet the preset conditions within the angle range as the rays, wherein the rays to be determined are several rays within the angle range that start from the center point and end at the contour point of the target to be segmented.

[0039] Specifically, see Figure 3 Below, for the image input to be segmented, the calculator calculates the polar points, i.e., the endpoints of the rays. First, the center point of the target to be segmented needs to be determined based on the visual embedding vector and the text embedding vector. Here, the center point can be either the center of the coordinate box corresponding to the target to be segmented or the centroid of the target to be segmented. Then, rays are emitted from the center point towards the contour of the target to be segmented. The direction of the first ray is 0 degrees, and then subsequent rays are determined in a clockwise or counterclockwise order. Here, we take the counterclockwise direction as an example, setting the number of rays to 36, and the angle range between two rays is δ (δ=θ±σ, θ=10°, σ=5°). For the second ray, within this angle range, all rays formed by points from the center point to points on the contour are considered as rays to be determined, and then rays that meet the preset conditions are taken as the second ray.

[0040] In some embodiments, the step of using the ray to be determined that meets the preset conditions within the angle range as the ray includes: calculating the length of the ray to be determined with the largest angle and the length of the ray to be determined with the largest length within the angle range; determining the difference between the length of the ray to be determined with the largest angle and the length of the ray to be determined with the largest length; when the difference is less than or equal to a preset threshold, using the ray to be determined with the largest angle as the ray; when the difference is greater than the preset threshold, using the ray to be determined with the largest length as the ray.

[0041] Specifically, see [link to relevant documentation] Figure 3 Below, continuing with the second ray as an example, the Vistall algorithm can be used to calculate and sort the angles between the ray to be determined and the first ray corresponding to all contour points within the aforementioned angle range. Then, the endpoint corresponding to the ray to be determined with the largest angle is determined, denoted as p. c Simultaneously, calculate and sort the lengths of all rays to be determined within the aforementioned angle range, and determine the endpoint corresponding to the ray with the longest length, denoted as p. fThen, p is located using a coordinate system. c and p f At the corresponding position in the segmented image, calculate p. c Distance d to the center point c and p f The distance df to the center point. Point p. c Used to represent the local details of the target to be segmented within a given angular range, while p f This represents the global structure of the target to be segmented. Then, a threshold τ is set, and d is calculated. c and d f The difference between them; if this difference is less than or equal to τ, then choose p. c As polar coordinate point d i That is, the ray with the largest angle to be determined is used as the second ray to preserve the local details of the target to be segmented; conversely, if d f and d c If the difference between them is greater than τ, then choose p. f As polar coordinate point d i The maximum length ray to be determined is used as the second ray to preserve the global contour of the target to be segmented. The specific calculation method is shown in the following formula.

[0042]

[0043] Similarly, the third, fourth...thirty-sixth rays are determined in the same way until the coordinates of the poles (i.e. the coordinates of the endpoints of the rays) corresponding to all thirty-six rays are determined.

[0044] In this embodiment of the invention, by adaptively adjusting the position of each extremum, the selection of each extremum is more flexible when facing images with disorder and complexity. Therefore, local details can be highlighted more effectively, thereby greatly improving the accuracy of the segmentation result of the target to be segmented.

[0045] Step 104: Perform segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0046] In some embodiments, step 104 can be implemented in the following way: determining the endpoints corresponding to the preset number of rays, and determining the coordinates of the endpoints in the image to be segmented; performing segmentation processing on the image to be segmented based on the coordinates of the endpoints to obtain the target to be segmented.

[0047] Specifically, see [link to relevant documentation] Figure 2 After obtaining a preset number of rays through large language model processing, the endpoints corresponding to the preset number of rays are determined, and the coordinates of the endpoints in the image to be segmented are determined, for example... Figure 2The coordinates of all endpoints shown in the figure are "[<bin_742><bin_439><bin_737><bin_449> …<bin_737><bin_426> ]",in"<bin_742><bin_439> " " indicates the center point of the target to be segmented.<bin_737> "This represents the distance from the first endpoint to the center point."<bin_449> "This indicates the distance from the second endpoint to the center point."<bin_737> "This represents the distance from the penultimate endpoint to the center point."<bin_426> "" indicates the distance from the last endpoint to the center point.

[0048] For example Figure 3 The coordinates of all endpoints shown are “[742, 439, 737, 449…, 737, 426]”, where “742, 439” represent the center point of the target to be segmented, “737” represents the distance from the first endpoint to the center point, “449” represents the distance from the second endpoint to the center point, “737” represents the distance from the penultimate endpoint to the center point, and “426” represents the distance from the last endpoint to the center point.

[0049] It should be noted that in this invention, the endpoint, pole, and polar coordinate point all have the same meaning.

[0050] Then, the image to be segmented is segmented based on the coordinates of the endpoint to obtain the target to be segmented. That is, according to the coordinates of the poles arranged in order, all poles are connected in a preset clockwise or counterclockwise order to obtain the outline of the target to be segmented. Then, the image to be segmented is segmented according to the outline to obtain the target to be segmented.

[0051] In some embodiments, see Figure 4 , Figure 4 This is a comparative diagram showing the segmentation results of the image segmentation method based on a large language model provided by the present invention. Figure 4 The left side shows the image segmentation result based on the VistaLLM large language model. It can be seen that the head of the person in the image was not segmented, and the segmentation effect was poor. Figure 4 The middle section shows the image segmentation result based on PolarMask. Although it also selects a preset number of extrema on the contour of the target to be segmented for segmentation, the angle between the rays corresponding to each extrema is fixed. It does not employ the adaptive extrema selection method of this invention. Therefore, the segmentation effect on some local details is not ideal; for example, some details of the face and hands of the person to be segmented are not completely segmented. Figure 4On the right is the segmentation effect based on the SegLLM large language model of this application. It can be seen that the whole person to be segmented is segmented, and even some details of the hands and face are completely segmented. Therefore, it can be seen that the segmentation accuracy of this invention is greatly improved.

[0052] The image segmentation apparatus based on a large language model provided by the present invention is described below. The image segmentation apparatus based on a large language model described below and the image segmentation method based on a large language model described above can be referred to and correspond to each other.

[0053] See Figure 5 , Figure 5 This is a schematic diagram of the image segmentation device based on a large language model provided by the present invention, as shown below. Figure 5 As shown, the image segmentation device 500 based on a large language model provided by the present invention includes an acquisition module 501, a feature extraction module 502, a determination module 503, and an image segmentation module 504. The acquisition module 501 acquires an image to be segmented and question text, wherein the question text indicates the target to be segmented in the image to be segmented. The feature extraction module 502 performs feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector. The determination module 503 determines the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determines a preset number of rays starting from the center point, wherein the endpoints of the rays indicate the contour of the target to be segmented. The image segmentation module 504 performs segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0054] In some embodiments, the determining module 503 is further configured to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector; starting from the center point, determine the rays in the following order: set the direction of the first ray, and for each subsequent ray, set the angle range between the ray and the previous ray; take the rays to be determined that meet the preset conditions within the angle range as the rays, wherein the rays to be determined are a plurality of rays within the angle range that start from the center point and end at the contour point of the target to be segmented.

[0055] In some embodiments, the determining module 503 is further configured to calculate the length of the ray to be determined at the maximum angle within the angle range and the length of the ray to be determined at the maximum length; determine the difference between the length of the ray to be determined at the maximum angle and the length of the ray to be determined at the maximum length; when the difference is less than or equal to a preset threshold, the ray to be determined at the maximum angle is taken as the ray; when the difference is greater than the preset threshold, the ray to be determined at the maximum length is taken as the ray.

[0056] In some embodiments, the image segmentation module 504 is further configured to determine the endpoints corresponding to the preset number of rays, and determine the coordinates of the endpoints in the image to be segmented; and perform segmentation processing on the image to be segmented based on the coordinates of the endpoints to obtain the target to be segmented.

[0057] In some embodiments, the feature extraction module 502 is further configured to perform image interpolation processing on the image to be segmented to obtain an image to be segmented at a preset resolution.

[0058] In some embodiments, the feature extraction module 502 is further configured to perform visual encoding processing on the image to be segmented at a preset resolution to obtain image encoding features; perform projection processing on the image encoding features to obtain the visual embedding vector; and perform text feature extraction processing on the question text to obtain the text embedding vector.

[0059] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an image segmentation method based on a large language model. This method includes: acquiring an image to be segmented and question text, wherein the question text indicates the target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays originating from the center point, wherein the endpoints of the rays indicate the contour of the target to be segmented; and performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0060] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image segmentation method based on a large language model provided by the above methods. The method includes: acquiring an image to be segmented and question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays starting from the center point, wherein the endpoint of the rays is used to indicate the contour of the target to be segmented; and performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0062] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image segmentation method based on a large language model provided by the methods described above. The method includes: acquiring an image to be segmented and question text, wherein the question text is used to indicate a target to be segmented in the image to be segmented; performing feature extraction processing on the image to be segmented and the question text respectively to obtain a visual embedding vector and a text embedding vector; determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays originating from the center point, wherein the endpoints of the rays are used to indicate the contour of the target to be segmented; and performing segmentation processing on the image to be segmented based on the preset number of rays to obtain the target to be segmented.

[0063] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0064] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image segmentation method based on a large language model, characterized in that, include: Obtain the image to be segmented and the question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; Feature extraction is performed on the image to be segmented and the question text respectively to obtain visual embedding vectors and text embedding vectors; The center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector, and a preset number of rays starting from the center point are determined, wherein the endpoint of the ray is used to indicate the outline of the target to be segmented. The image to be segmented is segmented based on the preset number of rays to obtain the target to be segmented. The step of determining the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and determining a preset number of rays originating from the center point, includes: The center point of the target to be segmented is determined based on the visual embedding vector and the text embedding vector; Starting from the center point, the rays are determined sequentially in the following manner: the direction of the first ray is set, and for each subsequent ray, the angular range between the ray and the previous ray is set; The rays to be determined that meet the preset conditions within the angle range are taken as the rays, wherein the rays to be determined are a number of rays within the angle range that start from the center point and end at the contour point of the target to be segmented. The step of selecting the ray to be determined that meets the preset conditions within the angle range as the ray includes: Calculate the length of the ray to be determined at the maximum angle within the stated angle range, and the length of the ray to be determined at the maximum length; Determine the difference between the length of the ray to be determined at the maximum angle and the length of the ray to be determined at the maximum length; When the difference is less than or equal to a preset threshold, the ray with the largest angle to be determined is taken as the ray; When the difference is greater than the preset threshold, the ray with the longest length to be determined is taken as the ray.

2. The image segmentation method based on a large language model according to claim 1, characterized in that, The image to be segmented is segmented based on the preset number of rays to obtain the target to be segmented, including: Determine the endpoints corresponding to the preset number of rays, and determine the coordinates of the endpoints in the image to be segmented; The image to be segmented is segmented based on the coordinates of the endpoint to obtain the target to be segmented.

3. The image segmentation method based on a large language model according to claim 1, characterized in that, Before performing feature extraction processing on the image to be segmented and the question text respectively, the method further includes: Image interpolation processing is performed on the image to be segmented to obtain an image to be segmented at a preset resolution.

4. The image segmentation method based on a large language model according to claim 3, characterized in that, The step of performing feature extraction processing on the image to be segmented and the question text respectively to obtain visual embedding vectors and text embedding vectors includes: The image to be segmented at the preset resolution is subjected to visual encoding processing to obtain image encoding features; The image encoding features are projected to obtain the visual embedding vector. The text feature extraction process is performed on the question text to obtain the text embedding vector.

5. An image segmentation device based on a large-scale language model, characterized in that, include: An acquisition module is used to acquire an image to be segmented and question text, wherein the question text is used to indicate the target to be segmented in the image to be segmented; The feature extraction module is used to perform feature extraction processing on the image to be segmented and the question text respectively to obtain visual embedding vector and text embedding vector; The determination module is used to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector, and to determine a preset number of rays starting from the center point, wherein the endpoint of the ray is used to indicate the outline of the target to be segmented; The image segmentation module is used to segment the image to be segmented based on the preset number of rays to obtain the target to be segmented; The determining module specifically includes: The center point determination module is used to determine the center point of the target to be segmented based on the visual embedding vector and the text embedding vector. The setting module is used to determine the rays sequentially from the center point in the following manner: setting the direction of the first ray, and for each subsequent ray, setting the angle range between the ray and the previous ray; A ray determination module is used to identify rays within the angle range that meet preset conditions as the rays to be determined, wherein the rays to be determined are a number of rays within the angle range that start from the center point and end at the contour point of the target to be segmented; The ray determination module is specifically used for: Calculate the length of the ray to be determined at the maximum angle within the stated angle range, and the length of the ray to be determined at the maximum length; Determine the difference between the length of the ray to be determined at the maximum angle and the length of the ray to be determined at the maximum length; When the difference is less than or equal to a preset threshold, the ray with the largest angle to be determined is taken as the ray; When the difference is greater than the preset threshold, the ray with the longest length to be determined is taken as the ray.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image segmentation method based on a large language model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method based on a large language model as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method based on a large language model as described in any one of claims 1 to 4.

Images