Entity segmentation method and device based on dynamic programming, equipment and medium

Abstract

The invention discloses an entity segmentation method and device based on dynamic programming, equipment and a medium, and relates to the field of computer vision, and the method comprises the steps: inputting text information in an obtained preset data set into a preset image recognition model to obtain corresponding vector information, training a preset entity segmentation model based on the vector information to obtain a target model; performing entity segmentation on each target image in the preset data set based on a preset entity segmentation model to obtain a corresponding first mask set; obtaining a target query request, and generating a second mask set corresponding to the target query request based on the target model; and determining target mask information corresponding to the second mask set from the first mask set by using a preset dynamic programming algorithm, determining target semantic mask information based on the target mask information and the second semantic mask information, segmenting a target entity image from the target image by using the target semantic mask information, and displaying the segmented target entity image. And returning the target entity image to the user side to respond to the target query request.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, apparatus, device and medium for entity segmentation based on dynamic programming. Background Art

[0002] With the rapid development of artificial intelligence (AI), interactive applications such as augmented reality, virtual reality, and image and video editing have experienced explosive growth. In these scenarios, accurately segmenting and identifying specific objects in images has become a critical requirement. In the field of computer vision, CLIP (Contrastive Language-Image Pre-Training), a multimodal pre-training model based on contrastive learning, and SAM (Segment Anything Model), a pre-training model for image segmentation, are two highly influential models. CLIP can identify objects in images without being exposed to samples of specific categories, while SAM can perform interactive segmentation in a variety of fields. However, each has limitations. CLIP is relatively weak in segmentation capabilities, while SAM, while having strong segmentation capabilities, has shortcomings in identifying the specific categories of segmented objects.

[0003] There are already some relevant research results. For example, the Semantic-SAM universal image segmentation model, the Open-Vocabulary SAM model based on the integration of SAM and CLIP, and the TAP (Tokenize Anything via Prompting) model based on the SAM architecture, a unified promptable visual foundation model. However, these research results still face many problems. The labeling cost remains high, the training process is complex and tedious, and the performance in specific verticals is unsatisfactory. For example, in image segmentation tasks in specific industries, the requirement for large amounts of accurately labeled data and complex training processes leads to high application costs and unsatisfactory results. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a dynamic programming-based entity segmentation method, apparatus, device, and medium that can organically combine the zero-shot recognition capability of CLIP with the segmentation capability of SAM to achieve more efficient and accurate interactive open vocabulary segmentation and recognition. The specific solution is as follows:

[0005] In a first aspect, the present application provides an entity segmentation method based on dynamic programming, comprising:

[0006] Obtaining a preset data set, inputting text information in the preset data set into a preset image recognition model to obtain vector information corresponding to the text information, and training a preset entity segmentation model based on the vector information to obtain a target model; the preset data set includes target images, text information corresponding to each target image, and first semantic mask information including semantic information of a first target object;

[0007] Performing entity segmentation on each of the target images in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set; the first mask set includes mask information that does not contain semantic information after performing the entity segmentation on the target image;

[0008] Obtaining a target query request input by a user terminal, and generating a second mask set corresponding to a second target object corresponding to the target query request in the target image based on the target model; the second mask set includes second semantic mask information including semantic information of the second target object;

[0009] Using a preset dynamic programming algorithm, target mask information corresponding to the second semantic mask information in the second mask set is determined from the first mask set, and target semantic mask information is determined based on the target mask information and the second semantic mask information, so as to use the target semantic mask information to segment a target entity image from the target image, and return the target entity image to the user end to respond to the target query request.

[0010] Optionally, the training of a preset entity segmentation model based on the vector information to obtain a target model includes:

[0011] Fusing the vector information with an encoder in a preset entity segmentation model to obtain a text encoder, so that the preset entity segmentation model recognizes the acquired text information based on the text encoder after fusion with itself;

[0012] Based on the target images in the preset data set, the text information corresponding to each target image and the first semantic mask information containing the semantic information of the first target object, the preset entity segmentation model including the text encoder is trained to obtain a target model; wherein, the first target object is an object with semantic information in the preset data set.

[0013] Optionally, performing entity segmentation on each target image in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set includes:

[0014] Based on the preset entity segmentation model, each target image in the preset data set is entity segmented to obtain a third target object, and mask information that does not contain semantic information corresponding to each third target object is determined to obtain a first mask set; wherein the third target objects include objects with semantic information and objects without semantic information.

[0015] Optionally, generating, based on the target model, a second mask set corresponding to a second target object corresponding to the target query request in the target image includes:

[0016] The target query request is obtained through the target model, so that the target model determines the second target object and the corresponding target area corresponding to each target query request from the target image, and generates second semantic mask information corresponding to the second target object and containing semantic information of the second target object, so as to obtain a second mask set corresponding to each target query request.

[0017] Optionally, the determining, from the first mask set, target mask information corresponding to the second semantic mask information in the second mask set by using a preset dynamic programming algorithm includes:

[0018] Determining second semantic mask information corresponding to the target query request, and determining a first object area corresponding to the second semantic mask information;

[0019] Selecting a piece of mask information from the unselected mask information in the first mask set as current mask information;

[0020] Determining a second object region corresponding to the current mask information, and merging the second object region corresponding to the current mask information with merged object regions corresponding to all existing mask information in the target mask information combination to obtain a current merged object region; wherein an initial target mask information combination is empty;

[0021] calculating a current IoU between the first object region and the currently merged object region, and if the current IoU is not zero and is greater than a target IoU between the first object region and the merged object regions corresponding to all mask information already present in the target mask information combination, adding the current mask information to the target mask information combination;

[0022] Jump to the step of selecting a mask information from the mask information that has not been selected in the current first mask set as the current mask information, until the first mask set is traversed, and the mask information in the target mask information combination is used as the target mask information corresponding to the second semantic mask information in the second mask set corresponding to the target query request.

[0023] Optionally, determining target semantic mask information based on the target mask information and the second semantic mask information includes:

[0024] Using the semantic information corresponding to the second semantic mask information, the mask information that does not include the semantic information in the target mask information is removed to obtain new target mask information, and the semantic information corresponding to the second semantic mask information is fused with the new target mask information to obtain target semantic mask information containing semantic information.

[0025] Optionally, before determining, from the first mask set, target mask information corresponding to the second semantic mask information in the second mask set using a preset dynamic programming algorithm, the method further includes:

[0026] The mask information in the first mask set and the second mask set is downsampled based on a preset step size to obtain a preprocessed first mask set and a preprocessed second mask set; the preset step size includes a preset horizontal step size and a preset vertical step size.

[0027] In a second aspect, the present application provides an entity segmentation device based on dynamic programming, comprising:

[0028] a model determination module, configured to obtain a preset data set, input text information in the preset data set into a preset image recognition model to obtain vector information corresponding to the text information, and train a preset entity segmentation model based on the vector information to obtain a target model; the preset data set includes target images, text information corresponding to each target image, and first semantic mask information including semantic information of a first target object;

[0029] a first mask set acquisition module, configured to perform entity segmentation on each target image in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set; the first mask set includes mask information obtained after performing the entity segmentation on the target image and excluding semantic information;

[0030] a second mask set acquisition module, configured to acquire a target query request input by a user terminal and generate a second mask set corresponding to a second target object corresponding to the target query request in the target image based on the target model; the second mask set includes second semantic mask information including semantic information of the second target object;

[0031] An image segmentation module is configured to determine, from the first mask set, target mask information corresponding to the second semantic mask information in the second mask set using a preset dynamic programming algorithm, and determine target semantic mask information based on the target mask information and the second semantic mask information, so as to segment a target entity image from the target image using the target semantic mask information, and return the target entity image to the user terminal to respond to the target query request.

[0032] In a third aspect, the present application provides an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] The processor is configured to execute the computer program to implement the aforementioned entity segmentation method based on dynamic programming.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned entity segmentation method based on dynamic programming when executed by a processor.

[0036] In the present application, a preset data set is obtained, and the text information in the preset data set is input into a preset image recognition model to obtain vector information corresponding to the text information, and a preset entity segmentation model is trained based on the vector information to obtain a target model; the preset data set includes a target image, text information corresponding to each target image, and first semantic mask information containing semantic information of a first target object; based on the preset entity segmentation model, each target image in the preset data set is entity segmented to obtain a corresponding first mask set; the first mask set includes mask information that does not contain semantic information obtained after the target image is entity segmented; user-side input is obtained , a target query request is received, and a second mask set corresponding to the second target object corresponding to the target query request in the target image is generated based on the target model; the second mask set includes second semantic mask information containing semantic information of the second target object; a preset dynamic programming algorithm is used to determine the target mask information corresponding to the second semantic mask information in the second mask set from the first mask set, and the target semantic mask information is determined based on the target mask information and the second semantic mask information, so as to segment the target entity image from the target image using the target semantic mask information, and return the target entity image to the user terminal to respond to the target query request. From the above, it can be seen that the present application selects a preset data set, obtains the vector information corresponding to the text information in the preset data set based on a preset image recognition model, and uses the vector information to train the preset entity segmentation model to obtain the target model, which only requires a small amount of preset data, reduces the dependence on large-scale labeled data, and reduces the labeling cost. The training process is simple, without the need for complex solutions or the introduction of large-scale new data sets, thus reducing training costs and improving training efficiency. Furthermore, a preset entity segmentation model is used to obtain a first mask set that does not contain semantic information corresponding to the target image in the preset data set, and a target model is used to obtain a second mask set that contains semantic information corresponding to the target query request input by the user in the target image. A preset dynamic programming algorithm is then used to determine the target mask information corresponding to the second mask set from the first mask set. The target semantic mask information is then determined based on the target mask information and the second semantic mask information in the second mask set, so as to segment the target entity image from the target image using the target semantic mask information. The target model obtains image semantic results and provides semantic information, while the preset entity segmentation model provides detailed mask information. Dynamic programming is used to bridge the results of the target model and the preset entity segmentation model. The combination of the two ensures that the final segmentation result, i.e., the target semantic mask information, has precise boundaries and carries semantic labels. This allows for more accurate identification and segmentation of specific objects in vertical fields, meeting the requirements of scenarios with high segmentation results and improving vertical performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0038] Figure 1 This is a flow chart of an entity segmentation method based on dynamic programming disclosed in this application;

[0039] Figure 2 A schematic diagram of the content of a specific publicly available LIP dataset;

[0040] Figure 3 This is a schematic diagram of an image obtained after entity segmentation using the SAM model disclosed in this application;

[0041] Figure 4 A schematic diagram of a specific dynamic programming algorithm disclosed in this application;

[0042] Figure 5 A schematic diagram of a specific entity segmentation method based on dynamic programming disclosed in this application;

[0043] Figure 6 This is a schematic structural diagram of an entity segmentation device based on dynamic programming disclosed in this application;

[0044] Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] Currently, existing Semantic-SAM general image segmentation models, the Open-Vocabulary SAM model based on the integration of SAM and CLIP, and the TAP model based on the SAM architecture still have many problems, such as high annotation costs, complex and cumbersome training processes, and unsatisfactory performance in vertical fields. To this end, this application provides an entity segmentation method based on dynamic programming, which can organically combine the zero-shot recognition capability of CLIP with the segmentation capability of SAM to achieve more efficient and accurate interactive open-vocabulary segmentation and recognition.

[0047] See also Figure 1 As shown, the embodiment of the present application discloses an entity segmentation method based on dynamic programming, including:

[0048] Step S11, obtain a preset data set, and input the text information in the preset data set into a preset image recognition model to obtain vector information corresponding to the text information, and train a preset entity segmentation model based on the vector information to obtain a target model; the preset data set includes target images, text information corresponding to each of the target images, and first semantic mask information containing semantic information of the first target object.

[0049] In this embodiment, a small number of target images (e.g., several thousand) can be selected to form a preset data set. For example, target images can be selected from the LIP (Look in Person, a large-scale data set) data set. The LIP data set includes images of various categories. The LIP data set also includes text information (Text) corresponding to the image and mask information (Mask) containing semantic information of the target object. Figure 2 As shown in the figure, the mask information is used to represent the area of the target object in the image. The categories of the mask information include but are not limited to hat, hair, gloves, sunglasses, upper-clothes, dress, coat, socks, pants, jumpsuits, scarf, skirt, face, left-arm, right-arm, left-leg, right-leg, left-shoe, and right-shoe.

[0050] The text information in the preset data set is then input into a preset image recognition model to obtain vector information corresponding to the text information, and the preset entity segmentation model is trained based on the vector information to obtain a target model. Specifically, the vector information can first be fused with the encoder in the preset entity segmentation model to obtain a text encoder, so that the preset entity segmentation model recognizes the acquired text information based on its own fused text encoder; then, the preset entity segmentation model including the text encoder is trained based on the target images in the preset data set, the text information corresponding to each target image, and the first semantic mask information containing the semantic information of the first target object to obtain the target model; wherein the first target object is an object with semantic information in the preset data set.

[0051] For example, the CLIP model can be selected as the preset image recognition model, and the SAM model can be selected as the preset entity segmentation model. The CLIP model is used to process the text information in the preset dataset to convert it into vector form. Because the CLIP model has strong cross-modal understanding capabilities, it can map text into a feature space and represent the text's semantic information as vectors. The resulting text vector information can then be fused with the SAM model's encoder to produce a text encoder. This fusion method is similar to the SAM model's fusion of vectors obtained by point encoding and bbox encoding. Fusion of the text vector with the SAM encoder enables the SAM model to incorporate text semantic information for image segmentation. Furthermore, the SAM model, including the text encoder, is trained based on the target images in the preset dataset, the text information corresponding to each target image, and a first semantic mask containing semantic information about the first target object to produce a target model, namely, a text-guided SAM model.

[0052] Step S12: performing entity segmentation on each target image in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set; the first mask set includes mask information that does not contain semantic information after performing the entity segmentation on the target image.

[0053] In this embodiment, performing entity segmentation on each of the target images in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set may include: performing entity segmentation on each of the target images in the preset data set based on the preset entity segmentation model to obtain a third target object, and determining mask information corresponding to each third target object that does not contain semantic information to obtain a first mask set; wherein the third target object includes objects with semantic information and objects without semantic information.

[0054] For example, the SAM model can be used to perform entity segmentation on each target image in the preset data set to obtain the corresponding third target object. The third target object is not less than the first target object in step S11, and since the amount of text information in the preset data set is limited, there are entity objects containing semantic information and entity objects not containing semantic information in the target image in the preset data set. The SAM model is used to segment each entity object in the target image to distinguish each object in the target image as an independent entity and determine the specific position and outline of each object in the image. The result of entity segmentation performed by the SAM model is relatively accurate, but the Mask information corresponding to the segmented third target object does not contain semantic information, such as clothing categories. For example Figure 3 The image shown is after entity segmentation using the SAM model. Different color masks represent different entity semantic contents, but lack specific natural language semantics.

[0055] Step S13: Obtain a target query request input by the user terminal, and generate a second mask set corresponding to a second target object corresponding to the target query request in the target image based on the target model; the second mask set includes second semantic mask information containing semantic information of the second target object.

[0056] In this embodiment, a target query request input by the user (i.e., text information in a preset data set) is obtained, and then a second mask set corresponding to the second target object corresponding to the target query request in the target image is generated based on the target model. Specifically, the target query request can be obtained through the target model, so that the target model determines the second target object and the corresponding target area corresponding to each target query request from the target image, and generates second semantic mask information corresponding to the second target object and containing semantic information of the second target object, so as to obtain a second mask set corresponding to each target query request.

[0057] For example, a text-guided SAM model is used to generate a second mask set corresponding to a second target object in a target image corresponding to a target query request. It should be noted that the text-guided SAM model supports user input of text information from a preset dataset, and the model then outputs corresponding mask information, thereby achieving image segmentation based on text prompts. However, due to the limited amount of text information in the preset dataset, the features and patterns learned by the SAM model during training are not rich enough. As a result, the mask information output by the trained text-guided SAM model is coarse, the segmentation boundaries are not precise, and the details are poorly represented, which cannot well meet the application scenarios with high requirements for fine segmentation results.

[0058] Step S14: Using a preset dynamic programming algorithm, determine the target mask information corresponding to the second semantic mask information in the second mask set from the first mask set, and determine the target semantic mask information based on the target mask information and the second semantic mask information, so as to use the target semantic mask information to segment the target entity image from the target image, and return the target entity image to the user end to respond to the target query request.

[0059] In this embodiment, a preset dynamic programming algorithm is used to determine the target mask information corresponding to the second semantic mask information in the second mask set from the first mask set, which can include: first determining the second semantic mask information corresponding to the target query request, and determining the first object area corresponding to the second semantic mask information; then selecting a mask information from the mask information that has not been selected in the current first mask set as the current mask information; determining the second object area corresponding to the current mask information, and merging the second object area corresponding to the current mask information with the merged object areas corresponding to all the mask information already existing in the target mask information combination to obtain the current merged object area; wherein , the initial target mask information combination is empty; calculating the current intersection-over-union (IoU) between the first object region and the currently merged object region, if the current IoU is not zero and the current IoU is greater than the target IoU between the first object region and the merged object regions corresponding to all mask information already existing in the target mask information combination, then adding the current mask information to the target mask information combination; jumping to the step of selecting a mask information from the mask information that has not been selected in the current first mask set as the current mask information, until the first mask set is traversed, and using the mask information in the target mask information combination as the target mask information corresponding to the second semantic mask information in the second mask set corresponding to the target query request.

[0060] Before determining target mask information corresponding to the second semantic mask information in the second mask set from the first mask set using a preset dynamic programming algorithm, the method may further include: downsampling the mask information in the first mask set and the second mask set based on a preset step size to obtain preprocessed first and second mask sets; the preset step size includes a preset horizontal step size and a preset vertical step size. In this way, downsampling can reduce the amount of mask information data, lower the computational complexity of subsequent processing, and at the same time, avoid overfitting to a certain extent, thereby improving processing efficiency.

[0061] It should be noted that different dynamic programming algorithms can be selected according to actual conditions to determine the target mask information corresponding to the second semantic mask information in the second mask set from the first mask set. Figure 4 The dynamic programming algorithm shown:

[0062] 1. Define a function to calculate IoU: The function CALCULATEIOU is used to calculate the IoU (Intersection over Union) between mask_a (i.e., the first mask set) and mask_b (i.e., the second mask set). The IoU value is obtained by calculating the intersection and union of mask_a and mask_b, and then dividing the number of pixels in the intersection by the number of pixels in the union.

[0063] 2. Initialization and preprocessing: Get the height h and width w of the reference mask mask_b, and calculate the step size step_h and step_w of sliding on the mask accordingly. The step size is calculated for the subsequent sampling operation of the mask; sample the input mask list masks_a, and generate a new mask list that is adapted to the size of the reference mask mask_b through the [tmp[::step_h, ::step_w] for tmp in masks_a] operation to facilitate subsequent calculations; create a zero matrix dp of size (n + 1) * (m + 1) to store intermediate results in the dynamic programming process, where n is the number of masks in masks_a and m is the number of masks to be selected; create a zero matrix mask_all with the same shape as mask_b to temporarily store mask information, and repeat it through the repeat(mask_all, m + 1) operation to adapt to subsequent calculations.

[0064] 3. Dynamic programming to calculate IoU: Use two nested loops to traverse the masks in masks_a, the number of masks to be selected (m), and each mask position j to be selected (from 1 to m). Specifically, for each current mask current_mask (taken from masks_a), calculate the IoU value iou_cur between the current mask and the reference mask; calculate the IoU value iou_top between the current mask and the selected mask combination (the selected mask combination here refers to the mask set that has been determined to be selected before the current loop); merge the current mask with the previous mask in the selected mask combination, calculate the IoU value iou_left between the merged mask and the reference mask, compare iou_cur, iou_top, and iou_left, and take the maximum value. Based on the corresponding maximum value, update the dp matrix and the mask information in mask_all.repeat. If the maximum value is iou_cur, the current mask is stored in mask_all.repeat[i][j]; if the maximum value is iou_top, mask_all.repeat[i][j] is kept as the previous value; if the maximum value is iou_left, the merged mask is stored in mask_all.repeat[i][j].

[0065] 4. Select the mask index: After the loop ends, check whether adjacent elements in the dp matrix are different by traversing backward from m to 1. If they are different, add the corresponding index i - 1 to the selected_indices list. This process determines the index of the final selected mask in the original masks_a based on the results of dynamic programming.

[0066] 5. Output results: According to the index in the selected_indices list, the corresponding mask is extracted from the original masks_a to form the selected_masks list, and the maximum IoU value max_iou of the whole process is recorded. Finally, selected_masks and max_iou are returned as the output of the algorithm.

[0067] Furthermore, the semantic information corresponding to the second semantic mask information can be used to remove the mask information that does not include semantic information in the target mask information to obtain new target mask information, and the semantic information corresponding to the second semantic mask information can be fused with the new target mask information to obtain target semantic mask information containing semantic information.

[0068] It is understandable that the first mask set is obtained by performing entity segmentation on the target image using a preset entity segmentation model, wherein the target mask information is usually finer on the segmentation boundary and can accurately delineate the outline of the object. The second semantic mask information is generated by the target model based on the user's query request. Although it has clear semantics, it may be insufficient in the accuracy of the segmentation boundary. Combining the two, the new target mask information can more accurately segment the target object, reduce segmentation errors, and improve the quality of the segmentation results. At the same time, the semantic information corresponding to the second semantic mask information is fused with the new target mask information to obtain the target semantic mask information, so that the target semantic mask information not only accurately divides the target area, but also has clear semantic labels, so that the segmentation results can better meet the user's semantic-based query needs, and facilitate the subsequent understanding, analysis and application of the segmented entity images, such as in image retrieval, intelligent monitoring and other scenarios.

[0069] For example Figure 5 As shown in , the target query request input by the user is to obtain specific clothing on the person (such as blue top, blue backpack, orange shorts and black shoes). The text-guided SAM model is used to generate a second mask set corresponding to the second target object corresponding to the target query request in the target image, as shown in Figure 5 In the yellow area, the entity segmentation results output by the second mask set and the SAM model will enter the "Dynamic Programming" module. Based on the dynamic programming algorithm, this information is analyzed and integrated, and according to certain rules and goals, the optimal segmentation scheme is screened and combined from the numerous entity segmentation results. The target entity image is segmented from the target image based on the optimal segmentation scheme, and the target entity image is returned to the user end to respond to the target query request. Figure 5 The purple area in the figure is the target entity image returned to the user.

[0070] As can be seen from the above, this embodiment selects a preset data set, obtains vector information corresponding to the text information in the preset data set based on a preset image recognition model, and uses the vector information to train the preset entity segmentation model to obtain a target model. Only a small amount of preset data is required, which reduces the dependence on large-scale labeled data and reduces the labeling cost. In addition, the training process is simple, without the need for complex solutions or the introduction of large-scale new data sets, which reduces training costs and improves training efficiency. On the other hand, a first mask set that does not contain semantic information corresponding to the target image in the preset data set is obtained using the preset entity segmentation model, and a second mask set containing semantic information corresponding to the target query request input by the user is obtained in the target image using the target model. A preset dynamic programming algorithm is used to determine the target mask information corresponding to the second mask set from the first mask set, and then the target semantic mask information is determined based on the target mask information and the second semantic mask information in the second mask set, so as to use the target semantic mask information to segment the target entity image from the target image. The target model obtains image semantic results and provides semantic information, while the preset entity segmentation model provides detailed mask information. Dynamic programming bridges the results of the target model and the preset entity segmentation model. The combination of the two ensures that the final segmentation result, namely the target semantic mask information, has precise boundaries and semantic labels. This allows for more accurate identification and segmentation of specific objects in vertical domains, meeting the needs of scenarios with high segmentation requirements and improving vertical performance. At the same time, it effectively integrates the zero-shot recognition capabilities of the preset image recognition model and the segmentation capabilities of the preset entity segmentation model, avoiding the drawbacks of simple fusion and improving the accuracy and efficiency of open vocabulary interactive segmentation and recognition.

[0071] See also Figure 6 As shown, the embodiment of the present application further discloses an entity segmentation device based on dynamic programming, comprising:

[0072] The model determination module 11 is configured to obtain a preset data set, input text information in the preset data set into a preset image recognition model to obtain vector information corresponding to the text information, and train a preset entity segmentation model based on the vector information to obtain a target model; the preset data set includes target images, text information corresponding to each target image, and first semantic mask information including semantic information of a first target object;

[0073] a first mask set acquisition module 12, configured to perform entity segmentation on each target image in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set; the first mask set includes mask information obtained after performing the entity segmentation on the target image and excluding semantic information;

[0074] a second mask set acquisition module 13 configured to acquire a target query request input by a user terminal and generate a second mask set corresponding to a second target object in the target image corresponding to the target query request based on the target model; the second mask set includes second semantic mask information containing semantic information of the second target object;

[0075] The image segmentation module 14 is used to use a preset dynamic programming algorithm to determine the target mask information corresponding to the second semantic mask information in the second mask set from the first mask set, and determine the target semantic mask information based on the target mask information and the second semantic mask information, so as to use the target semantic mask information to segment the target entity image from the target image, and return the target entity image to the user terminal to respond to the target query request.

[0076] As can be seen from the above, the present application selects a preset data set, obtains vector information corresponding to the text information in the preset data set based on a preset image recognition model, and uses the vector information to train the preset entity segmentation model to obtain a target model. Only a small amount of preset data is required, which reduces the dependence on large-scale labeled data and reduces the labeling cost. In addition, the training process is simple, without the need for complex solutions or the introduction of large-scale new data sets, which reduces training costs and improves training efficiency. On the other hand, the preset entity segmentation model is used to obtain a first mask set that does not contain semantic information corresponding to the target image in the preset data set, and the target model is used to obtain a second mask set that contains semantic information corresponding to the target query request input by the user in the target image, and a preset dynamic programming algorithm is used to determine the target mask information corresponding to the second mask set from the first mask set, and then the target semantic mask information is determined based on the target mask information and the second semantic mask information in the second mask set, so as to use the target semantic mask information to segment the target entity image from the target image. Among them, the target model obtains the image semantic results and provides semantic information, and the preset entity segmentation model provides detailed mask information. The results of the target model and the preset entity segmentation model are bridged through dynamic planning. The two are combined to make the final segmentation result, that is, the target semantic mask information, have precise boundaries and semantic labels. It can more accurately identify and segment specific objects in vertical fields, meet the needs of scenarios with high requirements for segmentation results, and improve vertical performance.

[0077] In some specific implementations, the model determination module 11 includes:

[0078] An information recognition unit, configured to fuse the vector information with an encoder in a preset entity segmentation model to obtain a text encoder, so that the preset entity segmentation model recognizes the acquired text information based on the text encoder after fusion with the preset entity segmentation model;

[0079] A model training unit is used to train a preset entity segmentation model including the text encoder based on the target images in the preset data set, the text information corresponding to each target image, and the first semantic mask information containing the semantic information of the first target object to obtain a target model; wherein the first target object is an object with semantic information in the preset data set.

[0080] In some specific implementations, the first mask set acquisition module 12 includes:

[0081] A first mask set determination unit is configured to perform entity segmentation on each of the target images in the preset data set based on the preset entity segmentation model to obtain a third target object, and determine mask information corresponding to each of the third target objects that does not contain semantic information to obtain a first mask set; wherein the third target objects include objects with semantic information and objects without semantic information.

[0082] In some specific implementations, the second mask set acquisition module 13 includes:

[0083] A second mask set determination unit is used to obtain the target query request through the target model, so that the target model determines the second target object and the corresponding target area corresponding to each target query request from the target image, and generates second semantic mask information corresponding to the second target object and containing semantic information of the second target object, so as to obtain a second mask set corresponding to each target query request.

[0084] In some specific implementations, the image segmentation module 14 includes:

[0085] a first region determining unit, configured to determine second semantic mask information corresponding to the target query request, and determine a first object region corresponding to the second semantic mask information;

[0086] a mask information determining unit, configured to select a piece of mask information from the unselected mask information in the first mask set as current mask information;

[0087] a second region determining unit, configured to determine a second object region corresponding to the current mask information, and merge the second object region corresponding to the current mask information with merged object regions corresponding to all existing mask information in the target mask information combination to obtain a current merged object region; wherein an initial target mask information combination is empty;

[0088] a mask information updating unit, configured to calculate a current intersection-of-union (IoU) between the first object region and the currently merged object region, and add the current mask information to the target mask information combination if the current IoU is not zero and is greater than a target IoU between the first object region and the merged object regions corresponding to all mask information already present in the target mask information combination;

[0089] A set traversal unit is used to jump to the step of selecting a mask information from the mask information that has not been selected in the current first mask set as the current mask information until the first mask set is traversed, and the mask information in the target mask information combination is used as the target mask information corresponding to the second semantic mask information in the second mask set corresponding to the target query request.

[0090] In some specific implementations, the image segmentation module 14 includes:

[0091] A mask information determination unit is used to use the semantic information corresponding to the second semantic mask information to remove the mask information that does not include the semantic information in the target mask information to obtain new target mask information, and to fuse the semantic information corresponding to the second semantic mask information with the new target mask information to obtain target semantic mask information containing semantic information.

[0092] In some specific implementations, the image segmentation module 14 further includes:

[0093] A sampling unit is configured to downsample the mask information in the first mask set and the second mask set based on a preset step size to obtain the preprocessed first mask set and the second mask set; the preset step size includes a preset horizontal step size and a preset vertical step size.

[0094] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0095] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the entity segmentation method based on dynamic programming disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0096] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0097] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0098] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the entity segmentation method based on dynamic programming performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0099] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned dynamic programming-based entity segmentation method. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0101] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0103] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0104] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for entity segmentation based on dynamic programming, characterized in that: include: Obtaining a preset data set, and inputting text information in the preset data set into a preset image recognition model to obtain vector information corresponding to the text information, and training a preset entity segmentation model based on the vector information to obtain a target model; The preset data set includes target images, text information corresponding to each target image, and first semantic mask information containing semantic information of a first target object; Performing entity segmentation on each of the target images in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set; the first mask set includes mask information that does not contain semantic information after performing the entity segmentation on the target image; Obtaining a target query request input by a user terminal, and generating a second mask set corresponding to a second target object corresponding to the target query request in the target image based on the target model; the second mask set includes second semantic mask information including semantic information of the second target object; Using a preset dynamic programming algorithm, target mask information corresponding to the second semantic mask information in the second mask set is determined from the first mask set, and target semantic mask information is determined based on the target mask information and the second semantic mask information, so as to use the target semantic mask information to segment a target entity image from the target image, and return the target entity image to the user end to respond to the target query request.

2. The entity segmentation method based on dynamic programming according to claim 1, characterized in that: The step of training a preset entity segmentation model based on the vector information to obtain a target model includes: Fusing the vector information with an encoder in a preset entity segmentation model to obtain a text encoder, so that the preset entity segmentation model recognizes the acquired text information based on the text encoder after fusion with itself; Based on the target images in the preset data set, the text information corresponding to each target image and the first semantic mask information containing the semantic information of the first target object, the preset entity segmentation model including the text encoder is trained to obtain a target model; wherein, the first target object is an object with semantic information in the preset data set.

3. The entity segmentation method based on dynamic programming according to claim 1, characterized in that: The performing entity segmentation on each target image in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set includes: Based on the preset entity segmentation model, each target image in the preset data set is entity segmented to obtain a third target object, and mask information that does not contain semantic information corresponding to each third target object is determined to obtain a first mask set; wherein the third target objects include objects with semantic information and objects without semantic information.

4. The entity segmentation method based on dynamic programming according to claim 1, characterized in that: Generating, based on the target model, a second mask set corresponding to a second target object corresponding to the target query request in the target image includes: The target query request is obtained through the target model, so that the target model determines the second target object and the corresponding target area corresponding to each target query request from the target image, and generates second semantic mask information corresponding to the second target object and containing semantic information of the second target object, so as to obtain a second mask set corresponding to each target query request.

5. The entity segmentation method based on dynamic programming according to any one of claims 1 to 4, characterized in that: The determining, from the first mask set, target mask information corresponding to the second semantic mask information in the second mask set by using a preset dynamic programming algorithm includes: Determining second semantic mask information corresponding to the target query request, and determining a first object area corresponding to the second semantic mask information; Selecting a mask information from the mask information that has not been selected in the first mask set as the current mask information; Determining a second object region corresponding to the current mask information, and merging the second object region corresponding to the current mask information with merged object regions corresponding to all existing mask information in the target mask information combination to obtain a current merged object region; wherein an initial target mask information combination is empty; calculating a current IoU between the first object region and the currently merged object region, and if the current IoU is not zero and is greater than a target IoU between the first object region and the merged object regions corresponding to all mask information already present in the target mask information combination, adding the current mask information to the target mask information combination; Jump to the step of selecting a mask information from the mask information that has not been selected in the current first mask set as the current mask information, until the first mask set is traversed, and the mask information in the target mask information combination is used as the target mask information corresponding to the second semantic mask information in the second mask set corresponding to the target query request.

6. The entity segmentation method based on dynamic programming according to claim 1, characterized in that: The determining target semantic mask information based on the target mask information and the second semantic mask information includes: Using the semantic information corresponding to the second semantic mask information, the mask information that does not include the semantic information in the target mask information is removed to obtain new target mask information, and the semantic information corresponding to the second semantic mask information is fused with the new target mask information to obtain target semantic mask information containing semantic information.

7. The entity segmentation method based on dynamic programming according to claim 1, characterized in that: Before determining, from the first mask set, target mask information corresponding to the second semantic mask information in the second mask set by using a preset dynamic programming algorithm, the method further includes: The mask information in the first mask set and the second mask set is downsampled based on a preset step size to obtain a preprocessed first mask set and a preprocessed second mask set; the preset step size includes a preset horizontal step size and a preset vertical step size.

8. An entity segmentation device based on dynamic programming, characterized in that: include: a model determination module, configured to obtain a preset data set, input text information in the preset data set into a preset image recognition model to obtain vector information corresponding to the text information, and train a preset entity segmentation model based on the vector information to obtain a target model; the preset data set includes target images, text information corresponding to each target image, and first semantic mask information including semantic information of a first target object; a first mask set acquisition module, configured to perform entity segmentation on each target image in the preset data set based on the preset entity segmentation model to obtain a corresponding first mask set; the first mask set includes mask information obtained after performing the entity segmentation on the target image and excluding semantic information; a second mask set acquisition module, configured to acquire a target query request input by a user terminal and generate a second mask set corresponding to a second target object corresponding to the target query request in the target image based on the target model; the second mask set includes second semantic mask information including semantic information of the second target object; An image segmentation module is configured to determine, from the first mask set, target mask information corresponding to the second semantic mask information in the second mask set using a preset dynamic programming algorithm, and determine target semantic mask information based on the target mask information and the second semantic mask information, so as to segment a target entity image from the target image using the target semantic mask information, and return the target entity image to the user terminal to respond to the target query request.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the entity segmentation method based on dynamic programming according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the entity segmentation method based on dynamic programming according to any one of claims 1 to 7.

Images