Picture segmentation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a picture segmentation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a to-be-segmented picture and natural language description information related to the to-be-segmented picture, carrying out the semantic analysis of the natural language description information, and obtaining a plurality of first semantic features; performing association expansion on each first semantic feature to generate a corresponding second semantic feature, and mapping each second semantic feature into a corresponding target visual feature; and sequentially carrying out pixel positioning and boundary calibration on each target visual feature in the to-be-segmented picture to generate a target picture segmentation mask, and carrying out segmentation processing on the to-be-segmented picture based on the target picture segmentation mask to obtain a segmentation result. And the pixel positioning range and the integrity and accuracy of image segmentation are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image segmentation method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, image segmentation, as a key technology in this field, has demonstrated significant application value in numerous areas. For example, in autonomous driving scenarios, image segmentation technology helps vehicles accurately identify roads, pedestrians, traffic signs, etc., ensuring driving safety; in medical image analysis, it can assist doctors in accurately segmenting lesions, aiding in disease diagnosis; and e-commerce platforms use it to remove backgrounds from product images and perform product cutouts, improving the product display effect.

[0003] Traditional image segmentation methods suffer from problems such as fuzzy semantic analysis of the target object and insufficient multi-scale segmentation accuracy in complex scenarios. This makes it difficult to capture complete target features, resulting in incomplete or incorrect segmentation and failing to guarantee the accuracy of image segmentation. Summary of the Invention

[0004] This application provides an image segmentation method, apparatus, device, and storage medium, which solves the problems of semantic ambiguity in the target segmentation object and insufficient multi-scale segmentation accuracy, which make it difficult to capture complete target features, easily leading to incomplete or missegmentation and compromising image segmentation accuracy. This solution improves the pixel localization range by analyzing and expanding natural language description information, thereby enhancing the completeness and accuracy of image segmentation. By sequentially performing pixel localization and boundary calibration on each target visual feature in the image to be segmented, the accuracy of image segmentation can be significantly improved.

[0005] In a first aspect, embodiments of this application provide an image segmentation method, comprising: Obtain the image to be segmented and the natural language description information associated with the image to be segmented, perform semantic parsing on the natural language description information, and obtain multiple first semantic features; Each of the first semantic features is associated and expanded to generate a corresponding second semantic feature, and each of the second semantic features is mapped to a corresponding target visual feature. In the image to be segmented, each of the target visual features is sequentially located at pixels and its boundaries are calibrated to generate a target image segmentation mask. Based on the target image segmentation mask, the image to be segmented is segmented to obtain the segmentation result.

[0006] Optionally, the step of associating and expanding each of the first semantic features to generate corresponding second semantic features includes: Each of the first semantic features is expanded in spatial dimension, attribute dimension, and semantic dimension to generate multiple extended features corresponding to each first semantic feature. The extended features are randomly combined, and the combination results are integrated into natural language to generate corresponding extended description information. Multiple second semantic features are determined from the multiple extended features based on the semantic distance between the extended description information and the natural language description information.

[0007] Optionally, determining multiple second semantic features among the multiple extended features based on the semantic distance between the extended description information and the natural language description information includes: Each first semantic feature corresponding to the natural language description information is matched with multiple extended features that make up each extended description information to obtain matching features and matching quantity; Calculate the semantic distance between each of the extended description information and the natural language description information based on the matching features and the number of matches; If the semantic distance is less than a preset semantic distance threshold, the corresponding extended description information is determined as the target extended description information, and the multiple extended features corresponding to the target extended description information are determined as the second semantic features.

[0008] Optionally, calculating the semantic distance between each of the extended description information and the natural language description information based on the matching features and the number of matches includes: Determine the weighting coefficient corresponding to the feature type to which the matching feature belongs; The semantic distance between each of the extended descriptive information and the natural language descriptive information is calculated based on the weighting coefficient and the number of matches. The feature types include core features, attribute features, and auxiliary features.

[0009] Optionally, the step of sequentially performing pixel localization and boundary calibration on each of the target visual features in the image to be segmented to generate a target image segmentation mask includes: Identify the seed pixels corresponding to each target visual feature in the image to be segmented, recursively judge the neighboring pixels based on the seed pixels and preset judgment rules, and determine multiple target pixels and the mask value of each target pixel based on the judgment results. An initial segmentation mask is generated based on each of the mask values. Boundary calibration is performed on the initial segmentation mask to generate a target image segmentation mask.

[0010] Optionally, the step of recursively judging neighboring pixels based on the seed pixel and a preset judgment rule, and determining multiple target pixels based on the judgment result, includes: Using the color and texture values ​​of the seed pixels as initial recursive judgment values, calculate the color difference and texture difference between every two adjacent pixels; If the color difference is less than a preset color threshold and the texture difference is less than a preset texture threshold, then two adjacent pixels are determined to be target pixels.

[0011] Optionally, the semantic parsing of the natural language description information yields multiple first semantic features, including: The natural language description information is parsed according to the grammatical structure to obtain multiple target features, and each target feature is semantically adapted to determine the adapted multiple target features as the first semantic feature.

[0012] In a second aspect, embodiments of this application provide an image segmentation apparatus, comprising: The semantic parsing module is used to obtain the image to be segmented and the natural language description information associated with the image to be segmented, and to perform semantic parsing on the natural language description information to obtain multiple first semantic features; The feature association extension module is used to associate and extend each of the first semantic features to generate the corresponding second semantic features; The feature mapping module is used to map each of the second semantic features to the corresponding target visual features; The segmentation mask generation module is used to sequentially perform pixel localization and boundary calibration on each of the target visual features in the image to be segmented, and generate a target image segmentation mask. The image segmentation module is used to segment the image to be segmented based on the target image segmentation mask to obtain the segmentation result.

[0013] In a third aspect, embodiments of this application provide an electronic device, the device comprising: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation method described in the first aspect.

[0014] In a fourth aspect, embodiments of this application provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image segmentation method as described in the first aspect.

[0015] This application embodiment obtains the image to be segmented and the natural language description information associated with it, performs semantic parsing on the natural language description information to obtain multiple first semantic features, expands each first semantic feature to generate a corresponding second semantic feature, and maps each second semantic feature to a corresponding target visual feature. In the image to be segmented, pixel localization and boundary calibration are performed sequentially on each target visual feature to generate a target image segmentation mask. Based on the target image segmentation mask, the image to be segmented is segmented to obtain the segmentation result. In the above scheme, the pixel localization range can be improved by analyzing and expanding the natural language description information, thereby improving the completeness and accuracy of image segmentation. By sequentially performing pixel localization and boundary calibration on each target visual feature in the image to be segmented, the image segmentation accuracy can be significantly improved. Attached Figure Description

[0016] Figure 1 This is a flowchart of an image segmentation method provided in an embodiment of this application; Figure 2 This is a flowchart of a second semantic feature generation method provided in an embodiment of this application; Figure 3 This is a flowchart of another second semantic feature determination method provided in the embodiments of this application; Figure 4 This is a flowchart of a target image segmentation mask generation method provided in an embodiment of this application; Figure 5 This is a schematic diagram of a segmentation mask provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0018] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0020] The image segmentation method, apparatus, device, and medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0021] The image segmentation method provided in this application is used in scenarios such as autonomous driving and medical image analysis, to help vehicles accurately identify roads, pedestrians, and traffic signs. Based on the above application scenarios, it is understood that the executing entity for each step can be a computer device. This computer device refers to any electronic device with data computing, processing, and storage capabilities, such as mobile phones, PCs (Personal Computers), tablets, and other terminal devices, or it can be a server or other devices. This application does not limit this specific device.

[0022] Figure 1 This is a flowchart of an image segmentation method provided in an embodiment of this application, such as... Figure 1 As shown, it includes: Step S101: Obtain the image to be segmented and the natural language description information associated with the image to be segmented, perform semantic parsing on the natural language description information, and obtain multiple first semantic features.

[0023] The image to be segmented can be any image requiring segmentation processing. This image can be downloaded from a location or uploaded by the user as needed, such as road images taken in autonomous driving scenarios or images of lesions in medical image analysis. Natural language description information can be textual descriptions related to the image to be segmented. For example, in autonomous driving scenarios, the natural language description information could be "identify roads, pedestrians, traffic signs, etc."; in medical image analysis, it could be "segment lesions"; and in e-commerce platforms, it could be "remove the background of the product image and extract the product." The first semantic feature is used to describe the descriptive information corresponding to the object to be segmented in the image, such as the shape information or position information of the object to be segmented within the image.

[0024] In one embodiment, the system receives a natural language description of the image to be segmented, which is being viewed by a user. This description is then semantically parsed, and the object to be segmented and descriptive features about that object are extracted from the natural language description based on the parsing results. For example, if the natural language description is "extract the woman in the image wearing a red dress and holding a coffee cup in her left hand," then semantic parsing of this description yields first semantic features such as "woman," "coffee cup," "red dress," and "left hand," providing crucial information for subsequent image segmentation.

[0025] Step S102: Perform association expansion on each first semantic feature to generate a corresponding second semantic feature, and map each second semantic feature to the corresponding target visual feature.

[0026] The second semantic feature refers to a feature similar to the first semantic feature obtained by further refining or expanding the first semantic feature. For example, if the first semantic feature is "skirt", the corresponding second semantic feature could be "dress", "skirt", or "top", etc. The target visual feature can be image features that can be understood and processed by a computer.

[0027] In one embodiment, the first semantic feature is expanded horizontally and vertically to obtain multiple corresponding second semantic features. Mapping each second semantic feature to a target visual feature converts these semantic features into image features that a computer can understand and process, for use in subsequent image segmentation. For example, if the second semantic feature is "red apple," the target visual features mapped to this second semantic feature are: color (RGB: 255, 50, 50), shape (circularity > 0.8), and texture (smoothness > 0.7). Horizontal expansion can refer to expansion from multiple dimensions such as associated scenes, associated attributes, and associated objects. For example, expanding from the associated scene dimension: morning running shoes, marathon running shoes, daily training running shoes; expanding from the associated attribute dimension: lightweight running shoes, shock-absorbing running shoes, breathable running shoes, etc. Vertical expansion refers to breaking down the core keyword's internal logical chain downwards, delving into more granular levels or more specific attributes. For example, running shoes → professional running shoes → marathon running shoes → top-tier carbon plate marathon running shoes, etc.

[0028] Step S103: In the image to be segmented, perform pixel localization and boundary calibration on each target visual feature in sequence to generate a target image segmentation mask. Based on the target image segmentation mask, perform segmentation processing on the image to be segmented to obtain the segmentation result.

[0029] Here, the image segmentation mask refers to a two-dimensional matrix with the same size as the image to be segmented, where each element value represents whether the corresponding pixel belongs to the target object region. In one embodiment, the pixel location of each target visual feature in the image to be segmented is determined, the image region composed of the pixels of each target visual feature is determined, and the contour of the image region is finely adjusted to obtain a precise image region contour. An image segmentation mask is generated for the target object region to which each element value in the precise image region contour belongs. Based on the image segmentation mask, the image to be segmented is precisely segmented to obtain a segmentation result containing only the target object. For example, if the target visual features corresponding to the second semantic feature "red apple" are color (RGB: 255, 50, 50), shape (circularity > 0.8), and texture (smoothness > 0.7), then all target pixels with RGB: 255, 50, 50, circularity > 0.8, and smoothness > 0.7 are determined in the image to be segmented. The region composed of all target pixels is determined as the target region, and the pixel value corresponding to each pixel in the target region is determined to be 1. The second semantic feature, "angry expression," corresponds to the target visual features of eyebrows (tilt angle > 30°), corners of the mouth (drooping angle > 0.5), and pupils (contraction rate > 0.6). Therefore, in the image to be segmented, all target pixels with eyebrow tilt angle > 30°, corners of the mouth drooping angle > 0.5, and pupil contraction rate > 0.6 are identified. The region formed by all these target pixels is defined as another target region, and the pixel value corresponding to each pixel within this target region is set to 2. The pixel values ​​corresponding to the pixels within each region can be randomly set and are not limited here, as long as the pixel values ​​within each target region are different, i.e., the pixel values ​​corresponding to each descriptive object are different. In another possible embodiment, if the second semantic feature "angry expression" describes an object "woman," and the second semantic feature "long black hair" also describes an object "woman," then the two second semantic features describe the same object, but the corresponding target regions are different: the target region for "angry expression" and the target region for "long black hair." However, in this case, the pixel values ​​of the target regions for "angry expression" and "long black hair" can be set to the same value. After generating the pixel values ​​of each target region, an image segmentation mask is generated based on each pixel value. The contour of the region to be segmented is determined based on the image segmentation mask, and the original contour is fitted by a mathematical curve. Redundant points are removed, and the contour is forced to conform to a smooth or regular shape, thereby generating an accurate region contour. Based on the accurate region contour, the image to be segmented is segmented to obtain the segmentation result.

[0030] This application embodiment obtains the image to be segmented and the natural language description information associated with it, performs semantic parsing on the natural language description information to obtain multiple first semantic features, expands each first semantic feature to generate a corresponding second semantic feature, and maps each second semantic feature to a corresponding target visual feature. In the image to be segmented, pixel localization and boundary calibration are performed sequentially on each target visual feature to generate a target image segmentation mask. Based on the target image segmentation mask, the image to be segmented is segmented to obtain the segmentation result. In the above scheme, the pixel localization range can be improved by analyzing and expanding the natural language description information, thereby improving the completeness and accuracy of image segmentation. By sequentially performing pixel localization and boundary calibration on each target visual feature in the image to be segmented, the image segmentation accuracy can be significantly improved.

[0031] In one embodiment, semantic parsing of natural language description information is performed to obtain multiple first semantic features, including: performing feature parsing of natural language description information according to grammatical structure to obtain multiple target features, performing semantic adaptation on each target feature, and determining the adapted multiple target features as first semantic features.

[0032] Feature parsing refers to analyzing natural language descriptions word by word or sentence by sentence to extract keywords or phrases relevant to image segmentation; these keywords or phrases are the target features. For example, in the natural language description "extracting the image of a woman wearing a red dress, holding a millet cup in her left hand, and talking on the phone," feature parsing can yield target features such as "woman," "red dress," "left hand," and "coffee cup." Semantic adaptation involves matching the semantics of the extracted target features with the content of the image to be segmented to determine whether these target features accurately describe the target object in the image. If the target feature matches the image content, it means that the target feature accurately describes the target object in the image and can be identified as the first semantic feature. The image content can be obtained through image recognition of the image to be segmented. For example, if the image content includes "a woman wearing a yellow dress and holding a coffee cup, and there are no red elements in the image," then the feature "red dress" is considered mismatched with the image and cannot accurately describe the target object, therefore it cannot be used as the first semantic feature; the remaining features can be identified as the first semantic features.

[0033] This application embodiment performs feature parsing on natural language description information based on grammatical structure to obtain multiple target features, and performs semantic adaptation on each target feature, determining the adapted multiple target features as the first semantic feature. In the above scheme, the method of feature parsing and semantic adaptation of natural language description information can ensure accurate understanding and parsing of natural language description information, providing accurate key information for subsequent image segmentation processing.

[0034] Figure 2 This is a flowchart of a second semantic feature generation method provided in an embodiment of this application, such as... Figure 2 As shown, it includes: Step S1021: Expand the spatial dimension, attribute dimension and semantic dimension of each first semantic feature to generate multiple extended features corresponding to each first semantic feature.

[0035] Spatial dimension expansion can involve further description of the spatial attributes of the object to be segmented in the image, such as its position, size, or shape. Attribute dimension expansion can involve detailed description of attributes such as color, texture, and brightness of the object to be segmented. Semantic dimension expansion can involve description of the relationship between the object to be segmented and other objects or the background, such as "located on the right side of the road" or "near a traffic sign."

[0036] In one embodiment, the first semantic feature is expanded in three dimensions: spatial, attribute, and semantic, generating expanded features for each dimension. For example, when expanding the spatial dimension, information such as the relative position, absolute position, size ratio, and relative distance to surrounding objects of the object to be segmented in the image can be analyzed. For the first semantic feature "pedestrian," the spatial dimension expansion can include the pedestrian's specific location in the image, the ratio of the pedestrian's height to the image height, and the distance of the pedestrian from the lane lines. This information helps to more accurately locate the pedestrian region during image segmentation. When expanding the attribute dimension, the focus is on a detailed description of the inherent attributes of the object to be segmented. For example, if the first semantic feature is "traffic sign," the attribute dimension expansion can include the sign's color, shape, and whether it contains text or patterns. This attribute information can enhance the image segmentation algorithm's ability to recognize traffic signs and improve segmentation accuracy. When expanding the semantic dimension, attention is paid to the semantic relationships between the object to be segmented and other objects or the background. For example, for the first semantic feature "road," the semantic dimension expansion can include the relationship between the road and surrounding buildings, and the relationship between the road and traffic signs. This information helps to understand the context of roads in an image during image segmentation, thus enabling more accurate segmentation of road regions.

[0037] Step S1022: Randomly combine the extended features, integrate the combination results using natural language, and generate corresponding extended description information.

[0038] Natural language integration refers to combining various extended features into complete sentences or phrases in a manner that conforms to the rules of natural language grammar, so as to generate extended descriptive information that can accurately describe the features of the object to be segmented. Extended descriptive information can refer to a description containing more detailed information obtained after refining or expanding the first semantic features. For example, if the natural language descriptive information is: extracting "woman wearing a red dress and holding a coffee cup in her left hand," the corresponding first semantic features are "woman," "red dress," "left hand," and "coffee cup." "Woman" can be expanded to "elderly," "man," "child," etc.; "left hand" can be expanded to "right hand," "both hands," "raised hand," etc.; "coffee cup" can be expanded to "stemmed glass," "tea cup," "wine glass," etc.; and "red dress" can be expanded to "yellow dress," "blue skirt," etc. In one embodiment, the extended features are randomly combined to form multiple possible feature combination schemes, and natural language integration is performed on each combination scheme to generate corresponding extended descriptive information. For example, combining "child", "yellow dress", "left hand" and "coffee cup" and integrating them into an extended natural language description could be: "Extract a child wearing a yellow dress and holding a coffee cup in their left hand".

[0039] Step S1023: Determine multiple second semantic features among multiple extended features based on the semantic distance between the extended description information and the natural language description information.

[0040] In one embodiment, the semantic similarity, or semantic distance, between the extended description information and the natural language description information is calculated. The semantic distance can be obtained by calculating the cosine similarity of word vectors or using a semantic matching model. The cosine similarity of word vectors refers to converting the extended description information and the natural language description information into word vectors and then calculating the cosine value between them; a larger cosine value indicates greater similarity. The semantic matching model refers to a model trained using deep learning and other techniques that can determine the semantic relationship between two texts. The calculated semantic distance is compared with a preset threshold. If the semantic distance is less than or equal to the threshold, the extended description information is considered sufficiently close or similar to the natural language description information, and its corresponding extended feature is determined as the second semantic feature. From multiple extended features, the second semantic feature that best matches or is most relevant to the original natural language description information is selected, providing more accurate and specific semantic guidance for subsequent image segmentation.

[0041] This application embodiment generates multiple extended features corresponding to each first semantic feature by expanding the spatial dimension, attribute dimension, and semantic dimension of each first semantic feature. The extended features are then randomly combined, and the combined results are integrated using natural language to generate corresponding extended descriptive information. Multiple second semantic features are determined from the multiple extended features based on the semantic distance between the extended descriptive information and the natural language descriptive information. In the above scheme, by randomly combining and integrating the extended features into extended descriptive statements, multiple extended descriptive information describing different aspects of the object to be segmented can be obtained, providing richer semantic information for subsequent image segmentation. By calculating the semantic distance between the extended descriptive statements and the natural language descriptive statements, the second semantic features that best match or are most relevant to the original natural language descriptive information are selected from the multiple extended features, providing more accurate and specific semantic guidance for subsequent image segmentation.

[0042] Figure 3 This is a flowchart of another second semantic feature determination method provided in the embodiments of this application, such as... Figure 3 As shown, it includes: Step S10231: Match each first semantic feature corresponding to the natural language description information with multiple extended features that make up each extended description information to obtain the matching features and the number of matches.

[0043] Step S10232: Calculate the semantic distance between each extended description information and the natural language description information based on the matching features and the number of matches.

[0044] Step S10233: When the semantic distance is less than the preset semantic distance threshold, determine the corresponding extended description information as the target extended description information, and determine the multiple extended features corresponding to the target extended description information as the second semantic features.

[0045] In one embodiment, each first semantic feature corresponding to the natural language description information is considered a query word, and the multiple extended features constituting each extended description information are considered candidate words. The query word is matched with each candidate word. If two words match, the first semantic feature is determined as a matching feature, and the number of matches for the matching features contained in each extended description information is determined. For example, if the first semantic features are "woman", "red dress", "left hand", and "coffee cup", and the extended features are "child", "red dress", "left hand", and "water cup", then the number of matches for the matching features "red dress" and "left hand" is 2.

[0046] The target weight coefficient is determined based on the objects associated with the matching features. These objects can be the subject or object modified by the matching features, such as "woman" or "coffee cup." Based on the semantics of the natural language description, the subject is "woman," and the object is "coffee cup." After determining the subject and object associated with the matching features, the target weight coefficient corresponding to the matching features of the associated subject is determined, for example, 0.6, and the target weight coefficient corresponding to the matching features of the associated object is determined, for example, 0.4. The ratio of the number of matches to the total number of extended features in the extended description is also determined, for example, 1 / 2. The semantic distance is calculated based on the ratio of the target weight coefficient corresponding to each matching feature to the number of features. According to the example above, "red dress" and "left hand" are both matching features used to modify "woman." Therefore, the target weight coefficient corresponding to "red dress" is 0.6, and the target weight coefficient corresponding to "left hand" is also 0.6. Therefore, the total target weight coefficient is 1 / 2 * (1 + 0.6 + 0.6) = 1.1.

[0047] The calculated semantic distance is compared with a preset semantic distance threshold. If the semantic distance is less than the preset threshold, the extended description information is considered sufficiently close or similar to the natural language description information, and it is identified as the target extended description information. Multiple extended features corresponding to the target extended description information are then identified as second semantic features. The preset semantic distance threshold can be set according to actual needs to ensure that the selected second semantic features have a high degree of matching and relevance to the natural language description information.

[0048] This application embodiment matches each first semantic feature corresponding to the natural language description information with multiple extended features constituting each extended description information to obtain matching features and matching quantity; calculates the semantic distance between each extended description information and the natural language description information based on the matching features and matching quantity; if the semantic distance is less than a preset semantic distance threshold, the corresponding extended description information is determined as the target extended description information, and the multiple extended features corresponding to the target extended description information are determined as second semantic features. In the above scheme, the degree of matching between the extended description information and the natural language description information is evaluated by calculating the matching features and matching quantity, thereby determining the extended description information that is closest to or similar to the target description information, which improves the accuracy and reliability of the determination of the second semantic features.

[0049] In one embodiment, calculating the semantic distance between each extended descriptive information and the natural language descriptive information based on the matching features and the number of matches includes: determining the weighting coefficient corresponding to the feature type to which the matching features belong; and calculating the semantic distance between each extended descriptive information and the natural language descriptive information based on the weighting coefficient and the number of matches. The feature types include core features, attribute features, and auxiliary features.

[0050] Core features refer to the main description of the object to be segmented, such as the object itself, like "cat" or "dog." Attribute features refer to the object's inherent attributes such as color, shape, and texture, such as "gray cat" or "round object." Auxiliary features refer to additional information such as the object's location and relationship with other objects, such as "cat in the center of the image" or "book on the table." Different feature types may have different degrees of influence on semantic distance, so different weighting coefficients can be set for different types of features. For example, the weighting coefficient for core features can be set to a higher value because core features directly determine the basic category of the object to be segmented; the weighting coefficient for attribute features can be set to a medium value because attribute features can further refine the description of the object to be segmented; and the weighting coefficient for auxiliary features can be set to a lower value because although auxiliary features provide additional contextual information, they have a smaller impact on the basic category of the object to be segmented. After determining the weighting coefficients, the semantic distance between each extended descriptive information and the natural language descriptive information can be calculated based on the weighting coefficients and the number of matches. For example, the weighting coefficient of each feature type is multiplied by the corresponding number of matches to obtain the weighted number of matches for that feature type. The weighted number of matches for each feature type is then summed to obtain the semantic distance value corresponding to the extended descriptive information. The smaller the semantic distance value, the higher the degree of matching between the extended descriptive information and the natural language descriptive information.

[0051] This application embodiment determines the weighting coefficient corresponding to the feature type to which the matching feature belongs; and calculates the semantic distance between each extended description and the natural language description based on the weighting coefficient and the number of matches. In the above scheme, by considering the weighting coefficients of different feature types, the degree of matching between the extended description and the natural language description can be evaluated more accurately, thereby improving the accuracy and reliability of the determination of the second semantic feature.

[0052] Figure 4 This is a flowchart of a target image segmentation mask generation method provided in an embodiment of this application, such as... Figure 4 As shown, it includes: Step S1031: Identify the seed pixels corresponding to the visual features of each target in the image to be segmented, recursively judge the neighboring pixels based on the seed pixels and preset judgment rules, and determine multiple target pixels and the mask value of each target pixel based on the judgment results.

[0053] Step S1032: Generate an initial segmentation mask based on each mask value, perform boundary calibration on the initial segmentation mask, and generate a target image segmentation mask.

[0054] Seed pixels can be pixels in the image to be segmented that have clear semantic features and can represent the visual features of the target. Preset judgment rules can be defined based on pixel attributes such as color, brightness, and texture, and are used to determine whether neighboring pixels match the visual features of the target.

[0055] Figure 5 This is a schematic diagram of a segmentation mask provided in an embodiment of this application, such as... Figure 5 As shown, seed pixels with clear semantic features and representative visual features of the target are identified in the image to be segmented. Starting from the seed pixels, adjacent pixels are judged sequentially according to preset judgment rules. If adjacent pixels meet the judgment rules, they are identified as target pixels and assigned corresponding mask values. If adjacent pixels have the same color value, both adjacent pixels are identified as target pixels. The mask value can be used to identify whether a pixel belongs to the target region. For example, the pixel mask value of the first target region corresponding to one target visual feature can be set to 1, the pixel mask value of the second target region corresponding to another target visual feature can be set to 2, and the pixel mask value of non-target regions can be set to 0. After determining multiple target pixels and their mask values, an initial segmentation mask is generated based on these mask values. The initial segmentation mask is a binary image with the same size as the image to be segmented, where the pixel value of the first target region is 1, the pixel value of the second target region is 2, and the pixel value of non-target regions is 0. However, the initial segmentation mask may have inaccurate boundaries. Therefore, it is necessary to smooth the boundaries of the initial segmentation mask, remove burrs, or fill holes to generate a more accurate target image segmentation mask. The target image segmentation mask can then guide subsequent image segmentation processing to ensure the accuracy and completeness of the segmentation results.

[0056] This application embodiment identifies seed pixels corresponding to the visual features of each target in the image to be segmented. Based on the seed pixels and preset judgment rules, it recursively judges neighboring pixels, determining multiple target pixels and their mask values ​​based on the judgment results. An initial segmentation mask is generated based on these mask values, and boundary calibration is performed on the initial segmentation mask to generate the target image segmentation mask. This scheme effectively identifies and segments the target region in the image to be segmented, improving the accuracy and efficiency of image segmentation. Through recursive judgment of seed pixels, target pixels and their mask values ​​can be accurately determined, thereby generating the initial segmentation mask. Boundary calibration of the initial segmentation mask further optimizes the segmentation results, making the target image segmentation mask more accurate and smooth, providing strong support for subsequent image analysis and processing.

[0057] In one embodiment, recursively judging neighboring pixels based on seed pixels and preset judgment rules, and determining multiple target pixels based on the judgment results, includes: using the color value and texture value of the seed pixel as initial recursive judgment values, calculating the color difference and texture difference between every two adjacent pixels; and determining that both adjacent pixels are target pixels when the color difference is less than a preset color threshold and the texture difference is less than a preset texture threshold.

[0058] Color values ​​refer to the component values ​​of a pixel in a specific color space, such as the red, green, and blue components in the RGB color space. Texture values, on the other hand, refer to the texture features of the region where the pixel is located, which can be obtained by calculating the statistical characteristics of pixel values ​​within a certain range around the pixel.

[0059] In one embodiment, the color and texture values ​​of the seed pixel are used as initial recursive judgment values, serving as a reference standard for subsequent judgment of neighboring pixels. The color and texture differences between any two adjacent pixels are calculated. If the color difference between adjacent pixels is less than a preset color threshold and the texture difference is less than a preset texture threshold, then the adjacent pixel is considered sufficiently close or similar to the target pixel in color and texture, and can be identified as the target pixel. This method is used to progressively determine all pixels that match the target visual features, i.e., the target pixels. The preset color and texture thresholds can be set according to actual needs to ensure that the selected target pixels accurately represent the target visual features in the image to be segmented.

[0060] This application embodiment uses the color and texture values ​​of seed pixels as initial recursive judgment values ​​to calculate the color difference and texture difference between every two adjacent pixels. If the color difference is less than a preset color threshold and the texture difference is less than a preset texture threshold, then both adjacent pixels are determined to be target pixels. This scheme comprehensively considers the color and texture features of pixels, more accurately determining whether neighboring pixels match the target visual features, thereby improving the accuracy and reliability of target pixel recognition.

[0061] Figure 6 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application, as shown below. Figure 6 As shown, it includes: The semantic parsing module 21 is used to obtain the image to be segmented and the natural language description information associated with the image to be segmented, and to perform semantic parsing on the natural language description information to obtain multiple first semantic features; The feature association extension module 22 is used to perform association extension on each of the first semantic features to generate the corresponding second semantic features; Feature mapping module 23 is used to map each of the second semantic features to the corresponding target visual features; The segmentation mask generation module 24 is used to sequentially perform pixel localization and boundary calibration on each of the target visual features in the image to be segmented, and generate a target image segmentation mask. Image segmentation module 25 is used to segment the image to be segmented based on the target image segmentation mask to obtain the segmentation result.

[0062] This application embodiment obtains the image to be segmented and the natural language description information associated with it, performs semantic parsing on the natural language description information to obtain multiple first semantic features, expands each first semantic feature to generate a corresponding second semantic feature, and maps each second semantic feature to a corresponding target visual feature. In the image to be segmented, pixel localization and boundary calibration are performed sequentially on each target visual feature to generate a target image segmentation mask. Based on the target image segmentation mask, the image to be segmented is segmented to obtain the segmentation result. In the above scheme, the pixel localization range can be improved by analyzing and expanding the natural language description information, thereby improving the completeness and accuracy of image segmentation. By sequentially performing pixel localization and boundary calibration on each target visual feature in the image to be segmented, the image segmentation accuracy can be significantly improved.

[0063] In one possible embodiment, the feature association extension module 22 is specifically used for: Each of the first semantic features is expanded in spatial dimension, attribute dimension, and semantic dimension to generate multiple extended features corresponding to each first semantic feature. The extended features are randomly combined, and the combination results are integrated into natural language to generate corresponding extended description information. Multiple second semantic features are determined from the multiple extended features based on the semantic distance between the extended description information and the natural language description information.

[0064] In one possible embodiment, the feature association extension module 22 is specifically used for: Each first semantic feature corresponding to the natural language description information is matched with multiple extended features that make up each extended description information to obtain matching features and matching quantity; Calculate the semantic distance between each of the extended description information and the natural language description information based on the matching features and the number of matches; If the semantic distance is less than a preset semantic distance threshold, the corresponding extended description information is determined as the target extended description information, and the multiple extended features corresponding to the target extended description information are determined as the second semantic features.

[0065] In one possible embodiment, the feature association extension module 22 is specifically used for: Determine the weighting coefficient corresponding to the feature type to which the matching feature belongs; The semantic distance between each of the extended descriptive information and the natural language descriptive information is calculated based on the weighting coefficient and the number of matches. The feature types include core features, attribute features, and auxiliary features.

[0066] In one possible embodiment, the segmentation mask generation module 24 is specifically used for: Identify the seed pixels corresponding to each target visual feature in the image to be segmented, recursively judge the neighboring pixels based on the seed pixels and preset judgment rules, and determine multiple target pixels and the mask value of each target pixel based on the judgment results. An initial segmentation mask is generated based on each of the mask values. Boundary calibration is performed on the initial segmentation mask to generate a target image segmentation mask.

[0067] In one possible embodiment, the segmentation mask generation module 24 is specifically used for: Using the color and texture values ​​of the seed pixels as initial recursive judgment values, calculate the color difference and texture difference between every two adjacent pixels; If the color difference is less than a preset color threshold and the texture difference is less than a preset texture threshold, then two adjacent pixels are determined to be target pixels.

[0068] In one possible embodiment, the semantic parsing module 21 is specifically used for: The natural language description information is parsed according to the grammatical structure to obtain multiple target features, and each target feature is semantically adapted to determine the adapted multiple target features as the first semantic feature.

[0069] This application also provides an electronic device that can integrate an image segmentation apparatus provided in this application. Figure 7 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application, with reference to... Figure 7 The image segmentation device includes: an input device 33, an output device 34, a memory 32, and one or more processors 31; the memory 32 is used to store one or more programs; when one or more programs are executed by one or more processors 31, the one or more processors 31 implement the image segmentation method provided in the above embodiments. The input device 33, output device 34, memory 32, and processors 31 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0070] The memory 32, as a computing device readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the image segmentation method provided in any embodiment of this application. The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include memory remotely located relative to the processor 31, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0071] Input device 33 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 34 may include display devices such as a display screen.

[0072] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, thereby realizing the above-mentioned image segmentation method.

[0073] The image segmentation apparatus, device, and computer provided above can be used to execute the image segmentation method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0074] This application embodiment also provides a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image segmentation method provided in the above embodiment, the image segmentation method including: The image to be segmented and the natural language description information associated with the image to be segmented are obtained. The natural language description information is semantically parsed to obtain multiple first semantic features. Each first semantic feature is associated and expanded to generate a corresponding second semantic feature, and each second semantic feature is mapped to a corresponding target visual feature. In the image to be segmented, the visual features of each target are sequentially located at pixels and their boundaries are calibrated to generate a target image segmentation mask. Based on the target image segmentation mask, the image to be segmented is processed to obtain the segmentation result.

[0075] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0076] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the image segmentation method described above, but can also perform related operations in the image segmentation method provided in any embodiment of this application.

[0077] The image segmentation apparatus, device, and storage medium provided in the above embodiments can execute the image segmentation method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the image segmentation method provided in any embodiment of this application.

[0078] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. An image segmentation method, characterized in that, include: Obtain the image to be segmented and the natural language description information associated with the image to be segmented, perform semantic parsing on the natural language description information, and obtain multiple first semantic features; Each of the first semantic features is associated and expanded to generate a corresponding second semantic feature, and each of the second semantic features is mapped to a corresponding target visual feature. In the image to be segmented, each of the target visual features is sequentially located at pixels and its boundaries are calibrated to generate a target image segmentation mask. Based on the target image segmentation mask, the image to be segmented is segmented to obtain the segmentation result.

2. The image segmentation method according to claim 1, characterized in that, The step of associating and expanding each of the first semantic features to generate corresponding second semantic features includes: Each of the first semantic features is expanded in spatial dimension, attribute dimension, and semantic dimension to generate multiple extended features corresponding to each first semantic feature. The extended features are randomly combined, and the combination results are integrated into natural language to generate corresponding extended description information. Multiple second semantic features are determined from the multiple extended features based on the semantic distance between the extended description information and the natural language description information.

3. The image segmentation method according to claim 2, characterized in that, The step of determining multiple second semantic features among the multiple extended features based on the semantic distance between the extended description information and the natural language description information includes: Each first semantic feature corresponding to the natural language description information is matched with multiple extended features that make up each extended description information to obtain matching features and matching quantity; Calculate the semantic distance between each of the extended description information and the natural language description information based on the matching features and the number of matches; If the semantic distance is less than a preset semantic distance threshold, the corresponding extended description information is determined as the target extended description information, and the multiple extended features corresponding to the target extended description information are determined as the second semantic features.

4. The image segmentation method according to claim 3, characterized in that, The step of calculating the semantic distance between each of the extended description information and the natural language description information based on the matching features and the number of matches includes: Determine the weighting coefficients corresponding to the feature type to which the matching feature belongs; The semantic distance between each of the extended descriptive information and the natural language descriptive information is calculated based on the weighting coefficient and the number of matches. The feature types include core features, attribute features, and auxiliary features.

5. The image segmentation method according to claim 1, characterized in that, The step of sequentially locating and calibrating the pixels of each target visual feature in the image to be segmented to generate a target image segmentation mask includes: Identify the seed pixels corresponding to each target visual feature in the image to be segmented, recursively judge the neighboring pixels based on the seed pixels and preset judgment rules, and determine multiple target pixels and the mask value of each target pixel based on the judgment results. An initial segmentation mask is generated based on each of the mask values. Boundary calibration is performed on the initial segmentation mask to generate a target image segmentation mask.

6. The image segmentation method according to claim 5, characterized in that, The recursive judgment of neighboring pixels based on the seed pixel and preset judgment rules, and the determination of multiple target pixels based on the judgment results, includes: Using the color and texture values ​​of the seed pixels as initial recursive judgment values, calculate the color difference and texture difference between every two adjacent pixels; If the color difference is less than a preset color threshold and the texture difference is less than a preset texture threshold, then two adjacent pixels are determined to be target pixels.

7. The image segmentation method according to claim 1, characterized in that, The semantic parsing of the natural language description information yields multiple first semantic features, including: The natural language description information is parsed according to the grammatical structure to obtain multiple target features, and each target feature is semantically adapted to determine the adapted multiple target features as the first semantic feature.

8. An image segmentation device, characterized in that, include: The semantic parsing module is used to obtain the image to be segmented and the natural language description information associated with the image to be segmented, and to perform semantic parsing on the natural language description information to obtain multiple first semantic features; The feature association extension module is used to associate and extend each of the first semantic features to generate the corresponding second semantic features; The feature mapping module is used to map each of the second semantic features to the corresponding target visual features; The segmentation mask generation module is used to sequentially perform pixel localization and boundary calibration on each of the target visual features in the image to be segmented, and generate a target image segmentation mask. The image segmentation module is used to segment the image to be segmented based on the target image segmentation mask to obtain the segmentation result.

9. An electronic device, the device comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image segmentation method as described in any one of claims 1-7.

10. A storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image segmentation method as described in any one of claims 1-7.

Images