Image segmentation model training method, image segmentation method and device
By labeling and shifting the single image and the training target image segmentation model, it solves the problem of predicting and segmentation of the obscure target with the difficulty of using a single image in the existing technology, and achieves efficient and accurate image segmentation effects.
Patent Information
- Application Number
- CN202411489875.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2025-01-24
AI Technical Summary
The prior art is difficult to accurately predict and segment the obstructed target using a single image, and it requires background reconstruction of pictures at multiple viewpoints or time points, which is inefficient, high cost and large errors.
By obtaining the sample image and marking the first background category object, target category object, and blocking category objects, determine the translation image peaceful migration area, and use translation images and real marking training target image segmentation models to output the image predictive segmentation results of the image to be divided.
It achieves accurate prediction and segmentation of occluded targets using a single image, improves the accuracy and efficiency of image segmentation, and reduces resource consumption and manual annotation costs.
Smart Images
Figure CN119360024A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to an image segmentation model training method, an image segmentation method, an apparatus, an electronic device and a storage medium. Background Art
[0002] With the advancement of science and technology, in more and more urban management scenarios, the types of objects in the surveillance images are increasing, so more abundant information assistance is needed to accurately warn urban management events, such as character information and background information in surveillance images. At present, image segmentation algorithms have made great progress in various fields, and various segmentation models have been proposed by scholars, so that target objects in images can be identified and segmented. However, in some scenarios, not only the visible target objects in the image need to be accurately segmented, but also the background object information behind the visible target objects needs to be predicted and completed, so as to provide complete environmental information to assist in judging the scene corresponding to the image content. However, in related technologies, when the target object is blocked, multiple pictures from multiple viewpoints or multiple time points need to be used to reconstruct the background in order to perform image prediction and segmentation, and manual target classification and labeling are required, which is inefficient, costly and error-prone.
[0003] Currently, no effective solution has been proposed for the problem of how to use a single image to predict and segment occluded targets in related technologies. Summary of the invention
[0004] The embodiments of the present application provide an image segmentation model training method, an image segmentation method, a device, an electronic device and a storage medium to at least solve the problem in the related art of how to use a single image to predict and segment an occluded target.
[0005] In a first aspect, an embodiment of the present application provides an image segmentation model training method.
[0006] In some embodiments, the image segmentation model training method includes:
[0007] Acquire a sample image, and mark a first background category object, a target category object, and an occlusion category object in the sample image;
[0008] Determine a translation image according to the first background category object, the target category object, and the occlusion category object;
[0009] Determine a translation region according to the translation image, determine a category object corresponding to the translation region in the translation image as a first true annotation, and determine a category object corresponding to the translation region in the sample image as a second true annotation;
[0010] The translated image, the first true annotation and the second true annotation are input into an original training model for training, and the trained original training model is determined as a target image segmentation model.
[0011] In some embodiments, determining the translation image according to the first background category object, the target category object and the occlusion category object comprises:
[0012] Determine that the image regions corresponding to the first background category object and the target category object are known type regions, and the image region corresponding to the occlusion category object is an occlusion region;
[0013] The occlusion category object is translated to the known type region, and the occlusion region is marked as a second background category object to determine a translated image.
[0014] In some embodiments, determining the translation region according to the translation image, determining the category object corresponding to the translation region in the translation image as the first real annotation, and determining the category object corresponding to the translation region in the sample image as the second real annotation comprises:
[0015] Determine, in the translated image, an area corresponding to the translated occlusion category object as the translated area;
[0016] A first real annotation and a second real annotation corresponding to the translation region are determined, wherein the first real annotation is the occlusion category object, and the second real annotation is the first background category object or the target category object.
[0017] In some embodiments, determining the first real annotation and the second real annotation corresponding to the translation area, wherein the first real annotation is the occlusion category object, and the second real annotation is the first background category object or the target category object, comprises:
[0018] In the case where the second real annotation is the target category object, the target category object may include one or more category objects;
[0019] In the case that the target category object includes objects of multiple categories, it is determined that the second real annotation corresponding to the translation area is one or more of the multiple target category objects.
[0020] On the other hand, the embodiment of the present application provides an image segmentation model training method, comprising:
[0021] Acquire a sample image, and mark a first background category object, a target category object, and an occlusion category object in the sample image, wherein the target category object is a sidewalk and / or a roadway, the occlusion category object is a person, and the first background category object is an object other than the target category object and the occlusion category object in the sample image;
[0022] Determine that the image area corresponding to the first background category object and the sidewalk and / or roadway is a known type area, and the image area corresponding to the person is an occluded area;
[0023] The person is translated to the known type area, and the blocked area is marked as a second background category object to determine a translated image;
[0024] Determining, in the translated image, an area corresponding to the translated person as a translated area;
[0025] Determine a first real annotation and a second real annotation corresponding to the translation area, wherein the first real annotation is the person, and the second real annotation is one or more of the first background category object, the sidewalk, and the roadway;
[0026] The translated image, the first true annotation and the second true annotation are input into an original training model for training, and the trained original training model is determined as a target image segmentation model.
[0027] In a second aspect, an embodiment of the present application provides an image segmentation method.
[0028] In some embodiments, the image segmentation method includes:
[0029] Acquire an image to be segmented, and input the image to be segmented into a target image segmentation model trained by the image segmentation model training method described in the first aspect above;
[0030] The target image segmentation model is used to output the image prediction segmentation result corresponding to the image to be segmented.
[0031] In some embodiments, the target image segmentation model includes a backbone network, a decoding structure network, a first graph convolutional neural network, and a second graph convolutional neural network, and outputting the image prediction segmentation result corresponding to the image to be segmented through the target image segmentation model includes:
[0032] Acquire an image to be segmented, and input the image to be segmented into the backbone network to obtain image spatial features of the image to be segmented;
[0033] Inputting the image spatial features into the decoding structure network to obtain pixel decoding features and global feature information;
[0034] Inputting the pixel decoding feature and the global feature information into the first graph convolutional neural network to obtain an occlusion category feature;
[0035] The occlusion category feature and the global feature information are input into the second graph convolutional neural network to obtain the image prediction segmentation result.
[0036] In some embodiments, inputting the pixel decoding feature and the global feature information into a first graph convolutional neural network to obtain the occlusion category feature includes:
[0037] Determine a first background category object, a target category object, and an occlusion category object in the image to be segmented;
[0038] Determine a first input feature according to the pixel decoding feature and the global feature information;
[0039] The first graph convolutional neural network performs feature extraction on the occlusion category object according to the first input feature to obtain an occlusion category feature.
[0040] In some embodiments, inputting the occlusion category feature and the global feature information into a second graph convolutional neural network to obtain an image prediction segmentation result includes:
[0041] Determining a second input feature according to the occlusion category feature and the global feature information;
[0042] The second graph convolutional neural network performs feature extraction on the first background category object and the target category object according to the second input feature to obtain background category features and target category features;
[0043] The background category features and the target category features are decoded to determine the image prediction segmentation result.
[0044] In a third aspect, an embodiment of the present application provides an image segmentation device.
[0045] In some embodiments, the image segmentation device includes an image space feature processing module, a scale feature processing module, an occlusion category feature processing module, and an image segmentation result acquisition module:
[0046] The image space feature processing module is used to obtain the image to be segmented, and input the image to be segmented into the backbone network to obtain the image space feature of the image to be segmented;
[0047] The scale feature processing module is used to input the image space feature into the decoding structure network to obtain pixel decoding features and global feature information;
[0048] The occlusion category feature processing module is used to input the pixel decoding feature and the global feature information into a first graph convolutional neural network to obtain an occlusion category feature;
[0049] The image segmentation result acquisition module is used to input the occlusion category features and the global feature information into a second graph convolutional neural network to obtain an image prediction segmentation result.
[0050] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image segmentation method as described in the second aspect above is implemented.
[0051] In a fifth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the image segmentation method as described in the second aspect above is implemented.
[0052] Compared with the related art, the image segmentation model training method, image segmentation method, device, electronic device and storage medium provided in the embodiments of the present application provide an end-to-end image segmentation model training method, and input the image to be segmented into the trained target image segmentation model, so as to output the image prediction segmentation result corresponding to the image to be segmented through the target image segmentation model, thereby providing an image segmentation method, which solves the problem of how to use a single image to predict and segment occluded targets in the related art, saves resources, and improves the accuracy and efficiency of image segmentation.
[0053] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0055] Figure 1 is a hardware structure block diagram of a terminal according to an image segmentation method of an embodiment of the present application;
[0056] Figure 2 is a flowchart of an image segmentation model training method according to an embodiment of the present application;
[0057] Figure 3is a schematic diagram of a sample image of an image segmentation model training method according to an embodiment of the present application;
[0058] Figure 4 is a schematic diagram of a translation image of an image segmentation model training method according to an embodiment of the present application;
[0059] Figure 5 is a flowchart of an image segmentation method according to an embodiment of the present application;
[0060] Figure 6 is a structural block diagram of an image segmentation device according to an embodiment of the present application;
[0061] Figure 7 It is a structural block diagram of an image segmentation model training device according to an embodiment of the present application. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.
[0063] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0064] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.
[0065] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the image segmentation method according to an embodiment of the present invention. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0066] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image segmentation method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0067] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.
[0068] The present application embodiment provides an image segmentation model training method, Figure 2 is a flowchart of an image segmentation model training method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:
[0069] Step S201: Acquire a sample image, and annotate a first background category object, a target category object, and an occlusion category object in the sample image.
[0070] In an embodiment of the present application, the video module captures videos of different scenes, extracts frames from the videos to obtain sample images, and then performs image segmentation and annotation on the sample images to obtain mask annotations of the first background category objects, target category objects, and occlusion category objects. Figure 3 is a schematic diagram of a sample image of the image segmentation model training method of an embodiment of the present application, such as Figure 3As shown, the first background category object refers to an object category that does not need to be segmented in the sample image processing, such as an irrelevant background object object in the image; the target category object refers to the original category object of the occluded area in the sample image, and the occluded object needs to be predicted and restored in subsequent processing to determine the corresponding object category when the occluded area is not occluded; the occluded category object refers to the object category that occludes the target category object. As an example, if the sample image obtained is a street monitoring image including people, sidewalks, roadways, and trees, in which part of the sidewalks and roadways are occluded by people, it is necessary to predict and restore the partially occluded sidewalks and roadways after segmenting the people, and the trees are irrelevant background objects in the embodiment of the present application, then in the embodiment of the present application, the sample image is segmented and labeled, and the trees are the first background category objects, the sidewalks and roadways in the unoccluded area are the target category objects, and the people are the occluded category objects, that is, the corresponding mask labels of trees, sidewalks and roadways, and people are determined respectively. In the subsequent embodiments, the specific processing of the sample image is explained based on this example sample image.
[0071] Step S202: determining a translation image according to the first background category object, the target category object and the occlusion category object.
[0072] A known type region is determined according to the first background category object, the target category object and the occlusion category object, and the occlusion category object is translated to the known type region to determine a translated image. The method for determining the known type region will be further described later. The translated image is used to generate a true category label for the target, thereby training the original training model.
[0073] Step S203 , determining a translation region according to the translation image, and determining a category object corresponding to the translation region in the translation image as a first true annotation, and determining a category object corresponding to the translation region in the sample image as a second true annotation.
[0074] The translation region refers to an image region formed after translating the occlusion class object to the known type region. The first true annotation corresponding to the translation region is the occlusion class, and the second true annotation is the first background class or the target class.
[0075] Step S204: input the translated image, the first true annotation, and the second true annotation into the original training model for training, and determine the trained original training model as the target image segmentation model.
[0076] In the embodiment of the present application, the translation image, the first real annotation and the second real annotation are input into the original training model for training, wherein the original training model can be a category-related prediction segmentation network model (Categories-Aware MaskPredicting MPFormer), referred to as CAP MPFormer. After inputting a large number of sample images for training, a qualified original training model is obtained and determined as the target image segmentation model.
[0077] Through the above steps, the embodiment of the present application generates a translated image through the object category annotated in the sample image, and further generates the real annotation of the translated area, thereby providing an end-to-end image segmentation model training method based on the translated image and the real annotation, and the image to be segmented can be further input into the target image segmentation model trained by the image segmentation model training method, so as to output the image prediction segmentation result corresponding to the image to be segmented through the target image segmentation model, which solves the problem of how to obtain a model that can use a single image to predict the segmentation of an occluded target in the related art, saves resources, optimizes the image segmentation model, and thus improves the accuracy and efficiency of image segmentation.
[0078] In some embodiments, step S202 includes:
[0079] Step S2021, determining that the image regions corresponding to the first background category object and the target category object are known type regions, and the image region corresponding to the occlusion category object is an occlusion region.
[0080] In the embodiment of the present application, the image regions corresponding to the first background category object and the target category object in the sample image are determined as known type regions, and the image regions corresponding to the occlusion category object in the sample image are determined as occlusion regions. For example, the image regions corresponding to trees, sidewalks, and roadways are determined as known type regions, and the regions corresponding to people are determined as occlusion regions.
[0081] Step S2022: translate the occluded category object to a known type region, and mark the occluded region as a second background category object to determine a translated image.
[0082] Figure 4 is a schematic diagram of a translation image of an image segmentation model training method according to an embodiment of the present application, such as Figure 4As shown, the image after the occlusion category object is translated to a known type area and the original occlusion area is marked as a second background category object is determined to be a translated image. As an example, the translated image can be an image formed after the person is translated to a known sidewalk area and the occlusion area corresponding to the person in the original sample image is marked as a second background type object. Among them, marking the occlusion area corresponding to the person in the original sample image as the second background type object can be that after the circumscribed rectangle of the occlusion area is smeared, it is updated and marked as the second background category object, and the marking of the second background category object can be consistent with the first background category object.
[0083] Through the above steps, the embodiment of the present application can generate the predicted segmentation samples of the predicted segmentation training data, that is, the translated image, by pixel translation and image smearing. Therefore, there is no need to collect additional sample image data with foreground and background separation. Instead, the image area corresponding to the occlusion category object is determined as the occlusion area. After translating the occlusion category object to the known type area, the occlusion area is smeared with a circumscribed rectangle, so that the pixel points of the occlusion area are not adjusted during the model training process to avoid interference, thereby providing a translation image determination method that can accurately train a usable model, which is efficient, feasible and low-cost.
[0084] In some embodiments, step S203 includes:
[0085] Step S2031: determine the area corresponding to the translated occlusion category object in the translated image as the translated area.
[0086] After determining the translation image, the image area corresponding to the translated occlusion category object is determined as the translation area. Continuing with the above example, the translation area can be the area where the person is located after translating the person to a known sidewalk area.
[0087] Step S2032: determine a first real annotation and a second real annotation corresponding to the translation area, wherein the first real annotation is an occlusion category object, and the second real annotation is a first background category object or a target category object.
[0088] After determining the translation region in the translation image, the translation region will have two corresponding annotations, one for the original first background category object or target category object annotation of the region before translation, and the other for the occlusion category object annotation corresponding to the region after translation. The first true annotation is determined to be the occlusion category object, and the second true annotation is determined to be the first background category object or the target category object.
[0089] Through the above steps, the embodiment of the present application marks the translation area to form two real annotations corresponding to before and after the translation. Therefore, during model training, the errors of the network model can be corrected based on the two real annotations to optimize the model, making the model training more accurate, saving resources and highly feasible.
[0090] In some embodiments, step S2032 includes:
[0091] Step S2132, when the second real annotation is a target category object, the target category object may include one or more category objects.
[0092] When the occluded object is translated to a known area consisting of a target object, the second ground truth is the target object, where the target object can be one or more objects. For example, the target object can be a sidewalk, a roadway, or both.
[0093] Step S2232: When the target category object includes multiple categories of objects, determine that the second real annotation corresponding to the translation area is one or more of the multiple target category objects.
[0094] When the target category object includes multiple categories of objects, the second real annotation corresponding to the translation area can be determined to be one or more of the multiple target category objects. For example, when the target category object can be a sidewalk and a roadway, the second real annotation can be a sidewalk, a roadway, or a sidewalk and a roadway.
[0095] Through the above steps, the embodiment of the present application can include multiple target category object types, and the second real annotation can also include one or more category objects. In this way, the translation area corresponding to the occluded category object is not limited, which is more flexible and efficient.
[0096] The present application also provides an image segmentation method. The method comprises the following steps:
[0097] Step S211, obtaining a sample image, and marking a first background category object, a target category object, and an occlusion category object in the sample image, wherein the target category object is a sidewalk and / or a roadway, the occlusion category object is a person, and the first background category object is an object other than the target category object and the occlusion category object in the sample image;
[0098] Step S221, determining that the image area corresponding to the first background category object and the sidewalk and / or the roadway is a known type area, and the image area corresponding to the person is an occlusion area;
[0099] Step S231, translating the person to a known type area, and marking the blocked area as a second background category object to determine a translated image;
[0100] Step S241, determining the area corresponding to the translated person in the translated image as the translated area;
[0101] Step S251, determining a first real annotation and a second real annotation corresponding to the translation area, wherein the first real annotation is a person, and the second real annotation is one or more of a first background category object, a sidewalk, and a roadway;
[0102] Step S261: input the translated image, the first true annotation, and the second true annotation into the original training model for training, and determine the trained original training model as the target image segmentation model.
[0103] Through the above steps, the embodiment of the present application generates a translated image through the object category annotated in the sample image, and further generates the real annotation of the translated area, thereby providing an end-to-end image segmentation model training method based on the translated image and the real annotation, and the image to be segmented can be further input into the target image segmentation model trained by the image segmentation model training method, so as to output the image prediction segmentation result corresponding to the image to be segmented through the target image segmentation model, which solves the problem of how to obtain a model that can use a single image to predict the segmentation of an occluded target in the related art, saves resources, optimizes the image segmentation model, and thus improves the accuracy and efficiency of image segmentation.
[0104] The embodiment of the present application also provides an image segmentation method. Figure 5 is a flow chart of an image segmentation method according to an embodiment of the present application. Figure 5 As shown, the process includes the following steps:
[0105] Step S301: Acquire an image to be segmented, and input the image to be segmented into a target image segmentation model trained by the image segmentation model training method described in an embodiment of the present application.
[0106] The image segmentation method in the embodiment of the present application is implemented based on the aforementioned target image segmentation model. Accordingly, the image to be segmented may include a first background category object, a target category object, and an occlusion category object.
[0107] Step S302: outputting the image prediction segmentation result corresponding to the image to be segmented through the target image segmentation model.
[0108] The image to be segmented is input into the target image segmentation model trained by the image segmentation model training method of the above embodiment, and the image prediction segmentation result corresponding to the image to be segmented can be output. Among them, the output image prediction segmentation result may include: the image after the pixels corresponding to the occluded category object are marked after the prediction, and the image after the pixels corresponding to the target category object of the occluded part are marked after the prediction is restored. In addition, according to the needs, the image prediction segmentation result may also include the image after the pixels corresponding to the irrelevant background objects are marked, the image after the pixels corresponding to the combination of several categories of objects are marked, etc. As an example, if the acquired image to be segmented is a street surveillance image including people, sidewalks, roadways, and trees, wherein part of the sidewalks and roadways are blocked by people, it is necessary to predict and restore the blocked sidewalks and roadways after segmenting the people, and the trees are irrelevant background objects in the embodiment of the present application, then in the embodiment of the present application, the trees in the image to be segmented are objects of the first background category, the sidewalks and roadways are objects of the target category, and the people are objects of the blocked category; the output image prediction segmentation result may include: an image with the pixels corresponding to the segmented people annotated, an image with the pixels corresponding to the predicted and restored blocked sidewalks and roadways annotated, an image with the pixels corresponding to the segmented trees annotated, and a combined image of the above-mentioned objects. In the subsequent embodiments, the specific processing of the image to be segmented is explained based on this example image to be segmented.
[0109] Through the above steps, the embodiment of the present application inputs the image to be segmented into the target image segmentation model trained by the image segmentation model training method, and then outputs the image prediction segmentation result corresponding to the image to be segmented through the end-to-end target image segmentation model. There is no need to manually label and distinguish the background, which solves the problem of how to use a single image to predict and segment occluded targets in the related art, saves resources, and improves the accuracy and efficiency of image segmentation.
[0110] In some embodiments, the target image segmentation model includes a backbone network, a decoding structure network, a first graph convolutional neural network, and a second graph convolutional neural network, and step S302 includes:
[0111] Step S3021, obtaining the image to be segmented, and inputting the image to be segmented into the backbone network to obtain the image spatial features of the image to be segmented.
[0112] The backbone network is used for image feature extraction. In order to improve the utilization of image features at all scales, the embodiment of the present application selects the residual network (optionally Resnet50) of the iterative aggregation structure (Iterative Deep Aggregation, IDA) as the backbone network. Furthermore, in order to simultaneously utilize the high resolution of low-level features and the high semantic information of high-level features, the prediction effect is achieved by fusing the features of these different layers, and the prediction is performed separately on each fused feature layer, the embodiment of the present application adds the Feature Pyramid Network (FPN) algorithm to extract image features.
[0113] The iterative aggregation structure can start from the level close to the input end and iteratively aggregate the deep features of each level step by step, so that the shallow spatial features can be passed to the deep levels.
[0114] Optionally, IDA's expression is as follows:
[0115]
[0116] Among them, N represents each level in the IDA structure Resnet50. The outputs of different levels of Resnet50 are input into the feature pyramid network FPN, and then the outputs of different layers of FPN are spliced as the image spatial features of the image to be segmented extracted by the backbone network, and used to input into the subsequent decoding structure network.
[0117] Step S3022: input the image spatial features into the decoding structure network to obtain pixel decoding features and global feature information.
[0118] The decoding structure network includes a pixel decoding network and a transform decoding network. The image spatial features are input into the pixel decoding network to output pixel decoding features. Further, the obtained pixel decoding features are input into the transform decoding network to output global feature information.
[0119] Step S3023: Input the pixel decoding features and the global feature information into the first graph convolutional neural network to obtain the occlusion category features.
[0120] The pixel decoding features and global feature information are input into the first graph convolutional neural network. The first graph convolutional neural network segments and identifies the corresponding occlusion category objects in the input feature information, thereby accurately extracting the features corresponding to the occlusion category objects, and determining the features corresponding to the occlusion category objects as occlusion category features.
[0121] Step S3024: input the occlusion category features and global feature information into the second graph convolutional neural network to obtain an image prediction segmentation result.
[0122] Among them, the second graph convolutional neural network is in a cascade relationship with the first graph convolutional neural network. The second graph convolutional neural network fuses the global feature information and the occlusion category features output by the first graph convolutional neural network, and combines the graph convolutional neural network's own function of processing non-local feature information between pixels to decode the fused features, thereby predicting and segmenting background category objects and target category objects. Specifically, the second graph convolutional neural network fuses the global feature information and the occlusion category features output by the first graph convolutional neural network by splicing the feature information of the global feature information and the occlusion category features.
[0123] Furthermore, after the image prediction segmentation result predicts the target category object of the occluded area, each pixel corresponding to the target category object is labeled according to the predicted classification, thereby forming a predicted segmented image. The labeled output form may include n groups of results such as {mask1, class1} and {mask2, class2}..., where mask represents the mask and class represents the category of the pixel corresponding to the mask. Taking the above-mentioned image to be segmented as an example, the class1 (category 1) of the pixel corresponding to mask1 (mask 1) can be a background category object (tree), and the class2 (category 2) of the pixel corresponding to mask2 (mask 2) can be a target category object (sidewalk), and the rest of the results are similar.
[0124] Through the above steps, the embodiment of the present application uses a cascade mask-driven transform decoding network and a graph convolutional neural network (Graph Convolutional Networks, GCN for short) method to segment foreground target categories and background categories, and uses GCN to focus on the characteristics of non-local relationships between pixels, and predictively segment target category objects or background category objects that are occluded by specific categories. Specifically, based on the backbone network, decoding structure network, first graph convolutional neural network and second graph convolutional neural network in the target image segmentation model, the image to be segmented is predictively segmented to obtain an image prediction segmentation result, thereby improving the detection effect of small targets and the accuracy of the model.
[0125] In addition, in an embodiment of the present application, the first true annotation after translation is used to calculate the loss of the first graph convolutional neural network, and the second true annotation before translation is used to calculate the loss of the second graph convolutional neural network.
[0126] The loss function used to train an image segmentation model consists of two parts:
[0127] L=L screen +L pred
[0128] Where, L screen represents the segmentation loss of the occluded category, Lpred Represents the loss of image prediction segmentation results.
[0129] Among them, L screen In , the targets of the occluded category are usually small, and the first graph convolutional neural network only predicts the occluded category, which is equivalent to the segmentation of two categories. Therefore, the focal loss function (Focal_loss) is used instead of the original binary cross entropy loss for calculation:
[0130] L screen =L s_mask +λ s_cls L s_cls
[0131] Where, L s_mask represents the loss of mask prediction, L s_cls represents the loss of category prediction, where λ s_cls It is a fixed parameter that can be adjusted to control L s_cls proportion.
[0132] in,
[0133]
[0134] In the formula, α and β are fixed parameters, which are used to control the balance of positive and negative samples and the weight of difficult and easy samples respectively. pred Represents the predicted value, y true Represents a true value.
[0135] L pred The same calculation method as the general image segmentation model (Mask2Former) is used, using a random scatter method, which includes two parts: mask loss and category loss.
[0136] L pred =L p_mask +λ p_cls L p_cls
[0137] Where, L p_mask represents the loss of mask prediction, L p_cls represents the loss of category prediction, where λ p_cls It is a fixed parameter that can be adjusted to control L s_cls proportion.
[0138] The category loss is calculated using binary cross entropy loss, and the mask loss is calculated using a combination of binary cross entropy loss and Dice loss.
[0139] L p_mask =λ p_bce L p_bce +λ p_dice Lp_dice
[0140] Where, L p_bce represents the binary cross entropy loss, λ p_bce To control L p_bce Specific gravity, L p_dice represents the Dice loss, λ p_dice It is a fixed parameter that can be adjusted to control L p_dice proportion.
[0141] In some embodiments, step S3023 includes:
[0142] Step S3123, determining the first background category objects, target category objects and occlusion category objects in the image to be segmented.
[0143] Corresponding to the sample images in the model training process, in the embodiment of the present application, the object objects in the image to be segmented are determined as a first background category object, a target category object and an occlusion category object.
[0144] Step S3223: Determine the first input feature according to the pixel decoding feature and the global feature information.
[0145] In the embodiment of the present application, the first graph convolutional neural network regards both the target category object and the first background category object as background category objects, and only extracts features of the occluded category objects in the segmented image. Determine the first input feature X based on the pixel decoding features and the global feature information roi .
[0146] Step S3323: The first graph convolutional neural network performs feature extraction on the occlusion category object according to the first input feature to obtain the occlusion category feature.
[0147] The first graph convolutional neural network extracts features of the occlusion category object based on the first input feature based on the following formula to obtain the occlusion category feature:
[0148] S=σ(A 0 X roi W g )+X roi
[0149] Among them, S represents the output occlusion category feature, σ represents the nonlinear activation function, and A 0 represents the adjacency matrix in the first graph convolutional neural network, X roi represents the first input feature, W g Represents the parameters of the first graph convolutional neural network layer. In addition, the occlusion category feature can be decoded into the form of the mask and category of the occlusion category object to output.
[0150] Through the above steps, the embodiment of the present application determines the first input feature based on the pixel decoding feature and the global feature information, so that the first graph convolutional neural network focuses on the image feature extraction of the occluded category objects, thereby avoiding interference, making the model better and more efficient, and the segmentation result more accurate.
[0151] In some embodiments, step S3024 includes:
[0152] Step S3124: Determine the second input feature according to the occlusion category feature and the global feature information.
[0153] In the embodiment of the present application, the second graph convolutional neural network determines the second input feature X by splicing the global feature information and the occlusion category features according to the following formula: i ,
[0154] X i =SW fcn +X roi
[0155] Among them, the second input feature X i represents the input of the second graph convolutional neural network, W fcn Represents the parameters of the fully connected layer.
[0156] In step S3224, the second graph convolutional neural network performs feature extraction on the first background category object and the target category object according to the second input feature to obtain background category features and target category features.
[0157] The second graph convolutional neural network is used to extract the relationship between occlusion and occlusion. At the same time, combined with the characteristics of the graph convolutional neural network that supports the extraction of non-local features, the occluded area in the image to be segmented is predicted and segmented based on the following formula:
[0158] O=σ(A 1 X i W g ′)+X i
[0159] Among them, A 1 represents the adjacency matrix in the second graph convolutional neural network, W g ′ represents the parameters of the second graph convolutional neural network layer, and O represents the background category features and the target category features.
[0160] Step S3324, decoding the background category features and the target category features to determine the image prediction segmentation result.
[0161] Furthermore, by decoding the background category features and the target category features, the image prediction segmentation result can be obtained.
[0162] Through the above steps, the embodiment of the present application provides a second graph convolutional neural network cascaded with the first graph convolutional neural network, which is used to process occlusion category features and global feature information, further optimize the image segmentation model, and make the image prediction segmentation result accurate.
[0163] The following describes and illustrates the embodiment of the present application through a preferred embodiment. The process of the image segmentation method of the preferred embodiment of the present application is as follows:
[0164] Step S401 , obtaining an initial image, scaling the initial image with a locked aspect ratio and grayscale filling to obtain an image to be segmented with a fixed size.
[0165] In step S402, the image to be segmented is input into a backbone network including an iterative aggregation structure and a feature pyramid algorithm, and multi-scale deformation attention is used as a pixel decoding network module to extract features of different scales from the image to be segmented, and a feature map including image spatial features is output.
[0166] Step S403, input the first three layers of features in the pixel decoding network into the transform decoding network, and perform three-layer one hundred queries to obtain a query result vector.
[0167] Step S404, fusing the query result vector output by the transform decoding network decoding layer and the image feature output by the fourth layer of the pixel decoding network to determine the first input feature.
[0168] Step S405: The first graph convolutional neural network performs feature extraction on the occlusion category object according to the first input feature to obtain an occlusion category feature.
[0169] Step S406, the occlusion category feature, the decoded feature output by the transform decoding network, and the image feature output by the fourth layer of the pixel decoding network are fused to determine the second input feature.
[0170] Step S407: The second graph convolutional neural network performs feature extraction on the first background category object and the target category object according to the second input feature to obtain background category features and target category features.
[0171] Step S408: decoding the background category features and the target category features to determine the image prediction segmentation result.
[0172] Among them, the first three layers of features in the pixel decoding network are input into the transform decoding network, and three layers of 100 queries are performed, which can improve the detection effect of small targets. Furthermore, in order to optimize the problem of inconsistent mask predictions between consecutive decoder layers in the transform decoding network, the real category mask mixed with noise can be combined with the native mask to improve the stability of the decoding decision. It is worth noting that during inference, the transform decoding network is consistent with the general image segmentation model (Mask2Former), and the mask driver module (MP module) is added only in the training stage.
[0173] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0174] The present application also provides an image segmentation device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0175] Figure 6 is a structural block diagram of an image segmentation device according to an embodiment of the present application, such as Figure 6 As shown, the device includes: an image space feature processing module 10, a scale feature processing module 20, an occlusion category feature processing module 30, and an image segmentation result acquisition module 40:
[0176] The image space feature processing module 10 is used to obtain the image to be segmented, and input the image to be segmented into the backbone network to obtain the image space feature of the image to be segmented;
[0177] The scale feature processing module 20 is used to input the image space feature into the decoding structure network to obtain pixel decoding features and global feature information;
[0178] The occlusion category feature processing module 30 is used to input the pixel decoding feature and the global feature information into the first graph convolutional neural network to obtain the occlusion category feature;
[0179] The image segmentation result acquisition module 40 is used to input the occlusion category features and global feature information into the second graph convolutional neural network to obtain the image prediction segmentation result.
[0180] The present application also provides an image segmentation model training device, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0181] Figure 7 is a structural block diagram of an image segmentation model training device according to an embodiment of the present application, such as Figure 7 As shown, the device includes: a sample image annotation module 100, a translation image determination module 200, a real annotation determination module 300 and a segmentation model determination module 400:
[0182] The sample image annotation module 100 is used to obtain a sample image and annotate a first background category object, a target category object, and an occlusion category object in the sample image;
[0183] A translation image determination module 200, configured to determine a translation image according to a first background category object, a target category object, and an occlusion category object;
[0184] A real annotation determining module 300 is used to determine a translation region according to the translation image, and determine a category object corresponding to the translation region in the translation image as a first real annotation, and determine a category object corresponding to the translation region in the sample image as a second real annotation;
[0185] The segmentation model determination module 400 is used to input the translated image, the first true annotation and the second true annotation into the original training model for training, and determine the trained original training model as the target image segmentation model.
[0186] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0187] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0188] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0189] Optionally, in the embodiment of the present application, the processor may be configured to perform the following steps through a computer program:
[0190] Acquire an image to be segmented, and input the image to be segmented into a target image segmentation model trained by the image segmentation model training method described in an embodiment of the present application;
[0191] Through the target image segmentation model, the image prediction segmentation result corresponding to the image to be segmented is output.
[0192] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0193] Acquire a sample image, and mark a first background category object, a target category object, and an occlusion category object in the sample image;
[0194] Determine a translation image according to the first background category object, the target category object, and the occlusion category object;
[0195] Determine a translation region according to the translation image, determine a category object corresponding to the translation region in the translation image as a first true annotation, and determine a category object corresponding to the translation region in the sample image as a second true annotation;
[0196] The translated image, the first true annotation and the second true annotation are input into the original training model for training, and the trained original training model is determined as the target image segmentation model.
[0197] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0198] In addition, in combination with the image segmentation method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the image segmentation methods in the above embodiment is implemented.
[0199] In addition, in combination with the image segmentation model training method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the image segmentation model training methods in the above embodiment is implemented.
[0200] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0201] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0202] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for training an image segmentation model, characterized in that: The following steps are involved: Acquire a sample image, and mark a first background category object, a target category object, and an occlusion category object in the sample image; Determine a translation image according to the first background category object, the target category object, and the occlusion category object; Determine a translation region according to the translation image, determine a category object corresponding to the translation region in the translation image as a first true annotation, and determine a category object corresponding to the translation region in the sample image as a second true annotation; The translated image, the first true annotation and the second true annotation are input into an original training model for training, and the trained original training model is determined as a target image segmentation model.
2. The image segmentation model training method according to claim 1, characterized in that: The determining of the translation image according to the first background category object, the target category object and the occlusion category object comprises: Determine that the image regions corresponding to the first background category object and the target category object are known type regions, and the image region corresponding to the occlusion category object is an occlusion region; The occlusion category object is translated to the known type region, and the occlusion region is marked as a second background category object to determine a translated image.
3. The image segmentation model training method according to claim 2, characterized in that: The step of determining the translation region according to the translation image, determining the category object corresponding to the translation region in the translation image as the first real annotation, and determining the category object corresponding to the translation region in the sample image as the second real annotation comprises: Determine, in the translated image, an area corresponding to the translated occlusion category object as the translated area; A first real annotation and a second real annotation corresponding to the translation region are determined, wherein the first real annotation is the occlusion category object, and the second real annotation is the first background category object or the target category object.
4. The image segmentation model training method according to claim 3, characterized in that: The determining of the first real annotation and the second real annotation corresponding to the translation area, wherein the first real annotation is the occlusion category object, and the second real annotation is the first background category object or the target category object comprises: In the case where the second real annotation is the target category object, the target category object may include one or more category objects; In the case that the target category object includes objects of multiple categories, it is determined that the second real annotation corresponding to the translation area is one or more of the multiple target category objects.
5. A method for training an image segmentation model, characterized in that: The following steps are involved: Acquire a sample image, and mark a first background category object, a target category object, and an occlusion category object in the sample image, wherein the target category object is a sidewalk and / or a roadway, the occlusion category object is a person, and the first background category object is an object other than the target category object and the occlusion category object in the sample image; Determine that the image area corresponding to the first background category object and the sidewalk and / or roadway is a known type area, and the image area corresponding to the person is an occluded area; The person is translated to the known type area, and the blocked area is marked as a second background category object to determine a translated image; Determining, in the translated image, an area corresponding to the translated person as a translated area; Determine a first real annotation and a second real annotation corresponding to the translation area, wherein the first real annotation is the person, and the second real annotation is one or more of the first background category object, the sidewalk, and the roadway; The translated image, the first true annotation and the second true annotation are input into an original training model for training, and the trained original training model is determined as a target image segmentation model.
6. An image segmentation method, characterized in that: The following steps are involved: Get the image to be segmented, and input the image to be segmented into A target image segmentation model trained by the image segmentation model training method according to any one of claims 1 to 4; The target image segmentation model is used to output the image prediction segmentation result corresponding to the image to be segmented.
7. The image segmentation method according to claim 6, characterized in that: The target image segmentation model includes a backbone network, a decoding structure network, a first graph convolutional neural network and a second graph convolutional neural network. The outputting of the image prediction segmentation result corresponding to the image to be segmented by the target image segmentation model includes: Acquire an image to be segmented, and input the image to be segmented into the backbone network to obtain image spatial features of the image to be segmented; Inputting the image spatial features into the decoding structure network to obtain pixel decoding features and global feature information; Inputting the pixel decoding feature and the global feature information into the first graph convolutional neural network to obtain an occlusion category feature; The occlusion category feature and the global feature information are input into the second graph convolutional neural network to obtain the image prediction segmentation result.
8. The image segmentation method according to claim 7, characterized in that: The step of inputting the pixel decoding feature and the global feature information into a first graph convolutional neural network to obtain an occlusion category feature comprises: Determine a first background category object, a target category object, and an occlusion category object in the image to be segmented; Determine a first input feature according to the pixel decoding feature and the global feature information; The first graph convolutional neural network performs feature extraction on the occlusion category object according to the first input feature to obtain an occlusion category feature.
9. The image segmentation method according to claim 8, characterized in that: The step of inputting the occlusion category feature and the global feature information into a second graph convolutional neural network to obtain an image prediction segmentation result includes: Determining a second input feature according to the occlusion category feature and the global feature information; The second graph convolutional neural network performs feature extraction on the first background category object and the target category object according to the second input feature to obtain background category features and target category features; The background category features and the target category features are decoded to determine the image prediction segmentation result.
10. An image segmentation device, characterized in that: It includes image space feature processing module, scale feature processing module, occlusion category feature processing module and image segmentation result acquisition module: The image space feature processing module is used to obtain the image to be segmented, and input the image to be segmented into the backbone network to obtain the image space feature of the image to be segmented; The scale feature processing module is used to input the image space feature into the decoding structure network to obtain pixel decoding features and global feature information; The occlusion category feature processing module is used to input the pixel decoding feature and the global feature information into a first graph convolutional neural network to obtain an occlusion category feature; The image segmentation result acquisition module is used to input the occlusion category features and the global feature information into a second graph convolutional neural network to obtain an image prediction segmentation result.
11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the image segmentation method according to any one of claims 6 to 9.
12. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the image segmentation method according to any one of claims 6 to 9 when executed.