Image processing method, device, storage medium and computer equipment

Abstract

The present invention discloses an image processing method, apparatus, storage medium, and computer equipment. The method comprises: obtaining an image to be segmented; predicting a mask of an object in the image to obtain a first result, and predicting the contour of the object in the image to obtain a second result; and processing the first and second results to obtain an image segmentation result. The present invention addresses the technical problem of inaccurate object segmentation in related art image segmentation methods based on maskrcnn.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, apparatus, storage medium and computer equipment. Background Art

[0002] In the current field of computer vision, there are many solutions for intelligently identifying and segmenting instances in images. In the process of instance segmentation of images, the method generally adopted is the instance segmentation method based on maskrcnn, which is a two-stage instance segmentation method. In the first stage, the RPN network detects the locations that may contain objects and obtains a certain number of candidate boxes. In the second stage, based on the candidate boxes generated in the first stage, the ROIAlign operation is performed to perform classification and regression of the segmentation bbox, as well as classification of the mask. However, this method has some disadvantages: 1. The number of instances that can be segmented depends on the number of candidate boxes detected. 2. The confidence of the segmentation is given by the confidence of the detection bbox, which cannot accurately measure the overlap between the mask and the GT.

[0003] Therefore, in the related art, the instance segmentation method based on maskrcnn has the problem of inaccurate instance segmentation.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide an image processing method, apparatus, storage medium, and computer equipment to at least solve the technical problem of inaccurate object segmentation in the image segmentation method based on maskrcnn in the related art.

[0006] According to one aspect of an embodiment of the present invention, there is provided an image processing method, comprising: acquiring an image to be segmented; predicting a mask of an object in the image to obtain a first result, and predicting a contour of the object in the image to obtain a second result; and processing the first result and the second result to obtain a segmentation result of the image.

[0007] According to another aspect of an embodiment of the present invention, an image processing method is provided, comprising: obtaining a remotely sensed building image to be segmented, wherein the remotely sensed building image comprises: an image including a building acquired by remote sensing detection; predicting a mask of the building in the remotely sensed building image to obtain a first result, and predicting an outline of the building in the remotely sensed building image to obtain a second result; and processing the first result and the second result to obtain a segmentation result of the remotely sensed building image.

[0008] According to another aspect of an embodiment of the present invention, an image processing device is also provided, including: a first acquisition module, used to acquire an image to be segmented; a first prediction module, used to predict the mask of an object in the image to obtain a first result, and predict the contour of the object in the image to obtain a second result; a first processing module, used to process the first result and the second result to obtain a segmentation result of the image.

[0009] According to another aspect of an embodiment of the present invention, an image processing device is further provided, including: a second acquisition module, used to acquire a remote sensing building image to be segmented, wherein the remote sensing building image includes: an image including a building acquired by remote sensing detection; a second prediction module, used to predict the mask of the building in the remote sensing building image to obtain a first result, and predict the outline of the building in the remote sensing building image to obtain a second result; and a second processing module, used to process the first result and the second result to obtain a segmentation result of the remote sensing building image.

[0010] According to another aspect of the embodiments of the present invention, a storage medium is provided, including: the storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute any one of the above-mentioned image processing methods.

[0011] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program; and the processor is configured to execute the computer program stored in the memory, wherein when the computer program is executed, the processor executes any one of the above-described data processing methods.

[0012] In an embodiment of the present invention, a method of obtaining an image to be segmented is adopted, and the mask and instance contours of the object in the image are predicted respectively, so as to achieve the purpose of identifying the object in the image by combining the predicted mask with the predicted contour, thereby achieving the technical effect of accurately segmenting the object in the image, and further solving the technical problem of inaccurate object segmentation in the image segmentation method based on maskrcnn in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0014] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown;

[0015] Figure 2is a flowchart of an image processing method 1 according to embodiment 1 of the present invention;

[0016] Figure 3 is a flowchart of a second image processing method according to embodiment 1 of the present invention;

[0017] Figure 4 is a schematic diagram of a dual decoder prediction mask and contour according to a preferred embodiment of the present invention;

[0018] Figure 5 is a schematic diagram of applying the watershed method to perform image instance segmentation according to a preferred embodiment of the present invention;

[0019] Figure 6 is a structural block diagram of an image processing device according to embodiment 2 of the present invention;

[0020] Figure 7 is a structural block diagram of an image processing device 2 according to embodiment 3 of the present invention;

[0021] Figure 8 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0025] Mask R-CNN: A general-purpose instance segmentation architecture, Mask R-CNN builds on the principles of Faster R-CNN, employing the ResNet-FPN architecture for feature extraction and adding a mask prediction branch. Faster R-CNN is a two-stage object detection algorithm consisting of a region proposal phase (stage 1) and bounding box regression and classification phase (stage 2).

[0026] FPN: A method in the field of computer vision science that uses conventional convolutional neural network (CNN) models to efficiently extract features of various dimensions in images, using the idea of ​​image pyramids to solve the problem of difficulty in detecting small-sized objects in object detection scenarios.

[0027] Intersection over Union (IoU): In object detection, the intersection over Union (IoU) refers to the overlap ratio between the generated candidate bound and the original labeled bound, and the value is between [0,1].

[0028] ROIAlign: A region of interest feature aggregation algorithm that uses bilinear interpolation to obtain image values ​​at pixel points with floating-point coordinates, thereby converting the entire feature aggregation process into a continuous operation.

[0029] Bounding box regression: During object detection, candidate bounding boxes are approximated to the annotated ground-truth bounding box. Because a bounding box in an image can be uniquely identified by its center point coordinates and width and height, this approximation process can be modeled as a regression problem. By performing bounding box regression on candidate bounding boxes, the final detected object location can be closer to the ground-truth, improving positioning accuracy.

[0030] Watershed algorithm: An algorithm that considers image segmentation based on the composition of the watershed, including topology, morphology, immersion simulation, precipitation simulation and other implementation algorithms.

[0031] Example 1

[0032] According to an embodiment of the present invention, a method embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] The method embodiment provided in Example 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0034] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the vulnerability detection method of the application program described above. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission device is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0038] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 is a flow chart of an image processing method 1 according to embodiment 1 of the present invention. Figure 2 As shown, the process includes the following steps:

[0039] Step S202, obtaining an image to be segmented;

[0040] Step S204, predicting the mask of the object in the image to obtain a first result, and predicting the outline of the object in the image to obtain a second result;

[0041] Step S206: Process the first result and the second result to obtain an instance segmentation result of the image.

[0042] Through the above steps, the method of obtaining the image to be segmented is adopted, and the mask and instance contour of the object in the image are predicted respectively, so as to achieve the purpose of identifying the object in the image by combining the predicted mask with the predicted contour, thereby achieving the technical effect of accurately segmenting the object in the image, and further solving the technical problem of inaccurate object segmentation in the image segmentation method based on maskrcnn in the related art.

[0043] As an optional embodiment, the objects in the image referred to above are targets to be identified from the image. For example, if an image includes a person, plants, toys, and some background environment, the objects in the image can be people, plants, toys, etc. Image segmentation, which is the process of segmenting these objects from the image, is also the process of image recognition. Furthermore, when using a predetermined algorithm to segment or recognize an image, the objects referred to here can also be referred to as instances.

[0044] As an optional embodiment, the method may include: predicting the mask of an object in an image to obtain a first result includes: using a first branch of a dual decoder to predict the mask of the object in the image to obtain the first result; and predicting the contour of the object in the image to obtain a second result includes: using a second branch of the dual decoder to predict the contour of the object in the image to obtain the second result. That is, in an image segmentation model, two branches are used to predict the mask and contour of the object in the image, and the same dual decoder is used to perform the predictions of both branches, effectively reducing the image prediction processing flow.

[0045] As an optional embodiment, the first result in the above method may include: a mask prediction result and the intersection-over-union ratio of the predicted mask boundary to the actual boundary; the second result may include: a contour prediction result and a contact contour prediction result. The parameters included in the above first and second results are merely examples of one type of result parameter; the above first and second results may also include other process parameters. Result parameters may be parameters output by the above dual decoder for subsequent processing, while process parameters may be parameters output by the above dual decoder that are not used in subsequent processing or are parameters used to derive result parameters.

[0046] As an optional embodiment, the first result in the above method may include: using the first branch of the dual decoder to predict the mask of the object in the image, obtaining the first result including: using the first branch of the dual decoder to predict the mask of the object in the image to obtain a mask class prediction result and a mask boundary prediction result; determining the intersection-and-union ratio of the predicted mask boundary and the actual mask boundary based on the mask boundary prediction result; using the second branch of the dual decoder to predict the contour of the object in the image, obtaining the second result including: using the second branch of the dual decoder to predict the contour of the instance in the image to obtain a contour class prediction result and a contour prediction result; and determining the contact contour class prediction result based on the contour prediction result. Through the above processing, it is achieved that: the mask boundary prediction result predicted by the first branch of the dual decoder is used as a process parameter to obtain the result parameter: the intersection-and-union ratio of the predicted mask boundary and the actual mask boundary; and the contour prediction result predicted by the second branch of the dual decoder is used as a process parameter to obtain the result parameter: the contact contour class prediction result.

[0047] As an optional embodiment, the above-mentioned processing of the first result and the second result to obtain an instance segmentation result of the image includes: determining a seed point for image segmentation based on the first result and the second result; and segmenting the image based on the seed point to obtain an image segmentation result. Through the above-mentioned processing, the seed point for image segmentation is determined based on the first result and the second result. That is, when performing image segmentation, the basis is not only the result obtained by predicting the mask of the object in the image, but also the result obtained by predicting the contour of the object in the image. That is, the seed point for image segmentation is determined based on the above-mentioned mask-class prediction result and the intersection-over-union ratio of the predicted mask boundary to the actual boundary, as well as the contour-class prediction result and the contact contour-class prediction result. Thereafter, the image is segmented based on the determined seed point, so that objects included in the segmentation result obtained by segmenting the image are not easily missed, and the final segmentation confidence is closely related to the segmentation result, and has nothing to do with the bbox in the instance segmentation based on Mask R-CNN in the related art.

[0048] As an optional embodiment, the above-mentioned images to be segmented may be images of various types, for example, images obtained by shooting or recording in various ways, that is, the above-mentioned image processing method can be applied to image segmentation in various scenarios. Preferably, considering the problem that the recognition accuracy of the target object in the image may be low due to different angles or the existence of overlapping areas, the processing method can be used to perform instance segmentation on the remote sensing image. That is, the above-mentioned acquisition of the image to be segmented may include: acquiring the image to be segmented by remote sensing detection. Since the remote sensing detection method can detect the target object from various angles, the acquired image information is more objective and less overlapping. Therefore, the image acquired by remote sensing detection is more suitable for the application of the above-mentioned image processing method, which can effectively improve the accuracy of image segmentation.

[0049] As an optional embodiment, the aforementioned image may include an image containing buildings. For example, when identifying and segmenting buildings in remote sensing images, it may be necessary to estimate the area of ​​each building in the remote sensing image. Therefore, building segmentation is necessary. However, due to the varying angles of buildings in remote sensing images, accurate segmentation of the buildings in the image is difficult. Using the aforementioned method, buildings in an image can be accurately identified.

[0050] In an embodiment of the present invention, an image processing method is also provided. Figure 3 is a flow chart of the second image processing method according to embodiment 1 of the present invention. Figure 3 As shown, the process includes the following steps:

[0051] Step S302: obtaining a remote sensing building image to be segmented, wherein the remote sensing building image includes: an image including a building obtained by remote sensing detection;

[0052] Step S304, predicting the mask of the building in the remote sensing building image to obtain a first result, and predicting the outline of the building in the remote sensing building image to obtain a second result;

[0053] Step S306: Process the first result and the second result to obtain a segmentation result of the remote sensing building image.

[0054] Through the above steps, the remote sensing building image to be segmented is obtained, and the mask and outline of the building in the remote sensing building image are predicted respectively, so as to achieve the purpose of identifying the building in the remote sensing building image by combining the predicted mask with the predicted outline, thereby achieving the technical effect of accurately segmenting the buildings in the remote sensing building image, and further solving the technical problem of inaccurate object segmentation in the image segmentation method based on maskrcnn in the related art when the image is a remote sensing building image including buildings.

[0055] As an optional embodiment, the above method may include: predicting the mask of a building in a remotely sensed building image to obtain a first result includes: using the first branch of a dual decoder to predict the mask of the building in the remotely sensed building image to obtain the first result; predicting the outline of the building in the remotely sensed building image to obtain a second result includes: using the second branch of the dual decoder to predict the outline of the building in the remotely sensed building image to obtain the second result.

[0056] As an optional embodiment, the above method may include: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image, and obtaining the first result includes: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image, and obtaining the following prediction results: mask class prediction result, mask boundary prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary, wherein the first result includes: mask class prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary; using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the second result includes: using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the following prediction results: outline class prediction result, outline prediction result, contact outline class prediction result, wherein the second result includes: contact outline class prediction result of the outline class prediction result.

[0057] Figure 4 Schematic diagram of a dual decoder prediction mask and contour according to a preferred embodiment of the present invention, as shown in FIG. Figure 4As shown, the dual decoder structure has two main branches: one predicts the mask and the other predicts the contour. The results obtained by the mask prediction branch include: mask_cls (corresponding to the mask class prediction result mentioned above), mask_reg (corresponding to the mask boundary prediction result mentioned above), and mask_iou (corresponding to the intersection-and-union ratio of the predicted boundary and the actual boundary of the mask mentioned above); the results obtained by the contour prediction branch include: cnt_cls (corresponding to the contour class prediction result mentioned above), cnt_reg (corresponding to the contour prediction result mentioned above), and sep_cls (corresponding to the contact contour class prediction result mentioned above). In this dual decoder, one branch is mainly used to predict segmentation (i.e., the predicted mask mentioned above), and the other branch is mainly used to predict contour. Decoupling the two types of information makes the predictions on segmentation and contour more accurate. The dual decoder can be symmetrical in the upper and lower branches, both formed by feature pyramids.

[0058] It should be noted that among the result parameters obtained by the above-mentioned predicted mask, mask_reg is an auxiliary prediction parameter, which can be used to assist the prediction of mask_iou, and mask_iou is used to indicate the accuracy of the segmentation mask. mask_iou predicts the intersection-over-union ratio between the result of mask_reg and the true value of the annotation (that is, the intersection-over-union ratio of the predicted boundary of the mask referred to above to the actual boundary). Therefore, mask_iou can measure the accuracy of the mask prediction to a certain extent. In addition, among the result parameters obtained by the above-mentioned predicted contour, sep_cls can represent the contact contour (that is, closely adjacent), where the contour includes both contacting and non-contacting contours.

[0059] Figure 5 Schematic diagram of applying watershed post-processing to perform image instance segmentation according to a preferred embodiment of the present invention, as shown in FIG. Figure 5 As shown in the figure, in the output results of the two branches of the dual decoder, mask_cls, cnt_cls, sep_cls, and mask_iou are combined into output, and the output is processed by watershed to obtain the result of instance segmentation.

[0060] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0061] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0062] Example 2

[0063] According to an embodiment of the present invention, there is further provided an image processing device for implementing the above data processing method. Figure 6 : is a structural block diagram of an image processing device according to embodiment 2 of the present invention. Figure 6 As shown, the image processing device 60 includes: a first acquisition module 62, a first prediction module 64 and a first processing module 66. The image processing device 60 is described below.

[0064] A first acquisition module 62 is used to acquire an image to be segmented;

[0065] A first prediction module 64, connected to the first acquisition module 62, configured to predict a mask of an object in an image to obtain a first result, and to predict a contour of an object in an image to obtain a second result;

[0066] The first processing module 66 is connected to the first prediction module 64 and is used to process the first result and the second result to obtain an image segmentation result.

[0067] It should be noted that the first acquisition module 62, the first prediction module 64, and the first processing module 66 correspond to steps S202 to S206 in Example 1. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0068] Example 3

[0069] According to an embodiment of the present invention, a second image processing device for implementing the above data processing method is also provided. Figure 7 : is a structural block diagram of an image processing device 2 according to embodiment 3 of the present invention. Figure 7As shown, the second image processing device 70 includes: a second acquisition module 72, a second prediction module 74 and a second processing module 76. The second image processing device 70 is described below.

[0070] The second acquisition module 72 is configured to acquire a remote sensing building image to be segmented, wherein the remote sensing building image includes an image including a building acquired by remote sensing detection;

[0071] a second prediction module 74, connected to the second acquisition module 72, configured to predict the mask of the building in the remote sensing building image to obtain a first result, and to predict the outline of the building in the remote sensing building image to obtain a second result;

[0072] The second processing module 76 is connected to the second prediction module 74 and is used to process the first result and the second result to obtain a segmentation result of the remote sensing building image.

[0073] It should be noted that the second acquisition module 72, the second prediction module 74, and the second processing module 76 described above correspond to steps S302 to S306 in Example 1. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0074] Example 4

[0075] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0076] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0077] In this embodiment, the computer terminal can execute the program codes of the following steps in the image processing method of the application program:.

[0078] Optionally, Figure 8 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 8 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 82, memory 84, etc.

[0079] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and device in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0080] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the image to be segmented; predict the mask of the object in the image to obtain a first result, and predict the outline of the object in the image to obtain a second result; process the first result and the second result to obtain the image segmentation result.

[0081] Optionally, the above-mentioned processor can also execute the program code of the following steps: predicting the mask of the object in the image to obtain a first result includes: using the first branch of the dual decoder to predict the mask of the object in the image to obtain the first result; predicting the contour of the object in the image to obtain a second result includes: using the second branch of the dual decoder to predict the contour of the object in the image to obtain the second result.

[0082] Optionally, the processor may also execute the program code of the following steps: the first result includes: the mask class prediction result and the intersection-union ratio of the predicted boundary of the mask to the actual boundary; the second result includes: the contour class prediction result and the contact contour class prediction result.

[0083] Optionally, the processor may also execute the program code of the following steps: using the first branch of the dual decoder to predict the mask of the object in the image, and obtaining the first result includes: using the first branch of the dual decoder to predict the mask of the object in the image, and obtaining a mask class prediction result and a mask boundary prediction result; determining the intersection-and-union ratio of the predicted boundary and the actual boundary of the mask based on the mask boundary prediction result; using the second branch of the dual decoder to predict the contour of the object in the image, and obtaining the second result includes: using the second branch of the dual decoder to predict the object of the instance in the image, and obtaining a contour class prediction result and a contour prediction result; determining the contact contour class prediction result based on the contour prediction result.

[0084] Optionally, the above-mentioned processor can also execute the program code of the following steps: processing the first result and the second result to obtain the image segmentation result, including: determining the seed point for image segmentation based on the first result and the second result; segmenting the image according to the seed point to obtain the image segmentation result.

[0085] Optionally, the processor may further execute program code of the following steps: acquiring the image to be segmented includes: acquiring the image to be segmented by remote sensing detection.

[0086] Optionally, the processor may further execute program code of the following steps: the image includes an image of a building.

[0087] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a remote sensing building image to be segmented, wherein the remote sensing building image includes: an image including a building obtained by remote sensing detection; predicting the mask of the building in the remote sensing building image to obtain a first result, and predicting the outline of the building in the remote sensing building image to obtain a second result; processing the first result and the second result to obtain a segmentation result of the remote sensing building image.

[0088] Optionally, the above-mentioned processor can also execute the program code of the following steps: predicting the mask of the building in the remote sensing building image to obtain a first result includes: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image to obtain the first result; predicting the outline of the building in the remote sensing building image to obtain a second result includes: using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image to obtain the second result.

[0089] Optionally, the processor may also execute the program code of the following steps: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image, and obtaining the first result includes: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image, and obtaining the following prediction results: mask class prediction result, mask boundary prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary, wherein the first result includes: mask class prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary; using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the second result includes: using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the following prediction results: outline class prediction result, outline prediction result, contact outline class prediction result, wherein the second result includes: contact outline class prediction result of the outline class prediction result.

[0090] The present invention provides an image processing solution that predicts the mask and contour of the image to be segmented, thereby achieving more accurate identification of objects in the image to be segmented. This solves the technical problem of inaccurate object segmentation in the related art image segmentation method based on MaskRCNN.

[0091] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 8 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 8 Different configurations shown.

[0092] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0093] Example 5

[0094] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method provided in the first embodiment.

[0095] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0096] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining an image to be segmented; predicting a mask of an object in the image to obtain a first result, and predicting a contour of the object in the image to obtain a second result; processing the first result and the second result to obtain a segmentation result of the image.

[0097] Optionally, in this embodiment, the storage medium is further configured to store program code for executing the following steps: predicting the mask of an object in an image to obtain a first result includes: using the first branch of the dual decoder to predict the mask of the object in the image to obtain a first result; predicting the contour of the object in the image to obtain a second result includes: using the second branch of the dual decoder to predict the contour of the object in the image to obtain a second result.

[0098] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: the first result includes: the mask class prediction result and the intersection-union ratio of the predicted boundary of the mask to the actual boundary; the second result includes: the contour class prediction result and the contact contour class prediction result.

[0099] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: using the first branch of the dual decoder to predict the mask of the object in the image, and obtaining a first result includes: using the first branch of the dual decoder to predict the mask of the object in the image, and obtaining a mask class prediction result and a mask boundary prediction result; determining the intersection-and-union ratio of the predicted boundary and the actual boundary of the mask based on the mask boundary prediction result; using the second branch of the dual decoder to predict the contour of the object in the image, and obtaining a second result includes: using the second branch of the dual decoder to predict the contour of the object in the image, and obtaining a contour class prediction result and a contour prediction result; determining a contact contour class prediction result based on the contour prediction result.

[0100] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: processing the first result and the second result to obtain an image segmentation result, including: determining a seed point for image segmentation based on the first result and the second result; segmenting the image according to the seed point, and processing to obtain an image segmentation result.

[0101] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: acquiring the image to be segmented includes: acquiring the image to be segmented by remote sensing detection.

[0102] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: the above-mentioned image includes: an image including a building.

[0103] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a remote sensing building image to be segmented, wherein the remote sensing building image includes: an image including a building obtained by remote sensing detection; predicting the mask of the building in the remote sensing building image to obtain a first result, and predicting the outline of the building in the remote sensing building image to obtain a second result; processing the first result and the second result to obtain a segmentation result of the remote sensing building image.

[0104] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: predicting the mask of a building in a remotely sensed building image to obtain a first result includes: using the first branch of a dual decoder to predict the mask of the building in the remotely sensed building image to obtain a first result; predicting the outline of the building in the remotely sensed building image to obtain a second result includes: using the second branch of the dual decoder to predict the outline of the building in the remotely sensed building image to obtain a second result.

[0105] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image, and obtaining the first result includes: using the first branch of the dual decoder to predict the mask of the building in the remote sensing building image, and obtaining the following prediction results: mask class prediction result, mask boundary prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary, wherein the first result includes: mask class prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary; using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the second result includes: using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the following prediction results: outline class prediction result, outline prediction result, contact outline class prediction result, wherein the second result includes: contact outline class prediction result of the outline class prediction result.

[0106] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0107] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0109] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0112] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Obtaining an image to be segmented; Predicting a mask of an object in the image to obtain a first result, and predicting a contour of the object in the image to obtain a second result, wherein the first result includes: a mask-type prediction result and an intersection-over-union ratio of a predicted boundary of the mask to an actual boundary, and the second result includes: a contour-type prediction result and a contact contour-type prediction result, and the first result and the second result are predicted based on different branches of the same decoder; Determining a seed point for image segmentation based on the first result and the second result; The image is segmented according to the seed points to obtain a segmentation result of the image.

2. The method according to claim 1, characterized in that Predicting the mask of the object in the image to obtain the first result includes: predicting the mask of the object in the image using a first branch of a dual decoder to obtain the first result; Predicting the contour of the object in the image to obtain the second result includes: using the second branch of the dual decoder to predict the contour of the object in the image to obtain the second result.

3. The method according to claim 1, characterized in that Predicting the mask of the object in the image using the first branch of the dual decoder to obtain the first result includes: predicting the mask of the object in the image using the first branch of the dual decoder to obtain a mask class prediction result and a mask boundary prediction result; and determining an intersection-over-union ratio of a predicted boundary and an actual boundary of the mask based on the mask boundary prediction result; Using the second branch of the dual decoder to predict the contour of the object in the image to obtain the second result includes: using the second branch of the dual decoder to predict the contour of the object in the image to obtain a contour class prediction result and a contour prediction result; determining the contact contour class prediction result based on the contour prediction result.

4. The method according to any one of claims 1 to 3, characterized in that Obtaining the image to be segmented includes: The image to be segmented is acquired by remote sensing.

5. The method according to claim 4, characterized in that The images include: images of buildings.

6. An image processing method, characterized in that: include: Acquire a remote sensing building image to be segmented, wherein the remote sensing building image includes: an image including a building acquired by remote sensing detection; Predicting a mask of a building in the remotely sensed building image to obtain a first result, and predicting a contour of the building in the remotely sensed building image to obtain a second result, wherein the first result includes: a mask-type prediction result and an intersection-over-union ratio of a predicted boundary of the mask to an actual boundary, and the second result includes: a contour-type prediction result and a contact contour-type prediction result, and the first result and the second result are predicted based on different branches of the same decoder; Determining a seed point for image segmentation based on the first result and the second result; The remote sensing building image is segmented according to the seed points to obtain a segmentation result of the remote sensing building image.

7. The method according to claim 6, characterized in that Predicting the mask of the building in the remote sensing building image to obtain the first result includes: predicting the mask of the building in the remote sensing building image using a first branch of a dual decoder to obtain the first result; Predicting the outline of the building in the remotely sensed building image to obtain the second result includes: using the second branch of the dual decoder to predict the outline of the building in the remotely sensed building image to obtain the second result.

8. The method according to claim 6, characterized in that Predicting the mask of the building in the remote sensing building image using the first branch of the dual decoder to obtain the first result includes: predicting the mask of the building in the remote sensing building image using the first branch of the dual decoder to obtain the following prediction results: a mask class prediction result, a mask boundary prediction result, and an intersection-and-union ratio of the predicted boundary of the mask to the actual boundary, wherein the first result includes: the mask class prediction result and the intersection-and-union ratio of the predicted boundary of the mask to the actual boundary; Using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the second result includes: using the second branch of the dual decoder to predict the outline of the building in the remote sensing building image, and obtaining the following prediction results: outline class prediction result, outline prediction result, contact outline class prediction result, wherein the second result includes: contact outline class prediction result of the outline class prediction result.

9. An image processing device, characterized in that: include: A first acquisition module is used to acquire an image to be segmented; a first prediction module, configured to predict a mask of an object in the image to obtain a first result, and predict a contour of the object in the image to obtain a second result, wherein the first result includes: a mask-type prediction result and an intersection-over-union ratio of a predicted mask boundary to an actual mask boundary, and the second result includes: a contour-type prediction result and a contact contour-type prediction result, wherein the first result and the second result are predicted based on different branches of the same decoder; The first processing module is used to determine a seed point for image segmentation according to the first result and the second result; and segment the image according to the seed point to obtain a segmentation result of the image.

10. An image processing device, characterized in that: include: A second acquisition module is configured to acquire a remote sensing building image to be segmented, wherein the remote sensing building image includes an image including a building acquired by remote sensing detection; a second prediction module, configured to predict a mask of a building in the remotely sensed building image to obtain a first result, and predict a contour of the building in the remotely sensed building image to obtain a second result, wherein the first result includes: a mask-type prediction result and an intersection-over-union ratio of a predicted boundary of the mask to an actual boundary; and the second result includes: a contour-type prediction result and a contact contour-type prediction result, and the first and second results are predicted based on different branches of the same decoder; The second processing module is used to determine the seed points for image segmentation according to the first result and the second result; and segment the remote sensing building image according to the seed points to obtain the segmentation result of the remote sensing building image.

11. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 8.

12. A computer device, characterized in that: include: memory and processor, The memory stores a computer program; The processor is configured to execute a computer program stored in the memory, and when the computer program is executed, the processor is enabled to execute the image processing method according to any one of claims 1 to 8.

Images