Instance segmentation method, device, electronic device and storage medium

Abstract

The present application provides an instance segmentation method, device, electronic device and computer-readable storage medium. The instance segmentation method includes: obtaining a first subnetwork, a second subnetwork and a target image containing an object to be segmented in a trained instance segmentation model; converting the first subnetwork and the second subnetwork respectively to obtain a first accelerated execution model and a second accelerated execution model; calling the first accelerated execution model to perform feature extraction on the target image to obtain first feature data of the target image; performing prediction processing based on the first feature data to obtain second feature data of the target image; calling the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image. The present application can avoid the problem that a high-precision instance segmentation model cannot be directly loaded onto an inference optimizer, and on the basis of ensuring segmentation accuracy, the instance segmentation model can be improved in real time.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an instance segmentation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Computer vision uses various imaging systems to replace visual organs as input sensitive means, and computers replace the brain to complete processing and interpretation.

[0003] Among them, instance segmentation is a common processing task in computer vision. Instance segmentation is to distinguish and label each object in an image. Instance segmentation has important guiding significance in the fields of behavior recognition, public safety, video surveillance, etc.

[0004] Usually, a deep learning network model is used to implement instance segmentation. However, in the prior art, some instance segmentation models with higher accuracy (for example, the Yolact++ model) have a slow reasoning speed, resulting in low real-time performance of image instance segmentation. Summary of the invention

[0005] The present application provides an instance segmentation method, device, electronic device and computer-readable storage medium, aiming to solve the problem of low real-time instance segmentation of images due to slow inference speed of a high-precision instance segmentation model.

[0006] In a first aspect, the present application provides an instance segmentation method, the method comprising:

[0007] Obtaining a first subnetwork in a trained instance segmentation model, a second subnetwork in the trained instance segmentation model, and a target image containing an object to be segmented;

[0008] Performing conversion processing on the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network;

[0009] Calling the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image;

[0010] Performing prediction processing according to the first feature data to obtain second feature data of the target image;

[0011] The second accelerated execution model is called to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image.

[0012] In a second aspect, the present application provides an instance segmentation device, the instance segmentation device comprising:

[0013] An acquisition unit, configured to acquire a first subnetwork in a trained instance segmentation model, a second subnetwork in the trained instance segmentation model, and a target image containing an object to be segmented;

[0014] a conversion unit, configured to convert the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network;

[0015] A segmentation processing unit, configured to call the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image;

[0016] The segmentation processing unit is further used to perform prediction processing according to the first feature data to obtain second feature data of the target image;

[0017] The segmentation processing unit is further used to call the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image.

[0018] In a possible implementation of the present application, the conversion unit is further configured to:

[0019] The first sub-network and the second sub-network are converted respectively to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network;

[0020] According to a preset inference optimizer, the first ONNX model is converted to obtain a first accelerated execution model corresponding to the first sub-network, and the second ONNX model is converted to obtain a second accelerated execution model corresponding to the second sub-network.

[0021] In a possible implementation of the present application, the conversion unit is further configured to:

[0022] Acquire a first model parameter of the first sub-network and a second model parameter of the second sub-network;

[0023] The first model parameters are loaded into the first benchmark model of the inference optimizer to obtain a first ONNX model corresponding to the first sub-network; the second model parameters are loaded into the second benchmark model of the inference optimizer to obtain a second ONNX model corresponding to the second sub-network.

[0024] In a possible implementation of the present application, the second ONNX model and the second accelerated execution model are both set to dynamic input and output.

[0025] In a possible implementation of the present application, the segmentation processing unit is further configured to:

[0026] Initializing the first accelerated execution model;

[0027] The target image is input into the first accelerated execution model, so as to call the first accelerated execution model to perform feature extraction processing on the target image and obtain first feature data of the target image.

[0028] In a possible implementation of the present application, the segmentation processing unit is further configured to:

[0029] Performing instance segmentation prediction processing according to the first feature data to obtain preliminary segmentation information of the target image;

[0030] The preliminary segmentation information is used as second feature data of the target image.

[0031] In a possible implementation of the present application, the segmentation processing unit is further configured to:

[0032] Initializing the second accelerated execution model;

[0033] Inputting the preliminary segmentation information into the second accelerated execution model, so as to call the second accelerated execution model to perform confidence prediction processing on the preliminary segmentation information, and obtain a first confidence of the preliminary segmentation information;

[0034] An instance segmentation result of the target image is determined according to the first confidence level.

[0035] In a possible implementation of the present application, the instance segmentation apparatus further includes a training unit, and the training unit is specifically configured to:

[0036] Acquire a plurality of training images, wherein the training images are labeled with segmentation contour information;

[0037] Calling a first subnetwork in a preset instance segmentation network to perform feature extraction processing on the training image to obtain image feature information of the training image;

[0038] Performing instance segmentation prediction processing according to the image feature information to obtain a first segmentation prediction result of the training image and a second confidence level of the first segmentation prediction result;

[0039] Calling a second subnetwork in a preset instance segmentation network to perform confidence prediction processing on the first segmentation prediction result to obtain a third confidence of the preliminary segmentation information;

[0040] Determining a second segmentation prediction result of the training image according to the third confidence level;

[0041] The preset instance segmentation network is trained according to the second segmentation prediction result and the segmentation contour information to obtain a trained instance segmentation model.

[0042] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the processor executes the steps of any instance segmentation method provided in the present application.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in the instance segmentation method.

[0044] This application splits the instance segmentation model into multiple sub-networks (i.e., the first sub-network and the second sub-network) according to the network structure of the instance segmentation model, and then converts each sub-network into an accelerated execution model (i.e., the first accelerated execution model and the second accelerated execution model), so that the sub-network of the instance segmentation model can be loaded onto the inference optimizer (such as TensorRT). And by calling the first accelerated execution model to perform feature extraction processing on the target image to obtain the first feature data of the target image, and calling the second accelerated execution model to perform prediction processing on the second feature data to obtain the instance segmentation result of the target image, the data processing process of the sub-network of the instance segmentation model can be accelerated, thereby achieving the speed-up of the instance segmentation model, and then improving the real-time performance of the instance segmentation model. The problem that the high-precision instance segmentation model cannot be directly loaded onto the inference optimizer is avoided, so that the instance segmentation model can be improved in real time on the basis of ensuring the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a flow chart of an instance segmentation method provided in an embodiment of the present application;

[0047] Figure 2 is a scene schematic diagram of a target image provided in an embodiment of the present application;

[0048] Figure 3 It is a schematic diagram of an embodiment of the refinement of step S20 provided in the embodiments of the present application;

[0049] Figure 4 It is a schematic diagram of an embodiment of the refinement of step S21 provided in the embodiments of the present application;

[0050] Figure 5 It is a flow chart of loading the instance segmentation model provided in the embodiment of the present application into the inference optimizer TensorRT to realize instance segmentation;

[0051] Figure 6 is a schematic diagram of the structure of an embodiment of an instance segmentation device provided in an embodiment of the present application;

[0052] Figure 7 It is a schematic diagram of the structure of an embodiment of an electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0054] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0055] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0056] The embodiments of the present application provide an instance segmentation method, device, electronic device and computer-readable storage medium. The instance segmentation device can be integrated in an electronic device, which can be a server or a terminal.

[0057] First of all, before introducing the embodiments of the present application, the relevant content about the application background of the embodiments of the present application is introduced.

[0058] With the development of artificial intelligence technology, deep learning models have been increasingly used in practical scenarios. However, for some scenarios with high real-time requirements, the current reasoning speed of deep learning models is limited, and many models that meet the accuracy requirements cannot meet the reasoning speed requirements.

[0059] For example, the Yolact++ model is an improved instance segmentation algorithm for the Yolact model. It improves the algorithm's detection and segmentation capabilities for target objects by adding variable convolutions, modifying the default box settings, and adding a re-scoring network. The Yolact++ model has good performance in both detection and segmentation, but if the Yolact++ model is directly deployed on a machine, there will be a problem of slow inference speed.

[0060] The Yolact++ model includes a backbone network and a rescoring network. The backbone network and the rescoring network are not seamlessly connected, that is, the output of the backbone network is not directly used as the input of the rescoring network. Instead, the output of the backbone network is decoded (Decode), NMS (Non-Maximum Suppression) and other post-processing, and then the post-processing results are used as the input of the rescoring network.

[0061] TensorRT is a high-performance deep learning inference optimizer. By converting the trained algorithm model to TensorRT, the inference speed of the model can be greatly improved, making the model suitable for occasions with high real-time requirements.

[0062] However, since the backbone network and the rescoring network of the Yolact++ model are not seamlessly connected, there is no corresponding plug-in implementation in TensorRT for decoding, NMS and other post-processing of the backbone network output results. Therefore, it is impossible to convert the Yolact++ model to TensorRT, resulting in the inability to speed up the Yolact++ model.

[0063] Based on the above-mentioned defects of the existing related technologies, an embodiment of the present application provides an instance segmentation method, which can load an instance segmentation model with non-seamless connection between sub-networks (that is, there is a post-processing process between the two sub-networks) into a deep learning inference optimizer to achieve reasoning acceleration of the instance segmentation model, at least to a certain extent overcoming the defects of the existing related technologies.

[0064] The executor of the instance segmentation method of the embodiment of the present application may be the instance segmentation device provided in the embodiment of the present application, or different types of electronic devices such as a server device, a physical host or a user equipment (UE) that integrates the instance segmentation device, wherein the instance segmentation device may be implemented in hardware or software, and the UE may specifically be a terminal device such as a smart phone, a tablet computer, a laptop computer, a PDA, a desktop computer or a personal digital assistant (PDA).

[0065] The electronic device can adopt a working mode of independent operation or a working mode of a device cluster. By applying the instance segmentation method provided in the embodiment of the present application, the problem that a high-precision instance segmentation model cannot be directly loaded onto the inference optimizer can be avoided. Thus, on the basis of ensuring the segmentation accuracy, the instance segmentation model can be improved in real time.

[0066] Next, the instance segmentation method provided in the embodiment of the present application is introduced. In the embodiment of the present application, an electronic device is used as an execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0067] Reference Figure 1 , Figure 1 is a flow chart of an instance segmentation method provided by an embodiment of the present application. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here. The instance segmentation method includes steps S10 to S50, wherein:

[0068] S10, obtaining a first sub-network in a trained instance segmentation model, a second sub-network in the trained instance segmentation model, and a target image containing an object to be segmented.

[0069] In order to better understand the embodiments of the present application, the following first introduces the concepts of "target image containing objects to be segmented", "first sub-network, second sub-network", and "post-processing process between sub-networks".

[0070] In the first aspect, a target image including an object to be segmented is provided.

[0071] The target image refers to an image containing the object to be segmented.

[0072] Please refer to Figure 2 , Figure 2 is a scene schematic diagram of a target image provided in an embodiment of the present application. Figure 2 (a) shows the target image where the object to be segmented is a person. Figure 2(b) shows the target image where the object to be segmented is a dog.

[0073] The objects to be segmented can be defined according to specific application scenarios. For example, in order to count people based on an image, it is necessary to first segment the mask area of ​​each person from the image, and the objects to be segmented are people. For another example, in order to detect the breed of dogs in an image, it is necessary to first segment the mask area of ​​each dog from the image, and the objects to be segmented are dogs.

[0074] There are many ways to obtain a target image containing an object to be segmented. For example, it can be obtained from a preset image database. For another example, an electronic device that applies the instance segmentation method provided in an embodiment of the present application may directly include a camera (such as a camera for photographing people in a public area) in hardware, and store the image captured by the camera locally, which can be directly read inside the electronic device; or, the electronic device may also establish a network connection with the camera, and obtain the image captured by the camera online from the camera according to the network connection; or, the electronic device may also read the image captured by the camera from a related storage medium storing the image captured by the camera, and the specific acquisition method is not limited here.

[0075] On the second aspect, the first sub-network and the second sub-network.

[0076] Among them, the first subnetwork and the second subnetwork both refer to the subnetworks in the trained instance segmentation model (hereinafter referred to as the instance segmentation model. Unless otherwise specified, the instance segmentation model referred to herein refers to the trained instance segmentation model), and the first subnetwork and the second subnetwork may be in a sequentially linked relationship. The purpose of distinguishing the first subnetwork and the second subnetwork here is: on the one hand, it is convenient to distinguish two different subnetworks, and on the other hand, it is convenient to associate the input data, output data, and post-processing data of the subnetworks. Among them, the first subnetwork and the second subnetwork are non-seamlessly connected, that is, there is a post-processing process between the first subnetwork and the second subnetwork, that is, step S40 is the post-processing process between the first subnetwork and the second subnetwork.

[0077] It is understandable that, although in the embodiment of the present application, the instance segmentation model is split into two sub-networks to achieve instance segmentation acceleration. However, the instance segmentation model can also be split into two or more sub-networks according to the network structure of the instance segmentation model to achieve instance segmentation acceleration. For example, the instance segmentation model is split into a first sub-network, a second sub-network, and a third sub-network, and there is a post-processing process between the first sub-network and the second sub-network, and there is a post-processing process between the second sub-network and the third sub-network.

[0078] For example, “obtaining a first subnetwork in a trained instance segmentation model and a second subnetwork in the trained instance segmentation model” may include: first, using the trained Yolact++ model as the trained instance segmentation model, and splitting the Yolact++ model into a backbone network and a rescoring network according to the network structure of the Yolact++ model. Then, using the backbone network as the first subnetwork of the trained instance segmentation model and using the rescoring network as the second subnetwork of the trained instance segmentation model.

[0079] The Yolact++ model is only an example of an instance segmentation model. Other types of trained models may be selected as the trained instance segmentation model according to actual conditions, but the present invention is not limited thereto.

[0080] The third aspect is the post-processing process between sub-networks.

[0081] The post-processing process between sub-networks is hereinafter referred to as the post-processing process, or also called the post-processing process of the instance segmentation model.

[0082] In some embodiments of the present application, the post-processing process between the first sub-network and the second sub-network is: after the first sub-network outputs the feature data of the target image, the instance segmentation prediction process is performed according to the feature data of the target image output by the first sub-network (specifically, it may include decoding, NMS, etc.) to obtain the preliminary segmentation information of the target image. Finally, the preliminary segmentation information of the target image is input into the second sub-network.

[0083] It can be understood that the post-processing process (instance segmentation prediction processing, decoding processing, NMS and other processing) between the first sub-network and the second sub-network here is only an example, and the post-processing process between the first sub-network and the second sub-network can be changed according to the processing requirements of the specific application scenario and the network structure of the instance segmentation model. For example, the first sub-network is used to output the feature information of the target image, and the post-processing process is to predict the detection box information of the target image based on the feature information of the target image, and the second sub-network is used to perform instance segmentation prediction processing based on the detection box information to obtain the instance segmentation result of the target image. For another example, the first sub-network is used to output the feature information of the target image, and the post-processing process is to predict the preliminary segmentation information of the target image based on the feature information of the target image, and the second sub-network is used to perform confidence prediction processing on the preliminary segmentation information to obtain the confidence of the preliminary segmentation information; so that the instance segmentation result of the target image can be determined later according to the confidence of the preliminary segmentation information.

[0084] S20. Convert the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network.

[0085] The first accelerated execution model refers to the execution engine corresponding to the first sub-network function implementation after the first sub-network is loaded into the inference optimizer. The second accelerated execution model refers to the execution engine corresponding to the second sub-network function implementation after the second sub-network is loaded into the inference optimizer.

[0086] Since the sub-networks of the instance segmentation model are not seamlessly connected, the entire instance segmentation model cannot be directly loaded into the inference optimizer. Therefore, in the embodiment of the present application, each sub-network after being split in the instance segmentation model is converted separately through step S20 to form each execution engine of the inference optimizer (hereinafter referred to as the Engine model). In order to enable the subsequent direct call of each execution engine of the inference optimizer to implement the functions of each sub-network of the instance segmentation network, thereby realizing the acceleration of the instance segmentation model and improving the real-time performance of instance segmentation.

[0087] Please refer to Figure 3 , Figure 3 It is a schematic diagram of an embodiment of the refinement of step S20 provided in the embodiments of the present application.

[0088] In order to enable each sub-network of the instance segmentation model to be normally loaded into the inference optimizer, the following steps S21 to S22 are implemented in the embodiment of the present application, that is, step S20 may specifically include the following steps S21 to S22, wherein:

[0089] S21. According to a preset inference optimizer, the first sub-network and the second sub-network are converted and processed respectively to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network.

[0090] The ONNX (Open Neural Network Exchange) model is an open neural network exchange tool, which means that no matter what framework is used to train the model, as long as the trained model is converted into an ONNX model, it can be placed on other frameworks to perform the reasoning process.

[0091] The first ONNX model refers to the ONNX model obtained after conversion by the first sub-network and can be placed on the inference optimizer to perform the inference process.

[0092] The second ONNX model refers to an ONNX model that is converted from the second sub-network and can be placed on the inference optimizer to perform the inference process.

[0093] Among them, the first sub-network can be converted into a first ONNX model and the second sub-network can be converted into a second ONNX model by referring to some existing implementation methods of “converting the model into an ONNX model after training”. The existing implementation methods of “converting the model into an ONNX model after training” will not be repeated here.

[0094] Please refer to Figure 4 , Figure 4 It is a schematic diagram of an embodiment of the refinement of step S21 provided in the embodiments of the present application.

[0095] In some embodiments, step S21 may specifically include the following steps S211 to S212, wherein:

[0096] S211. Obtain a first model parameter of the first sub-network and a second model parameter of the second sub-network.

[0097] The first model parameter refers to the model parameter corresponding to the first sub-network part of the trained instance segmentation network.

[0098] The second model parameters refer to the model parameters corresponding to the second sub-network part of the trained instance segmentation network.

[0099] For example, the backbone network of the trained Yolact++ model is used as the first sub-network, and the re-scoring network of the trained Yolact++ model is used as the second sub-network. Then the first model parameters refer to the model parameters of the backbone network of the trained Yolact++ model, and the second model parameters refer to the model parameters of the re-scoring network of the trained Yolact++ model.

[0100] S212. Load the first model parameters into the first benchmark model of the inference optimizer to obtain a first ONNX model corresponding to the first sub-network; load the second model parameters into the second benchmark model of the inference optimizer to obtain a second ONNX model corresponding to the second sub-network.

[0101] The first benchmark model and the second benchmark model both refer to the benchmark ONNX models of the preset inference optimizer, that is, the ONNX models that can be placed on the inference optimizer to perform the inference process and whose parameters are default values. The difference between the first benchmark model and the second benchmark model is that the first benchmark model is an ONNX model after the first sub-network is initialized using the default model parameters. The second benchmark model is an ONNX model after the second sub-network is initialized using the default model parameters.

[0102] For ease of understanding, the explanation will continue with the example of step S211 above.

[0103] For example, the model parameters of the backbone network of the trained Yolact++ model are loaded into the first benchmark model of the inference optimizer (such as TensorRT) to obtain the first ONNX model corresponding to the first sub-network.

[0104] The model parameters of the rescoring network of the trained Yolact++ model are loaded into the second benchmark model of the inference optimizer (such as TensorRT) to obtain the second ONNX model corresponding to the second subnetwork.

[0105] TensorRT is a high-performance deep learning inference optimizer that can provide low-latency, high-throughput deployment inference for deep learning applications. TensorRT can be used to accelerate inference in hyperscale data centers, embedded platforms, or autonomous driving platforms. TensorRT now supports almost all deep learning frameworks such as TensorFlow, Caffe, Mxnet, Pytorch, etc. Combining TensorRT with NVIDIA GPUs can quickly and efficiently deploy inference in almost all frameworks.

[0106] The above TensorRT is only an example of a preset inference optimizer. The preset inference optimizer can be selected according to actual conditions and is not limited to this.

[0107] From the above, it can be seen that since the trained model is converted into an ONNX model, it can be placed on other frameworks to perform the inference process. By loading the first model parameters into the first benchmark model of the inference optimizer, the first ONNX model corresponding to the first sub-network is obtained; it is ensured that the first sub-network can be loaded normally onto the inference optimizer. By loading the second model parameters into the second benchmark model of the inference optimizer, the second ONNX model corresponding to the second sub-network is obtained; it is ensured that the second sub-network can be loaded normally onto the inference optimizer.

[0108] S22. Convert the first ONNX model to obtain a first accelerated execution model corresponding to the first sub-network, and convert the second ONNX model to obtain a second accelerated execution model corresponding to the second sub-network.

[0109] In some embodiments, the execution of steps S30 and S50 is completed by calling the inference optimizer, and step S40 (i.e., the post-processing process) is performed by the electronic device. At this time, it is only necessary to convert the first ONNX model and the second ONNX model into two execution engines (referred to as Engine models) of the inference optimizer.

[0110] For ease of understanding, the explanation will continue with the example of step S212 above.

[0111] For example, on the one hand, the first ONNX model corresponding to the first sub-network (i.e., the backbone network) is converted into an execution engine (referred to as the Engine model) of the inference optimizer (TensorRT); and the Engine model after the first ONNX model is converted is used as the first accelerated execution model corresponding to the first sub-network. On the other hand, the second ONNX model corresponding to the second sub-network (i.e., the rescoring network) is converted into another execution engine (referred to as the Engine model) of the inference optimizer (TensorRT); and the Engine model after the second ONNX model is converted is used as the second accelerated execution model corresponding to the second sub-network.

[0112] In some embodiments, the execution of steps S30 to S50 is all completed by calling the inference optimizer, that is, the post-processing process (step S40) is also completed by calling the inference optimizer.

[0113] Since the sub-networks of the instance segmentation model are not seamlessly connected, the entire instance segmentation model cannot be directly loaded into the inference optimizer, that is, the post-processing process of the instance segmentation model has no corresponding plug-in implementation in the inference optimizer.

[0114] To this end, further, a plug-in (referred to as plug-in) for the post-processing process in the optimization accelerator is written. That is, "converting the first ONNX model to obtain the first accelerated execution model corresponding to the first sub-network" can specifically include: first, according to the initialization parameters of the post-processing process, obtain the plug-in implemented in the optimization accelerator for the post-processing process. Then, connect the input of the plug-in to the output of the first ONNX model. Finally, convert the first ONNX model and the plug-in to form an execution engine of the inference optimizer; and use the execution engine as the first accelerated execution model corresponding to the first sub-network.

[0115] Please refer to Figure 5 , Figure 5It is a flow chart of loading the instance segmentation model provided in the embodiment of the present application into the inference optimizer TensorRT to realize instance segmentation. For example, the post-processing process between the backbone network and the re-scoring network of the Yolact++ model is: decoding, NMS processing, etc. First, according to the initialization parameters of the post-processing process, write the plug-in of the post-processing process (decoding, NMS processing, etc.) of the Yolact++ model in TensorRT, connect the input of the plug-in to the output of the first ONNX model (corresponding to the backbone network), and connect the output of the plug-in to the input of the second ONNX model (corresponding to the re-scoring network). Then, the first ONNX model (corresponding to the backbone network) and the plug-in are converted into an execution engine (Engine model) of TensorRT, and the second ONNX model (corresponding to the re-scoring network) is converted into another execution engine (Engine model) of TensorRT. Make the post-processing process (decoding, NMS processing, etc.) all executed on Tensorrt. So that when the subsequent steps S30~S50 perform instance segmentation prediction processing, two Engine models are loaded at the same time for a serial reasoning.

[0116] From the above content, it can be seen that the first accelerated execution model corresponding to the first subnetwork is obtained by converting the first subnetwork, and the second accelerated execution model corresponding to the second subnetwork is obtained by converting the second subnetwork; thereby, the first subnetwork and the second subnetwork of the instance segmentation model can be loaded onto the inference optimizer, thereby speeding up the instance segmentation model to improve the real-time performance of the instance segmentation model.

[0117] S30. Call the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image.

[0118] The first feature data refers to the data outputted after the first accelerated execution model performs feature extraction processing on the target image. It is understandable that the first feature data can adapt to changes according to the network function corresponding to the first sub-network.

[0119] For example, the Yolact++ model can be divided into a backbone network (used to output preliminary segmentation information of the target image) and a rescoring network. Therefore, the first accelerated execution model corresponding to the backbone network is also used to output preliminary segmentation information of the target image. At this time, the first feature data specifically refers to the preliminary segmentation information of the target image.

[0120] In some embodiments, step S30 may specifically include: initializing the first accelerated execution model; inputting the target image into the first accelerated execution model to call the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image.

[0121] From the above content, it can be seen that since the first accelerated execution model is the execution engine after the first sub-network is loaded into the inference optimizer, by calling the first accelerated execution model to perform feature extraction processing on the target image and obtaining the first feature data of the target image, the data processing speed of the first sub-network can be improved, thereby improving the instance segmentation speed of the instance segmentation model, that is, improving the real-time performance of the instance segmentation model.

[0122] S40: Perform prediction processing according to the first feature data to obtain second feature data of the target image.

[0123] The second characteristic data refers to the data output after the post-processing process. It is understandable that the second characteristic data can adapt to changes according to the data output corresponding to the post-processing process.

[0124] In some embodiments, step S40 (i.e., the post-processing process) is not loaded onto the inference optimizer for implementation. At this time, step S40 may specifically include: first, according to the initialization parameters of the post-processing process, obtain a plug-in for implementing the post-processing process in the optimization accelerator. Then, call the plug-in to perform prediction processing based on the first feature data (the prediction processing described here refers to the post-processing process between sub-networks, which can be specifically referred to in the aforementioned step S10 and the relevant descriptions thereafter, which will not be repeated here), and obtain the second feature data of the target image.

[0125] In some embodiments, step S40 (i.e., post-processing) can also be performed on the inference optimizer, that is, step S40 can also be completed by calling the inference optimizer. At this time, the first accelerated execution model refers to an execution engine of the inference optimizer formed by converting the first ONNX model and plug-in. Steps S30 to S40 can be completed by the first accelerated execution model.

[0126] For example, the first accelerated execution model is an execution engine converted from the first ONNX model (corresponding to the backbone network) and the plug-in of the Yolact++ model to TensorRT. Then steps S30 to S40 can be specifically implemented by the following steps (1) to (6):

[0127] (1) Initialize the first accelerated execution model.

[0128] (2) Read the target image, perform image preprocessing operations such as fixing the size, subtracting the mean, dividing by the variance, and converting the BGR image mode to the RGB image mode, and convert the processed image data into pointer data.

[0129] (3) Define the data pointers output after backbone network decoding and NMS processing, including confidence pointer, detection box coordinate pointer, and segmentation mask pointer, and apply for the GPU video memory required for the output pointers respectively.

[0130] (4) The input data pointer, confidence pointer, detection box coordinate pointer, and segmentation mask pointer are combined together to form an inference buffer for the backbone network and post-processing process.

[0131] (5) Input the inference buffer of the backbone network and post-processing process, perform inference on the backbone network, decoding, and NMS processing, and obtain preliminary segmentation information of the target image.

[0132] (6) The preliminary segmentation information is used as the second feature data of the target image.

[0133] S50: Call the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image.

[0134] For ease of understanding, the above example of step S40 is continued. For example, the second accelerated execution model is an execution engine converted from the second ONNX model (corresponding to the rescoring network) of the Yolact++ model to TensorRT. Then step S50 can be specifically implemented by the following steps (7) to (10):

[0135] (7) Initialize the second accelerated execution model.

[0136] (8) Define the input and output pointers of the rescoring network, and apply for the GPU memory required by the input and output pointers respectively. According to the confidence output by the post-processing process, copy the segmentation mask GPU data corresponding to the confidence threshold that is greater than the set confidence threshold to the input pointer of the rescoring network, and combine the input and output pointers of the rescoring network to form the inference buffer of the rescoring network.

[0137] (9) Input the inference buffer of the rescoring network and perform inference on the rescoring network.

[0138] (10) The confidence, detection box coordinates, segmentation mask and re-scoring score output by the post-processing output are copied to the CPU for simple judgment to obtain the final output, that is, the instance segmentation result of the target image.

[0139] From the above content, it can be seen that in the embodiment of the present application, the instance segmentation model is split into multiple sub-networks (i.e., the first sub-network and the second sub-network) according to the network structure of the instance segmentation model, and each sub-network is converted into an accelerated execution model (i.e., the first accelerated execution model and the second accelerated execution model), so that the sub-network of the instance segmentation model can be loaded onto the inference optimizer (such as TensorRT). And by calling the first accelerated execution model to perform feature extraction processing on the target image, the first feature data of the target image is obtained, and the second accelerated execution model is called to perform prediction processing on the second feature data to obtain the instance segmentation result of the target image, so that the data processing process of the sub-network of the instance segmentation model can be accelerated, thereby realizing the speed-up of the instance segmentation model, and then improving the real-time performance of the instance segmentation model. The problem that the high-precision instance segmentation model cannot be directly loaded onto the inference optimizer is avoided, so that the instance segmentation model can be improved in real time on the basis of ensuring the segmentation accuracy.

[0140] In some embodiments, the first subnetwork is used to perform feature extraction processing on the image to obtain feature data of the image. The post-processing process is to decode, perform NMS and other processing on the output result of the first subnetwork (i.e., the feature data of the image) to obtain preliminary segmentation information of the image. The second subnetwork is used to perform confidence prediction processing on the preliminary segmentation information to obtain the confidence of the preliminary segmentation information; so that the instance segmentation result of the image can be determined later according to the confidence of the preliminary segmentation information. The first feature data refers to the feature information of the target image, and the second feature data refers to the preliminary segmentation information of the target image. Among them, the first feature data can be obtained by performing primary and multi-level convolution feature extraction and fusion on the target image.

[0141] At this time, step S40 may include: performing instance segmentation prediction processing according to the first feature data to obtain preliminary segmentation information of the target image; and using the preliminary segmentation information as second feature data of the target image.

[0142] The implementation of "performing instance segmentation prediction processing according to the first feature data to obtain preliminary segmentation information of the target image; and using the preliminary segmentation information as the second feature data of the target image" can be specifically referred to the example and steps (1) to (6) in the above step S40, as well as steps A2 to A5 of the instance segmentation model training process, which will not be repeated here.

[0143] Correspondingly, at this time, step 50 may specifically include: initializing the second accelerated execution model; inputting the preliminary segmentation information into the second accelerated execution model to call the second accelerated execution model to perform confidence prediction processing on the preliminary segmentation information to obtain a first confidence of the preliminary segmentation information; and determining an instance segmentation result of the target image according to the first confidence.

[0144] The implementation of "initializing the second accelerated execution model; inputting the preliminary segmentation information into the second accelerated execution model to call the second accelerated execution model to perform confidence prediction processing on the preliminary segmentation information to obtain a first confidence of the preliminary segmentation information; determining an instance segmentation result of the target image according to the first confidence" can be specifically referred to the example and steps (7) to (10) in the above step S50, as well as steps A2 to A5 of the instance segmentation model training process, which will not be repeated here.

[0145] From the above content, it can be seen that since the second accelerated execution model is the execution engine after the second sub-network is loaded into the inference optimizer, the first confidence of the preliminary segmentation information is obtained by calling the second accelerated execution model to perform confidence prediction processing on the preliminary segmentation information; so that the instance segmentation result of the target image can be determined according to the first confidence subsequently, the data processing speed of the second sub-network can be improved, thereby improving the instance segmentation speed of the instance segmentation model, that is, improving the real-time performance of the instance segmentation model.

[0146] In some scenarios, the output size of the post-processing process is dynamic, so the input size of the second sub-network also changes dynamically, which makes it impossible to directly convert the second sub-network to the inference optimizer, that is, the instance segmentation model cannot be directly converted to the inference optimizer.

[0147] For example, if the instance segmentation model is a Yolact++ model and the inference optimizer is TensorRT, the input size of the rescoring network depends on the number of post-processing (such as NMS) outputs, which is dynamic, so it is impossible to convert the Yolact++ model directly to TensorRT.

[0148] In some embodiments of the present application, in order to ensure that the instance segmentation model can be directly converted to the inference optimizer and realize that the input size of the second sub-network remains dynamic. Specifically, in step S21 or step S212, when the second model parameters are loaded into the second benchmark model of the inference optimizer and the second ONNX model corresponding to the second sub-network is obtained, the second ONNX model is set to dynamic input and output at the same time. In step S22, when the second ONNX model is converted to obtain the second accelerated execution model corresponding to the second sub-network, the second accelerated execution model is set to dynamic input and output at the same time.

[0149] From the above, it can be seen that when converting the second sub-network into the second ONNX model, the second ONNX model is set as dynamic input and output. When converting the second ONNX model into the second accelerated execution model, the second accelerated execution model is set as dynamic input and output. This allows the instance segmentation model to be directly converted to the inference optimizer and the input size of the second sub-network to be kept as a dynamic size.

[0150] In some embodiments, the instance segmentation model can be trained by following steps A1 to A6, where:

[0151] A1. Obtain multiple training images.

[0152] Among them, the training images have been annotated with segmentation contour information.

[0153] For example, images that need to be segmented are collected to form an original data set, and divided into a training set (i.e., multiple training images) and a test set. All images in the original data set are annotated, including the detection position boxes and segmentation contour maps of all instances (i.e., objects that need to be segmented), and the coordinates of the detection position boxes and the coordinates of the segmentation contour maps are stored in a Json format annotation file. At the same time, on the one hand, the Json format annotation file is formatted and converted to obtain an annotation file consistent with the COCO dataset format. On the other hand, the original dataset is converted to the standard COCO dataset format for subsequent data processing.

[0154] A2. Calling the first subnetwork in the preset instance segmentation network to perform feature extraction processing on the training image to obtain image feature information of the training image.

[0155] In some implementations, the preset instance segmentation network includes a first sub-network and a second sub-network.

[0156] The first subnetwork is used to output image feature information of the training image. For example, the first network takes the training image as input and performs feature extraction processing on the training image to obtain image feature information of the training image.

[0157] A3. Perform instance segmentation prediction processing according to the image feature information to obtain a first segmentation prediction result of the training image and a second confidence level of the first segmentation prediction result.

[0158] The first segmentation prediction result refers to a preliminary instance segmentation map obtained by performing instance segmentation prediction processing according to image feature information, and the second confidence level refers to the confidence level of the preliminary instance segmentation map.

[0159] A4, calling the second sub-network in the preset instance segmentation network to perform confidence prediction processing on the first segmentation prediction result to obtain a third confidence of the preliminary segmentation information;

[0160] A5. Determine a second segmentation prediction result of the training image according to the third confidence level.

[0161] Among them, the third confidence refers to the confidence of the preliminary instance segmentation map during model training. The difference between the first confidence, the second confidence, and the third confidence is that the first confidence is the confidence obtained by the second sub-network in the trained instance segmentation model, which performs confidence prediction processing on the preliminary instance segmentation information output by the first sub-network. The second confidence is the confidence obtained by predicting other network structure parts except the second sub-network. The third confidence is the confidence obtained by the second sub-network in the preset instance segmentation network, which performs confidence prediction processing on the first segmentation prediction result.

[0162] The second segmentation prediction result refers to the final instance segmentation result of the training image.

[0163] The second sub-network is used to output the final segmentation prediction results of the training image.

[0164] Specifically, first, the second subnetwork in the preset instance segmentation network is called to perform confidence prediction processing on the first segmentation prediction result to obtain the third confidence of the first segmentation prediction result. Then, the product of the second confidence and the third confidence is calculated to obtain the final confidence of the first instance segmentation result. Finally, the first instance segmentation result whose final confidence is greater than the preset threshold is retained as the second segmentation prediction result of the training image.

[0165] A6. According to the second segmentation prediction result and the segmentation contour information, the preset instance segmentation network is trained to obtain a trained instance segmentation model.

[0166] For example, according to the second segmentation prediction result and the segmentation contour information, the total training loss value of the preset instance segmentation network is determined. And back propagation is performed according to the total training loss value to update the model parameters of the preset instance segmentation network until the preset stop training condition is met, and the preset instance segmentation network is used as the trained instance segmentation model. At this time, the trained instance segmentation model can be used to perform instance segmentation according to the image.

[0167] The preset stop training condition can be set according to actual needs. For example, it can be when the total training loss value is less than the preset value, or when the total training loss value basically does not change, that is, the difference between the total training loss values ​​corresponding to multiple adjacent trainings is less than the preset value; or when the number of iterations of the preset instance segmentation network training reaches the maximum number of iterations.

[0168] From the above content, it can be seen that the instance segmentation network obtained after training through steps A1 to A5, after two confidence judgments, will determine the preliminary instance segmentation map as the final instance segmentation result when the product of the second confidence and the third confidence is greater than the preset threshold, thereby ensuring the instance segmentation accuracy of the instance segmentation model.

[0169] From the above, it can be seen that in the embodiment of the present application, the main purpose of steps S10 and S20 is to load the trained instance segmentation model into the inference optimizer. Steps S30 to S50 are mainly to execute the various functional structures of the instance segmentation model through the inference optimizer, thereby realizing the acceleration of the instance segmentation model, and then improving the real-time performance of instance segmentation of the target image.

[0170] In addition, in order to verify the effect of the instance segmentation method provided by the embodiment of the present application, an experimental test was also conducted on the solution provided by the embodiment of the present application, as follows:

[0171] The inference time of the Yolact++ model in the 1080Ti server with a single empty card is 36.4ms. By converting the Yolact++ model to TensorRT, the inference time is shortened to 22.5ms, which greatly improves the inference speed of the algorithm. It can be seen that the instance segmentation method of the embodiment of the present application realizes the full body of TensorRT, which can quickly detect and segment images, and is suitable for detection and segmentation tasks with high real-time requirements.

[0172] In order to better implement the instance segmentation method in the embodiment of the present application, based on the instance segmentation method, the embodiment of the present application also provides an instance segmentation device, such as Figure 6 FIG. 6 is a schematic diagram of a structure of an example segmentation device according to an embodiment of the present application. The example segmentation device 600 includes:

[0173] An acquisition unit 601 is used to acquire a first sub-network in a trained instance segmentation model, a second sub-network in the trained instance segmentation model, and a target image containing an object to be segmented;

[0174] A conversion unit 602 is used to convert the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network;

[0175] The segmentation processing unit 603 is used to call the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image;

[0176] The segmentation processing unit 603 is further used to perform prediction processing according to the first feature data to obtain second feature data of the target image;

[0177] The segmentation processing unit 603 is further used to call the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image.

[0178] In some embodiments of the present application, the conversion unit 602 is further configured to:

[0179] The first sub-network and the second sub-network are converted respectively to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network;

[0180] According to a preset inference optimizer, the first ONNX model is converted to obtain a first accelerated execution model corresponding to the first sub-network, and the second ONNX model is converted to obtain a second accelerated execution model corresponding to the second sub-network.

[0181] In some embodiments of the present application, the conversion unit 602 is further configured to:

[0182] Acquire a first model parameter of the first sub-network and a second model parameter of the second sub-network;

[0183] The first model parameters are loaded into the first benchmark model of the inference optimizer to obtain a first ONNX model corresponding to the first sub-network; the second model parameters are loaded into the second benchmark model of the inference optimizer to obtain a second ONNX model corresponding to the second sub-network.

[0184] In some embodiments of the present application, the second ONNX model and the second accelerated execution model are both set to dynamic input and output.

[0185] In some embodiments of the present application, the segmentation processing unit 603 is further configured to:

[0186] Initializing the first accelerated execution model;

[0187] The target image is input into the first accelerated execution model, so as to call the first accelerated execution model to perform feature extraction processing on the target image and obtain first feature data of the target image.

[0188] In some embodiments of the present application, the segmentation processing unit 603 is further configured to:

[0189] Performing instance segmentation prediction processing according to the first feature data to obtain preliminary segmentation information of the target image;

[0190] The preliminary segmentation information is used as second feature data of the target image.

[0191] In some embodiments of the present application, the segmentation processing unit 603 is further configured to:

[0192] Initializing the second accelerated execution model;

[0193] Inputting the preliminary segmentation information into the second accelerated execution model, so as to call the second accelerated execution model to perform confidence prediction processing on the preliminary segmentation information, and obtain a first confidence of the preliminary segmentation information;

[0194] An instance segmentation result of the target image is determined according to the first confidence level.

[0195] In some embodiments of the present application, the instance segmentation apparatus further includes a training unit (not shown in the figure), and the training unit is specifically used to:

[0196] Acquire a plurality of training images, wherein the training images are labeled with segmentation contour information;

[0197] Calling a first subnetwork in a preset instance segmentation network to perform feature extraction processing on the training image to obtain image feature information of the training image;

[0198] Performing instance segmentation prediction processing according to the image feature information to obtain a first segmentation prediction result of the training image and a second confidence level of the first segmentation prediction result;

[0199] Calling a second subnetwork in a preset instance segmentation network to perform confidence prediction processing on the first segmentation prediction result to obtain a third confidence of the preliminary segmentation information;

[0200] Determining a second segmentation prediction result of the training image according to the third confidence level;

[0201] The preset instance segmentation network is trained according to the second segmentation prediction result and the segmentation contour information to obtain a trained instance segmentation model.

[0202] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0203] Since the instance segmentation device can perform the present application as follows Figures 1 to 5Corresponding to the steps in the instance segmentation method in any embodiment, therefore, the present application can be implemented as follows Figures 1 to 5 The beneficial effects that can be achieved by the instance segmentation method in any embodiment are detailed in the previous description and will not be repeated here.

[0204] In addition, in order to better implement the instance segmentation method in the embodiment of the present application, on the basis of the instance segmentation method, the embodiment of the present application also provides an electronic device, referring to Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. Specifically, the electronic device provided by the embodiment of the present application includes a processor 701, and the processor 701 is used to execute a computer program stored in a memory 702 to implement the following Figures 1 to 5 Corresponding to each step of the instance segmentation method in any embodiment; or, the processor 701 is used to execute the computer program stored in the memory 702 to implement the following Figure 6 The functions of each unit in the corresponding embodiment.

[0205] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 702, and executed by the processor 701 to complete the embodiment of the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0206] The electronic device may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that the illustration is merely an example of an electronic device and does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the illustration, or a combination of certain components, or different components. For example, the electronic device may also include an input / output device, a network access device, a bus, etc., and the processor 701, the memory 702, the input / output device, and the network access device are connected via a bus.

[0207] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0208] The memory 702 can be used to store computer programs and / or modules. The processor 701 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 702 and calling the data stored in the memory 702. The memory 702 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, video data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0209] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described example segmentation device, electronic device and its corresponding units can refer to the following. Figures 1 to 5 The description of the instance segmentation method in any embodiment will not be repeated here in detail.

[0210] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0211] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the present application as follows: Figures 1 to 5 For the steps in the instance segmentation method in any embodiment, the specific operations can be referred to as follows: Figures 1 to 5 The description of the instance segmentation method in any embodiment will not be repeated here.

[0212] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0213] Due to the instructions stored in the computer-readable storage medium, the present application can be executed. Figures 1 to 5 Corresponding to the steps in the instance segmentation method in any embodiment, therefore, the present application can be implemented as follows Figures 1 to 5 The beneficial effects that can be achieved by the instance segmentation method in any embodiment are detailed in the previous description and will not be repeated here.

[0214] The above is a detailed introduction to an instance segmentation method, device, electronic device and computer-readable storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An instance segmentation method, characterized in that: The method comprises: Acquire a first subnetwork in a trained instance segmentation model, a second subnetwork in the trained instance segmentation model, and a target image containing an object to be segmented; Performing conversion processing on the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network; Calling the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image; Performing prediction processing according to the first feature data to obtain second feature data of the target image; Calling the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image; The converting the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network includes: The first sub-network and the second sub-network are converted respectively to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network; According to a preset inference optimizer, the first ONNX model is converted to obtain a first accelerated execution model corresponding to the first sub-network, and the second ONNX model is converted to obtain a second accelerated execution model corresponding to the second sub-network; The first sub-network and the second sub-network are converted and processed respectively according to a preset inference optimizer to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network, including: Acquire a first model parameter of the first sub-network and a second model parameter of the second sub-network; The first model parameters are loaded into the first benchmark model of the inference optimizer to obtain a first ONNX model corresponding to the first sub-network; the second model parameters are loaded into the second benchmark model of the inference optimizer to obtain a second ONNX model corresponding to the second sub-network.

2. The instance segmentation method according to claim 1, characterized in that The second ONNX model and the second accelerated execution model are both set to dynamic input and output.

3. The instance segmentation method according to claim 1, characterized in that The calling the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image includes: Initializing the first accelerated execution model; The target image is input into the first accelerated execution model, so as to call the first accelerated execution model to perform feature extraction processing on the target image and obtain first feature data of the target image.

4. The instance segmentation method according to claim 1, characterized in that The performing prediction processing according to the first feature data to obtain second feature data of the target image includes: Performing instance segmentation prediction processing according to the first feature data to obtain preliminary segmentation information of the target image; The preliminary segmentation information is used as second feature data of the target image.

5. The instance segmentation method according to claim 4, characterized in that: The calling the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image includes: Initializing the second accelerated execution model; Inputting the preliminary segmentation information into the second accelerated execution model, so as to call the second accelerated execution model to perform confidence prediction processing on the preliminary segmentation information, and obtain a first confidence of the preliminary segmentation information; An instance segmentation result of the target image is determined according to the first confidence level.

6. The instance segmentation method according to claim 5, characterized in that: The step of obtaining the first subnetwork of the trained instance segmentation model, the second subnetwork of the trained instance segmentation model, and the target image containing the object to be segmented further comprises: Acquire a plurality of training images, wherein the training images have been labeled with segmentation contour information; Calling a first subnetwork in a preset instance segmentation network to perform feature extraction processing on the training image to obtain image feature information of the training image; Performing instance segmentation prediction processing according to the image feature information to obtain a first segmentation prediction result of the training image and a second confidence level of the first segmentation prediction result; Calling a second subnetwork in a preset instance segmentation network to perform confidence prediction processing on the first segmentation prediction result to obtain a third confidence of the preliminary segmentation information; Determining a second segmentation prediction result of the training image according to the third confidence level; The preset instance segmentation network is trained according to the second segmentation prediction result and the segmentation contour information to obtain a trained instance segmentation model.

7. An instance segmentation device, characterized in that: The instance segmentation device comprises: An acquisition unit, configured to acquire a first subnetwork in a trained instance segmentation model, a second subnetwork in the trained instance segmentation model, and a target image containing an object to be segmented; a conversion unit, configured to convert the first sub-network and the second sub-network respectively to obtain a first accelerated execution model corresponding to the first sub-network and a second accelerated execution model corresponding to the second sub-network; A segmentation processing unit, configured to call the first accelerated execution model to perform feature extraction processing on the target image to obtain first feature data of the target image; The segmentation processing unit is further used to perform prediction processing according to the first feature data to obtain second feature data of the target image; The segmentation processing unit is further used to call the second accelerated execution model to perform prediction processing on the second feature data to obtain an instance segmentation result of the target image; The conversion unit is further used for: The first sub-network and the second sub-network are converted respectively to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network; According to a preset inference optimizer, the first ONNX model is converted to obtain a first accelerated execution model corresponding to the first sub-network, and the second ONNX model is converted to obtain a second accelerated execution model corresponding to the second sub-network; The first sub-network and the second sub-network are converted and processed respectively according to a preset inference optimizer to obtain a first ONNX model corresponding to the first sub-network and a second ONNX model corresponding to the second sub-network, including: Acquire a first model parameter of the first sub-network and a second model parameter of the second sub-network; The first model parameters are loaded into the first benchmark model of the inference optimizer to obtain a first ONNX model corresponding to the first sub-network; the second model parameters are loaded into the second benchmark model of the inference optimizer to obtain a second ONNX model corresponding to the second sub-network.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the instance segmentation method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the instance segmentation method according to any one of claims 1 to 6.

Images