A model training and image segmentation method, device, storage medium and equipment

Abstract

The present specification discloses a model training and image segmentation method, device, storage medium and equipment. The model training method includes: obtaining sample images and label data; determining the instance category information corresponding to each instance label in the label data; inputting the sample image and the instance category information into the image segmentation model to be trained, respectively determining the image features corresponding to the sample image and the text features corresponding to the instance category information through the encoder in the image segmentation model, determining the target features according to the sample image through the target feature extraction network in the image segmentation model; adjusting the target features according to the image features through the attention network in the image segmentation model to obtain enhanced target features; determining the image segmentation result according to the enhanced target features and the text features corresponding to the instance category information; determining the loss value and training the model according to the deviation between the image segmentation result and the label data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and equipment for model training and image segmentation. Background Art

[0002] With the rapid development of computer vision technology, image segmentation (instance segmentation) has been widely used in fields such as autonomous driving, medical diagnosis, robot navigation, and face recognition, providing an important foundation for the accurate understanding and analysis of image content in different business scenarios.

[0003] However, the current accuracy of image segmentation is low, which makes it impossible to accurately identify the instances contained in the image. It is difficult to accurately classify each pixel in the image and different instances in the same category, which seriously affects the smooth execution of subsequent business.

[0004] Therefore, how to improve the accuracy of image segmentation is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a model training and image segmentation method, device, storage medium and equipment to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a model training method, including:

[0008] Acquire a sample image and label data corresponding to the sample image, wherein the label data includes an instance label corresponding to each instance in the sample image;

[0009] Determine instance category information corresponding to each instance label in the label data;

[0010] Inputting the sample image and the instance category information into an image segmentation model to be trained, determining image features corresponding to the sample image and text features corresponding to the instance category information respectively through an encoder in the image segmentation model, and determining target features according to the sample image through a target feature extraction network in the image segmentation model;

[0011] By using the attention network in the image segmentation model, the target feature is adjusted according to the image feature to obtain an enhanced target feature;

[0012] Determining an image segmentation result according to the enhanced target feature and the text feature;

[0013] According to the deviation between the image segmentation result and the label data, the loss value of the image segmentation model is determined, and the image segmentation model is trained with minimizing the loss value as an optimization goal.

[0014] Optionally, the encoder in the image segmentation model includes an image encoder and a text encoder;

[0015] Determining the image features corresponding to the sample image and the text features corresponding to the instance category information respectively through an encoder in the image segmentation model, specifically including:

[0016] The image features are determined according to the sample image by an image encoder in the image segmentation model, and the text features are determined according to the instance category information by a text encoder in the image segmentation model.

[0017] Optionally, determining the image features corresponding to the sample image through an encoder in the image segmentation model specifically includes:

[0018] Determining initial image features corresponding to the sample image through an encoder in an image segmentation model;

[0019] Inputting the initial image features into an adapter in the image segmentation model, so as to adjust the initial image features through the adapter to obtain adjusted image features;

[0020] Based on a preset fusion ratio parameter, the initial image features and the adjusted image features are fused to obtain image features that match the image segmentation scene corresponding to the sample image.

[0021] Optionally, the target feature is adjusted according to the image feature through the attention network in the image segmentation model to obtain the enhanced target feature, specifically including:

[0022] Processing the image features through the attention network in the image segmentation model so that the feature channels of the processed image features match the feature channels of the target features;

[0023] Determining an attention weight according to the processed image features and the target features;

[0024] The target feature is adjusted according to the attention weight to obtain the enhanced target feature.

[0025] Optionally, determining an image segmentation result for the sample image according to the enhanced target feature and the text feature specifically includes:

[0026] Inputting the enhanced target features into a candidate region generation network in the image segmentation model, so as to determine regions in the sample image where instances may exist as candidate regions through the candidate region generation network;

[0027] Determining regional visual features according to the candidate regions and the enhanced target features;

[0028] The regional visual features and the text features are input into an instance segmentation network in the image segmentation model, so as to determine an image segmentation result corresponding to each instance contained in the sample image through the instance segmentation network.

[0029] Optionally, before obtaining the sample image and the label data corresponding to the sample image, the method further includes:

[0030] Get an initial image dataset;

[0031] Dividing the image data and labels corresponding to the visible object categories in the initial image data set into a training data set;

[0032] Acquiring a sample image and label data corresponding to the sample image, specifically including:

[0033] The sample image and label data corresponding to the sample image are obtained from the training data set.

[0034] Optionally, the method further comprises:

[0035] Dividing the image data and labels corresponding to the invisible target categories in the initial image data set into a test data set;

[0036] A test image is obtained from the test data set to test the trained image segmentation model through the test image.

[0037] This specification provides an image segmentation method, comprising:

[0038] Obtain an image to be recognized;

[0039] The image to be identified is input into a pre-trained image segmentation model to determine the image segmentation result for the image to be identified through the image segmentation model, wherein the image segmentation model is trained by the model training method of the above claim.

[0040] This specification provides a model training device, including:

[0041] An acquisition module, used to acquire a sample image and label data corresponding to the sample image, wherein the label data includes an instance label corresponding to each instance in the sample image;

[0042] A determination module, used to determine instance category information corresponding to each instance label in the label data;

[0043] An input module, used to input the sample image and the instance category information into the image segmentation model to be trained, determine the image features corresponding to the sample image and the text features corresponding to the instance category information respectively through the encoder in the image segmentation model, and determine the target features according to the sample image through the target feature extraction network in the image segmentation model;

[0044] An adjustment module, configured to adjust the target feature according to the image feature through an attention network in the image segmentation model to obtain an enhanced target feature;

[0045] A recognition module, used for determining an image segmentation result for the sample image according to the enhanced target feature and the text feature;

[0046] The optimization module is used to determine the loss value of the image segmentation model according to the deviation between the image segmentation result and the label data, and train the image segmentation model with the goal of minimizing the loss value.

[0047] This specification provides an image segmentation device, comprising:

[0048] An acquisition module, used for acquiring an image to be recognized;

[0049] The recognition module is used to input the image to be recognized into a pre-trained image segmentation model to determine the image segmentation result for the image to be recognized through the image segmentation model, wherein the image segmentation model is trained by the above-mentioned model training method.

[0050] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned model training and image segmentation methods are implemented.

[0051] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model training and image segmentation methods when executing the program.

[0052] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0053] In the model training method provided in the present specification, a sample image and label data are obtained; the instance category information corresponding to each instance label in the label data is determined; the sample image and the instance category information are input into the image segmentation model to be trained, and the image features corresponding to the sample image and the text features corresponding to the instance category information are respectively determined through the encoder in the image segmentation model, and the target features are determined according to the sample image through the target feature extraction network in the image segmentation model; the target features are adjusted according to the image features through the attention network in the image segmentation model to obtain enhanced target features; the image segmentation result is determined according to the enhanced target features and the text features; and the loss value is determined according to the deviation between the image segmentation result and the label data, and the model is trained.

[0054] It can be seen from the above method that this scheme can extract features from the input information of the two modalities respectively, so as to obtain text features and image features, and then enhance the features extracted by the feature extraction network based on the sample image based on the image features, and further perform instance segmentation and feature recognition based on the enhanced target features and text features. Compared with the current method of directly extracting features from image data and making predictions, this scheme can enhance and fuse features based on multi-modal features, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0056] Figure 1 A flowchart of a model training method provided in this specification;

[0057] Figure 2 A schematic diagram of a process for determining a strengthening target feature provided in this specification;

[0058] Figure 3 A schematic diagram of an image segmentation process provided in this specification;

[0059] Figure 4 A schematic diagram of the overall framework of a model training provided in this manual;

[0060] Figure 5 A flowchart of an image segmentation method provided in this specification;

[0061] Figure 6 A schematic diagram of a model training device provided in this specification;

[0062] Figure 7A schematic diagram of an image segmentation device provided in this specification;

[0063] Figure 8 A method corresponding to the Figure 1 or Figure 5 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0065] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0066] Figure 1 A flow chart of a model training method provided in this specification includes the following steps:

[0067] S101: Acquire a sample image and label data corresponding to the sample image, wherein the label data includes an instance label corresponding to each instance in the sample image.

[0068] S102: Determine instance category information corresponding to each instance label in the label data.

[0069] In real-world applications, obtaining a large amount of high-quality labeled data is often costly and inefficient, and labeled data may be extremely scarce. Therefore, how to achieve effective instance segmentation in the absence of labeled data, namely zero-shot instance segmentation, has become a hot topic in current research.

[0070] Zero-shot learning uses information from known categories to infer knowledge from unknown categories, which requires the model to have strong generalization and knowledge transfer capabilities. Researchers enhance the model's ability to understand unknown categories by introducing additional auxiliary information, such as semantic descriptions and attribute labels. This information can help the model establish connections between known and unknown categories, thereby achieving knowledge transfer.

[0071] Based on this, this specification provides a model training method, which adds the output features of the image encoder and the text encoder as additional information to the zero-sample instance segmentation model, and improves the model's recognition and segmentation capabilities on unseen object instances by fusing the multimodal information of text and images.

[0072] In this specification, the execution entity for implementing a model training and image segmentation method can be a designated device such as a server. For the sake of ease of description, the following will only take the server as the execution entity as an example to illustrate a model training method provided in this specification.

[0073] Before training the image segmentation model, the server can obtain an initial image data set, and then divide the image data and labels corresponding to the visible class targets in the initial image data set into a training data set, and divide the image data and labels corresponding to the invisible class targets in the initial image data set into a test data set.

[0074] In an embodiment provided in this specification, the MS-COCO dataset can be used as an initial image dataset, and the initial image dataset can be divided into 65 visible classes and 15 invisible classes. Then, a training dataset and a test dataset for training and testing a zero-sample image segmentation model are constructed.

[0075] The server can select all images containing visible class objects from the official training data of the initial image dataset and remove images containing any invisible objects to obtain the training dataset to ensure that the model does not recognize invisible objects during the training process. For the test dataset, the server extracts all images containing invisible objects from the official verification data of the initial image dataset. It should be noted that the images in the test dataset can also contain both visible and invisible objects.

[0076] The server can then obtain sample images and label data corresponding to the sample images from the training data set.

[0077] Among them, the above-mentioned label data can be structured data, which includes an instance label corresponding to each instance in the sample image. Specifically, for each instance contained in the sample image, the instance corresponds to an instance segmentation label, and each pixel point corresponding to the same instance in the sample image is marked with the same instance label.

[0078] The server may then further determine instance category information corresponding to each instance label in the label data, where the instance category information is used to characterize the category corresponding to the instance marked by the instance label.

[0079] In other words, instance labels are used to distinguish different instances in sample images, and different instance labels correspond to different instances, while instance category information is used to determine the specific category to which each instance label belongs. Different instance category information corresponds to different instance categories, but there may be multiple instance labels corresponding to the same instance category.

[0080] For example, when the instances in the sample image include people ,figure ,figure ,animal ,animal And when building B, the characters ,figure ,figure ,animal ,animal The five instances of building A and building B correspond to five different instance labels respectively. The pixels corresponding to these five instances in the sample image are annotated with five different instance labels. ,figure and characters The instance category information corresponding to the instance labels of these three is the same, that is, they are all "person P", "animal P" and "human P". and animals The instance category information corresponding to the instance labels of are the same, that is, they are all “animal A”, while for building B, the instance category corresponding to its instance label is “building B”.

[0081] It should be noted that the above only uses some instance category information as an example to illustrate the label data and instance category information. In actual applications, more specific instance category information may be included, such as "cat", "dog", etc. This manual does not give too many examples for this.

[0082] S103: Input the sample image and the instance category information into the image segmentation model to be trained, and determine the image features corresponding to the sample image and the text features corresponding to the instance category information through the encoder in the image segmentation model, and determine the target features according to the sample image through the target feature extraction network in the image segmentation model.

[0083] The server can input the sample image and instance category information into the image segmentation model to be trained, and determine the image features corresponding to the sample image and the text features corresponding to the instance category information through the encoder in the image segmentation model.

[0084] Specifically, the encoder in the image segmentation model includes an image encoder and a text encoder. The server can input a sample image into the image encoder, thereby determining image features according to the sample image through the image encoder in the image segmentation model.

[0085] At the same time, the server can input the instance category information into the text encoder, so as to determine the text features according to the instance category information through the text encoder in the image segmentation model.

[0086] The above image encoder and text encoder can be encoders in a pre-trained multimodal model (Contrastive Language-Image Pre-training, CLIP), and the generated CLIP image feature representation is: . Based on the target category information including the background class, a text template is constructed and passed into the CLIP text encoder module. The generated text features can be expressed as .

[0087] Furthermore, usually, the above-mentioned encoder is generally pre-trained based on wide-area image data, and the extracted features will not accurately match a specific image segmentation scenario. Therefore, considering that the image features extracted by the pre-trained image encoder may not match the image segmentation scenario corresponding to the current segmentation model, an adapter structure can be set in the image segmentation model, and the image features output by the image encoder can be adjusted through the adapter to match the image segmentation scenario corresponding to the current image segmentation model.

[0088] In practical applications, the above-mentioned image segmentation scenarios may include one or more of autonomous driving, medical diagnosis, robot navigation, and face recognition, and this specification does not make specific limitations on this.

[0089] Specifically, the server may use the image features output by the image encoder as the initial image features, and then input the initial image features into the adapter in the image segmentation model, so as to adjust the initial image features through the adapter to obtain adjusted image features.

[0090] Then, based on the preset fusion ratio parameters, the initial image features and the adjusted image features are fused to obtain image features that match the image segmentation scene corresponding to the sample image. The above fusion ratio parameters can be set according to actual conditions, and this specification does not specifically limit this.

[0091] In this specification, the structure of the adapter may include two fully connected layers, each of which is followed by a ReLU function. The fusion is performed to obtain the final image feature, which can be expressed as:

[0092]

[0093] Wherein, Adapter represents the adapter structure, α represents a parameter for controlling the fusion ratio between the adjusted image features output by the adapter and the initial image features, and preferably, the fusion ratio parameter can be taken as 0.2.

[0094] In addition, the server can input the sample image into the feature extraction network in the image segmentation model, thereby extracting features from the sample image through the feature extraction network to obtain the target feature. .

[0095] Among them, the above-mentioned feature extraction network can be a backbone network in an image segmentation model.

[0096] S104: adjusting the target feature according to the image feature through the attention network in the image segmentation model to obtain enhanced target feature.

[0097] The server can input the above-mentioned image features and target features into the attention network in the image segmentation model, and adjust the target features according to the image features through the attention network to enhance the target features, thereby obtaining enhanced target features.

[0098] Among them, the server can process the image features through the attention network in the image segmentation model so that the feature channels of the processed image features match the feature channels of the target features.

[0099] Then, the attention weight is determined according to the processed image features and the target features, and then the target features are adjusted according to the attention weight to obtain the enhanced target features. For ease of understanding, this specification provides a schematic diagram of the process of determining the enhanced target features, such as Figure 2 shown.

[0100] Figure 2 A schematic diagram of a process for determining enhanced target features provided in this specification.

[0101] Among them, the server can change The feature channel of the image is converted to (512, 1, 1), and then it is passed into a convolution layer with a convolution kernel of (2048, 1, 1) and a stride of 1 to generate a feature of size (2048, 1, 1). ,Change The feature channel of is changed to (2048, 1).

[0102] Afterwards Transpose the data, and after the transposition and target features Perform matrix multiplication between them to obtain the processed image features . The process can be expressed as:

[0103]

[0104] The processed image features Pass in the sigmoid function to generate attention weights . The process can be expressed as:

[0105]

[0106] The server can then use the attention network to analyze the target features based on the residual principle according to the attention weights. Adjust to get enhanced target features , the process can be expressed as:

[0107]

[0108] S105: Determine an image segmentation result according to the enhanced target feature and the text feature.

[0109] After determining the enhanced target features, the image segmentation model can determine the image segmentation result corresponding to the sample image according to the enhanced target features and the text features.

[0110] Specifically, the server can input the enhanced target features into the candidate region generation network in the image segmentation model to determine the regions where instances may exist in the sample image as candidate regions through the candidate region generation network. Then, the regional visual features are determined based on the candidate regions and the enhanced target features.

[0111] In this specification, the candidate region generation network can process the candidate region and enhance the target features through the ROI align method , thus generating regional visual features of size (256, 14, 14) .

[0112] The regional visual features can then be The text features corresponding to the instance category information are input into the instance segmentation network in the image segmentation model, so that the image segmentation result corresponding to each instance contained in the sample image is determined by the instance segmentation network under the guidance of the instance category information. For ease of understanding, this specification provides a schematic diagram of the image segmentation process, such as Figure 3 shown.

[0113] Figure 3 This is a schematic diagram of an image segmentation process provided in this specification.

[0114] Among them, the Four convolutional layers and one deconvolutional layer are input into the instance segmentation network to generate ROI visual features of size (512, 28, 28) .

[0115] Afterwards and text features Matrix multiplication is performed between them to obtain the image segmentation result Mask, with a size of (66, 28, 28). The process can be expressed as:

[0116]

[0117] S106: Determine the loss value of the image segmentation model according to the deviation between the image segmentation result and the label data, and train the image segmentation model with minimizing the loss value as an optimization goal.

[0118] After determining the image segmentation result of the sample image, the server can determine the loss value of the image segmentation model based on the deviation between the image segmentation result and the label data, and train the image segmentation model with minimizing the loss value as the optimization goal until the preset optimization conditions are met (such as the model converges to a preset range or reaches a preset number of training times).

[0119] In this specification, the loss function for training the model can include two parts: Region Proposal Network (RPN) loss and and segmentation loss Among them, RPN loss can include cross entropy loss and L1 smooth regression loss of foreground-background classification, etc.; segmentation loss can include cross entropy classification loss, L1 smooth regression loss, etc. Based on this, the loss value when training the image segmentation model can be expressed as:

[0120]

[0121] For ease of understanding, this manual provides a schematic diagram of the overall framework of model training, such as Figure 4 shown.

[0122] Figure 4 A schematic diagram of the overall framework of a model training provided in this manual.

[0123] The image segmentation model is equipped with encoders (text encoder and image encoder), backbone network (feature extraction network), attention module, adapter, candidate region extraction module and instance segmentation module. The text encoder extracts the text features corresponding to the instance label, the image encoder extracts the initial image features corresponding to the sample image, and the backbone network extracts the target features corresponding to the sample image. The initial image features are then adjusted through the adapter, and the adjusted image features and target features are input into the attention module to obtain enhanced target features. Then, instance segmentation is performed through the candidate region extraction module and the instance segmentation module to obtain the image segmentation result, and the model is trained based on the image segmentation result and the label data.

[0124] It should be noted that during the training of the image segmentation model, the server can lock the image encoder and the text encoder so that the network parameters of the image encoder and the text encoder will not be updated based on the loss value.

[0125] After completing the training of the image segmentation model, the server may further obtain a test image from the test data set divided in step S101, so as to test the trained image segmentation model through the test image.

[0126] After the test image is input into the trained image segmentation model, if the output result of the image segmentation model meets the preset conditions (such as the accuracy and recall rate of the model are higher than the preset values), the image segmentation model is determined to have passed the test, and the image segmentation model can be deployed for subsequent business. If the image segmentation model fails the test, the server can retrain the model.

[0127] Furthermore, this specification also provides an image segmentation method to illustrate the image segmentation model trained by the above method from the perspective of practical application, such as Figure 5 shown.

[0128] Figure 5 A flowchart of an image segmentation method provided in this specification includes the following steps:

[0129] S501: Acquire an image to be recognized;

[0130] S502: Inputting the image to be identified into a pre-trained image segmentation model to determine an image segmentation result for the image to be identified through the image segmentation model, wherein the image segmentation model is trained by the above-mentioned model training method.

[0131] It should be noted that after the image segmentation result is determined, the instance category information corresponding to each segmented instance can be further obtained, so as to achieve classification and recognition of all instances contained in the image, and then perform subsequent business according to the image segmentation result.

[0132] In practical applications, the above instance segmentation model can be applied to different classification situations. Specifically, the above image segmentation model can be used to segment and classify different instances contained in an image, that is, to segment each instance contained in an image and determine the instance category information corresponding to each segmented instance; in addition, the above image segmentation model can be used for classification between images, that is, after segmenting the instances in an image, the category to which the image belongs can be determined based on the instance category information corresponding to each instance in the image.

[0133] In practical applications, image segmentation usually involves different business scenarios. In different business scenarios, the collected images to be identified and the image segmentation results are also different. For example, in the scenario of autonomous driving, the image to be identified can be the image data collected by the driving device through the sensor, and its corresponding image segmentation results can include obstacles such as people, animals, roads, and buildings. For another example, in the scenario of medical diagnosis, the image to be identified can be a human image (such as CT images, MRI images, and camera images), and the image segmentation results of the human image can be the tissues or organs contained in the human image.

[0134] It can be seen from the above method that this scheme introduces the image encoder output features and text encoder output features of the pre-trained multimodal model CLIP into the zero-shot instance segmentation model at the same time. An attention network is constructed, and the target features output by the backbone network (feature extraction network) are enhanced through the image features adjusted by the adapter output by the CLIP image encoder. The text features corresponding to the instance category information output by the CLIP text encoder are passed to the instance segmentation module to calculate the similarity between it and the visual features of the segmentation model region to find the closest category. Through the above operations, this scheme introduces richer features, thereby enhancing the feature transfer capability of the zero-shot instance segmentation model to a certain extent and improving the segmentation accuracy of zero-shot targets.

[0135] The above are one or more implementations of the model training and image segmentation methods of this specification. Based on the same idea, this specification also provides corresponding model training and image segmentation devices, such as Figure 6 and Figure 7 shown.

[0136] Figure 6 A schematic diagram of a model training device provided in this specification includes:

[0137] An acquisition module 601 is used to acquire a sample image and label data corresponding to the sample image, wherein the label data includes an instance label corresponding to each instance in the sample image;

[0138] A determination module 602 is used to determine instance category information corresponding to each instance label in the label data;

[0139] An input module 603 is used to input the sample image and the instance category information into the image segmentation model to be trained, and to determine the image features corresponding to the sample image and the text features corresponding to the instance category information through an encoder in the image segmentation model, and to determine the target features according to the sample image through a target feature extraction network in the image segmentation model;

[0140] An adjustment module 604 is used to adjust the target feature according to the image feature through the attention network in the image segmentation model to obtain an enhanced target feature;

[0141] The recognition module 605 is used to determine the image segmentation result of the sample image according to the enhanced target feature and the text feature;

[0142] The optimization module 606 is used to determine the loss value of the image segmentation model according to the deviation between the image segmentation result and the label data, and train the image segmentation model with the goal of minimizing the loss value.

[0143] Optionally, the encoder in the image segmentation model includes an image encoder and a text encoder;

[0144] The input module 603 is specifically used to determine the image features according to the sample image through an image encoder in the image segmentation model, and to determine the text features according to the instance category information through a text encoder in the image segmentation model.

[0145] Optionally, the input module 603 is specifically used to determine the initial image features corresponding to the sample image through an encoder in the image segmentation model; input the initial image features into an adapter in the image segmentation model to adjust the initial image features through the adapter to obtain adjusted image features; and based on a preset fusion ratio parameter, perform feature fusion on the initial image features and the adjusted image features to obtain image features that match the image segmentation scene corresponding to the sample image.

[0146] Optionally, the adjustment module 604 is specifically used to process the image features through the attention network in the image segmentation model so that the feature channels of the processed image features match the feature channels of the target features; determine the attention weights based on the processed image features and the target features; and adjust the target features based on the attention weights to obtain the enhanced target features.

[0147] Optionally, the recognition module 605 is specifically used to input the enhanced target features into the candidate region generation network in the image segmentation model, so as to determine the regions in the sample image where instances may exist as candidate regions through the candidate region generation network; determine the regional visual features according to the candidate regions and the enhanced target features; and input the regional visual features and the text features into the instance segmentation network in the image segmentation model, so as to determine the image segmentation results corresponding to each instance contained in the sample image through the instance segmentation network.

[0148] Optionally, before obtaining the sample image and the label data corresponding to the sample image, the acquisition module 601 is further used to obtain an initial image data set; divide the image data and labels corresponding to the visible target category in the initial image data set into a training data set;

[0149] The acquisition module 601 is specifically used to acquire the sample image and label data corresponding to the sample image from the training data set.

[0150] Optionally, the device further comprises:

[0151] The testing module 607 is used to divide the image data and labels corresponding to the invisible target categories in the initial image data set into a test data set; obtain a test image from the test data set to test the trained image segmentation model through the test image.

[0152] Figure 7 A schematic diagram of an image segmentation device provided in this specification includes:

[0153] An acquisition module 701 is used to acquire an image to be recognized;

[0154] The recognition module 702 is used to input the image to be recognized into a pre-trained image segmentation model to determine the image segmentation result for the image to be recognized through the image segmentation model.

[0155] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A model training method and Figure 5 An image segmentation method is provided.

[0156] This manual also provides Figure 8 The one shown corresponds to Figure 1 or Figure 5 A schematic diagram of the electronic device. Figure 8 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training method or Figure 5 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0157] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0158] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0159] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0160] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0161] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0167] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0168] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0169] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0171] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0172] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A model training method, characterized in that: include: Acquire a sample image and label data corresponding to the sample image, wherein the label data includes an instance label corresponding to each instance in the sample image; Determine instance category information corresponding to each instance label in the label data; Inputting the sample image and the instance category information into an image segmentation model to be trained, determining image features corresponding to the sample image and text features corresponding to the instance category information respectively through an encoder in the image segmentation model, and determining target features according to the sample image through a target feature extraction network in the image segmentation model; The target feature is adjusted according to the image feature through the attention network in the image segmentation model to obtain an enhanced target feature, wherein the image feature is processed through the attention network in the image segmentation model so that the feature channel of the processed image feature matches the feature channel of the target feature, an attention weight is determined according to the processed image feature and the target feature, and the target feature is adjusted according to the attention weight to obtain an enhanced target feature; Determining an image segmentation result according to the enhanced target feature and the text feature; According to the deviation between the image segmentation result and the label data, the loss value of the image segmentation model is determined, and the image segmentation model is trained with minimizing the loss value as an optimization goal.

2. The method according to claim 1, characterized in that The encoder in the image segmentation model includes an image encoder and a text encoder; Determining the image features corresponding to the sample image and the text features corresponding to the instance category information respectively through an encoder in the image segmentation model, specifically including: The image features are determined according to the sample image by an image encoder in the image segmentation model, and the text features are determined according to the instance category information by a text encoder in the image segmentation model.

3. The method according to claim 1, characterized in that Determining the image features corresponding to the sample image through the encoder in the image segmentation model specifically includes: Determining initial image features corresponding to the sample image through an encoder in an image segmentation model; Inputting the initial image features into an adapter in the image segmentation model, so as to adjust the initial image features through the adapter to obtain adjusted image features; Based on a preset fusion ratio parameter, the initial image features and the adjusted image features are fused to obtain image features that match the image segmentation scene corresponding to the sample image.

4. The method according to claim 1, characterized in that Determining an image segmentation result according to the enhanced target feature and the text feature specifically includes: Inputting the enhanced target features into a candidate region generation network in the image segmentation model, so as to determine regions in the sample image where instances may exist as candidate regions through the candidate region generation network; Determining regional visual features according to the candidate regions and the enhanced target features; The regional visual features and the text features are input into an instance segmentation network in the image segmentation model, so as to determine an image segmentation result corresponding to each instance contained in the sample image through the instance segmentation network.

5. The method according to claim 1, characterized in that Before obtaining the sample image and the label data corresponding to the sample image, the method further includes: Get an initial image dataset; Dividing the image data and labels corresponding to the visible object categories in the initial image data set into a training data set; Acquiring a sample image and label data corresponding to the sample image, specifically including: The sample image and label data corresponding to the sample image are obtained from the training data set.

6. The method according to claim 5, characterized in that The method further comprises: Dividing the image data and labels corresponding to the invisible target categories in the initial image data set into a test data set; A test image is obtained from the test data set to test the trained image segmentation model through the test image.

7. An image segmentation method, characterized in that: include: Obtain an image to be recognized; The image to be identified is input into a pre-trained image segmentation model to determine the image segmentation result for the image to be identified through the image segmentation model, wherein the image segmentation model is trained by the method described in any one of claims 1 to 5 above.

8. A model training device, characterized in that: include: An acquisition module, used to acquire a sample image and label data corresponding to the sample image, wherein the label data includes an instance label corresponding to each instance in the sample image; A determination module, used to determine instance category information corresponding to each instance label in the label data; An input module, used to input the sample image and the instance category information into the image segmentation model to be trained, determine the image features corresponding to the sample image and the text features corresponding to the instance category information respectively through the encoder in the image segmentation model, and determine the target features according to the sample image through the target feature extraction network in the image segmentation model; An adjustment module, configured to adjust the target feature according to the image feature through the attention network in the image segmentation model to obtain an enhanced target feature, wherein the image feature is processed through the attention network in the image segmentation model so that the feature channel of the processed image feature matches the feature channel of the target feature, an attention weight is determined according to the processed image feature and the target feature, and the target feature is adjusted according to the attention weight to obtain an enhanced target feature; A recognition module, used for determining an image segmentation result according to the enhanced target feature and the text feature; The optimization module is used to determine the loss value of the image segmentation model according to the deviation between the image segmentation result and the label data, and train the image segmentation model with the goal of minimizing the loss value.

9. An image segmentation device, characterized in that: include: An acquisition module, used for acquiring an image to be recognized; A recognition module, used for inputting the image to be recognized into a pre-trained image segmentation model to determine the image segmentation result for the image to be recognized through the image segmentation model, wherein the image segmentation model is trained by the method described in any one of claims 1 to 6 above.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images