Model training method, service providing method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a model training method, a service provision method, an apparatus, a device and a storage medium, the method comprising: inputting an image sample into a semantic segmentation model so that the semantic segmentation model outputs a prediction result. Then, the prediction result output by the model and the result of manual annotation of the image sample are fused with the image sample respectively. Finally, while adjusting the model parameters according to the prediction result output by the semantic segmentation model, the model parameters are also adjusted according to the image features extracted from the fused image to achieve model training. Since the image features extracted from the fused image contain the semantic information of the image sample, and at the same time, the prediction result output by the semantic segmentation model can be considered as the classification result obtained after pixel-level classification of the image sample, the above method is actually a process of optimizing the model from both the semantic level and the pixel level of the image sample, thereby ensuring the training effect of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method, a service providing method, an apparatus, a device and a storage medium. Background Art

[0002] Computer vision is a technology that enables computers to have human vision capabilities. The ultimate goal of this technology is to enable computers to understand images. Semantic segmentation is a basic task in computer vision. Through semantic segmentation, different types of objects in an image can be distinguished.

[0003] In the prior art, semantic segmentation is usually achieved using a semantic segmentation model based on a neural network, and the accuracy of the semantic segmentation result directly affects the accuracy of image semantic understanding. Therefore, how to ensure the performance of the semantic segmentation model has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a model training method, a service providing method, an apparatus, a device and a storage medium to ensure the performance of a semantic segmentation model.

[0005] In a first aspect, an embodiment of the present invention provides a model training method, comprising:

[0006] Inputting the image sample into the semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0007] fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0008] According to the image features of the fused image and the prediction result, the model parameters of the semantic segmentation model are adjusted.

[0009] In a second aspect, an embodiment of the present invention provides a model training device, comprising:

[0010] An input module, used for inputting an image sample into a semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0011] A fusion module, used to fuse the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0012] An adjustment module is used to adjust the model parameters of the semantic segmentation model according to the image features of the fused image and the prediction result.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the model training method in the first aspect. The electronic device may also include a communication interface for communicating with other devices or communication networks.

[0014] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the model training method described in the first aspect.

[0015] In a fifth aspect, an embodiment of the present invention provides a service providing method, including:

[0016] Receive a request to call a training service, and execute the following steps according to the processing resources corresponding to the training service:

[0017] In response to an input operation by a user, inputting the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0018] fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0019] Adjusting model parameters of the semantic segmentation model according to the image features of the fused image and the prediction result;

[0020] Output the adjusted model parameters.

[0021] In a sixth aspect, an embodiment of the present invention provides a service providing device, including:

[0022] A receiving module, used to receive a request to call a training service;

[0023] An execution module is used to execute the following steps according to the processing resources corresponding to the training service:

[0024] In response to an input operation by a user, inputting the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0025] fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0026] Adjusting model parameters of the semantic segmentation model according to the image features of the fused image and the prediction result;

[0027] Output the adjusted model parameters.

[0028] In a seventh aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the service providing method in the fifth aspect. The electronic device may also include a communication interface for communicating with other devices or a communication network.

[0029] In the eighth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the service providing method described in the fifth aspect.

[0030] The model training method provided by the embodiment of the present invention obtains an image sample and inputs it into a semantic segmentation model so that the semantic segmentation model outputs a prediction result for the image sample. The acquired image sample is also manually annotated to obtain an annotation result. Then, the prediction result output by the model is fused with the image sample, and the manual annotation result is fused with the image sample to obtain a fused image respectively. Finally, the model parameters are adjusted according to the prediction result output by the semantic segmentation model, and the model parameters are also adjusted according to the image features extracted from the fused image, thereby realizing model training.

[0031] In the above method, the image features extracted from the fused image contain the semantic information of the image samples. At the same time, the prediction results output by the semantic segmentation model can be considered as the classification results obtained after pixel-level classification of the image samples. Therefore, the process of adjusting the model parameters according to the image features and the prediction results in the above method can be considered as a process of optimizing the model from the semantic level and the pixel level of the image samples, so that the trained semantic segmentation model has the ability to perceive the image as a whole, thereby ensuring the training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A flow chart of a model training method provided by an embodiment of the present invention;

[0034] Figure 2 for Figure 1 A schematic diagram of a processing procedure corresponding to the model training method provided in the illustrated embodiment;

[0035] Figure 3 A schematic diagram of an image sample and its corresponding annotation results and prediction results;

[0036] Figure 4 is a schematic diagram of a first fused image and a second fused image;

[0037] Figure 5 A flow chart of a service providing method provided by an embodiment of the present invention;

[0038] Figure 6 A schematic diagram of the application of a model training method and a service providing method in an image processing scenario is provided for an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of the structure of a model training device provided by an embodiment of the present invention;

[0040] Figure 8 For Figure 7 A schematic diagram of the structure of an electronic device corresponding to the model training device provided in the illustrated embodiment;

[0041] Fig. 9 A schematic diagram of the structure of a service providing device provided by an embodiment of the present invention;

[0042] Fig.10 For Fig. 9 A schematic diagram of the structure of an electronic device corresponding to the service providing device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0045] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0046] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to identifying", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is identified" may be interpreted as "when determining" or "in response to determining" or "when identifying (stated condition or event)" or "in response to identifying (stated condition or event)", depending on the context.

[0047] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0048] The model training method provided by the embodiment of the present invention is described. Before that, it is necessary to first illustrate the significance of the use of the semantic segmentation model:

[0049] As described in the background technology, semantic segmentation of images is a basic task in computer vision. Semantic segmentation has a variety of application scenarios. For example, in video conferencing scenarios, semantic segmentation can be used to identify the portrait and background in the video, and the background can be blurred or replaced from the perspective of privacy protection. In live video scenarios, semantic segmentation can be used to identify the background and portraits in the live broadcast room, and further replace the background of the live broadcast room. For example, when the anchor introduces a scenic spot, the background of the live broadcast room can be replaced with a picture of the scenic spot to provide viewers with a better viewing experience. In image or video processing scenarios, semantic segmentation can be used to identify the portrait and background in the image or video, and the background of the photos or videos taken by the user can be replaced. For example, by changing the background, the user can "travel around the world" in place.

[0050] In the above different scenarios, in order to achieve background replacement, it is necessary to classify the image into two categories, that is, to distinguish the background area and the non-background area (i.e., the area where the portrait is located) in the image. This distinction can usually be achieved using a semantic segmentation model. For the training process of the semantic segmentation model, the model training method provided by the present invention can be used to achieve it.

[0051] In the above scenario, the semantic segmentation model can be used to classify objects in the image into two categories, such as distinguishing between portraits and non-portraits. In other scenarios, such as autonomous driving, geological detection, etc., the semantic segmentation model can also achieve multi-classification, and further implement corresponding functions based on the multi-classification results. To further expand the scope of application, the present invention does not limit the application scenarios of the semantic segmentation model. As long as there is a scenario with semantic segmentation requirements, the model training method provided by the present invention can be used to achieve training.

[0052] Based on the above description, some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0053] Figure 1 The flowchart of a model training method provided in an embodiment of the present invention is a flowchart of a model training method provided in an embodiment of the present invention. The model training method provided in an embodiment of the present invention can be executed by a training device. It can be understood that the training device can be implemented as software, or a combination of software and hardware. The training device in this embodiment and the following embodiments can be a server. Figure 1 As shown, the method comprises the following steps:

[0054] S101, inputting an image sample into a semantic segmentation model so that the semantic segmentation model outputs a prediction result.

[0055] Obtain image samples and input them into the semantic segmentation model to be trained so that the semantic segmentation model can output prediction results. The prediction results are actually the classification results obtained after pixel-level classification of the image samples. According to the classification results, the semantic segmentation model can set different types of pixels in the image samples to different pixel values.

[0056] When the semantic segmentation model performs binary classification on the pixels in the image sample to distinguish the target object and the non-target object in the image sample, the prediction result output by the semantic segmentation model can be expressed as an image with the same size as the image sample. In the prediction result, the pixel value of the pixel corresponding to the target object is set to A, and the pixel value of the pixel corresponding to the non-target object is set to B. Among them, the foreground object in the image sample can be considered as the target object, and the background object can be considered as the non-target object.

[0057] In accordance with the above-mentioned video conferencing and live video broadcasting scenarios, the image samples can be image frames in the video, the target object can be a portrait, and other objects in the image can be considered as non-target objects. In accordance with the above-mentioned image or video processing scenarios, the image samples can be image frames in the video, or static images, and the target object can be a portrait, scenery, or any other object in the image.

[0058] S102, fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image.

[0059] Furthermore, the prediction results output by the semantic segmentation model and the manual annotation results can be fused with the image samples respectively. That is, the prediction results and the image samples are fused to obtain a first fused image, and the annotation results and the image samples are fused to obtain a second fused image. After fusion, the semantic information of the target object in the image sample and the classification result corresponding to the target object in the image sample can be simultaneously included in the fused image.

[0060] Among them, for obtaining the annotation results, the user can optionally manually annotate the acquired image samples, that is, manually classify each pixel in the image sample. Similar to the prediction result, the annotation result can also be presented as an image with the same size as the image sample. In the annotation result, the pixel value of the pixel corresponding to the target object is marked as A, and the pixel corresponding to the non-target object is marked as B.

[0061] S103, adjusting the model parameters of the semantic segmentation model according to the image features and the prediction result of the fused image.

[0062] Finally, the first fused image and the second fused image may be input into a preset model, so that the preset model can extract features from the two fused images respectively. The preset model may be a model based on a convolutional neural network, such as a VGG model, a GoogLeNet model, a ResNets model, etc. For the sake of clarity in the subsequent description, the image features extracted from the first fused image may be referred to as first image features, and the image features extracted from the second fused image may be referred to as second image features.

[0063] On the one hand, the first loss value can be determined according to the difference between the prediction result and the annotation result. On the other hand, the second loss value can also be determined according to the similarity between the first image feature and the second image feature. Optionally, the similarity between the image features can be expressed as the distance between the two, such as Euclidean distance, cosine distance, etc.

[0064] Based on the above two loss values, optionally, the model parameters of the semantic segmentation model can be adjusted directly according to the sum of the two loss values ​​to ensure the training effect of the model. Optionally, different weight values ​​can be set for the two loss values, and the model parameters can be adjusted according to the loss values ​​and weight values ​​to complete the training of the semantic segmentation model. And in order to increase the overall perception ability of the model, the weight value of the second loss value is usually greater than the weight value corresponding to the first loss value.

[0065] The above training process can be combined with Figure 2 understand.

[0066] It should be noted that the size of the first loss value reflects the accuracy of the classification results of each pixel by the semantic segmentation model. Therefore, using the first loss value to adjust the model parameters actually takes into account the impact of the accuracy of the classification results of individual pixels on model training. In addition, since the fused image contains the semantic information of the target object, the extracted first image features and second image features also contain the semantic information of the target object. Therefore, using the second loss value to adjust the model parameters actually takes into account the impact of the semantics of the target object on model training.

[0067] In this embodiment, the image sample is input into the semantic segmentation model so that the semantic segmentation model outputs the prediction result. Then, the prediction result output by the model and the manual annotation result of the image sample are fused with the image sample respectively. Finally, the model parameters are adjusted according to the prediction result output by the semantic segmentation model and the image features extracted from the fused image, so as to realize model training. Since the image features extracted from the fused image contain the semantic information of the image sample, and at the same time, the prediction result output by the semantic segmentation model can be considered as the classification result obtained after pixel-level classification of the image sample, therefore, the process of adjusting the model parameters according to the image features and the prediction result in the above method can be considered as the process of optimizing the model from the semantic level and the pixel level of the image sample, so that the trained semantic segmentation model has the overall perception ability of the image, and the training effect of the model is guaranteed.

[0068] It is easy to understand that the annotation result is relatively accurate, and the difference between the annotation result and the prediction result, that is, the size of the first loss value, reflects the training effect of the semantic segmentation model. In addition, if the prediction result is the same as the annotation result, the first fused image and the second fused image are the same, and the first image features and the second image features extracted therefrom are also the same. Therefore, the similarity between the first image feature and the second feature can also reflect the training effect of the semantic segmentation model.

[0069] For the image fusion process in step 102, an optional way is to perform image operation on the prediction result and the image sample to obtain a first fused image. Similarly, the labeling result and the image sample can also be subjected to image operation to obtain a second fused image. Optionally, the image operation can include an image multiplication operation or a division operation.

[0070] Optionally, in practical applications, whether it is the annotation result or the prediction result output by the semantic segmentation model, the pixel values ​​of the pixels corresponding to the target object in the image sample can be set to A=255, and the pixel values ​​of the pixels corresponding to the non-target object can be set to B=0, so that the target object and the non-target object can be clearly distinguished. In this case, the image sample, the annotation result and the prediction result can be as follows Figure 3 However, due to the errors in the semantic segmentation model prediction, the prediction results and the annotation results are usually different, and the difference between the two can also be seen from Figure 3 It can be seen in.

[0071] When image fusion is performed in the manner of image multiplication, the pixel values ​​of the non-target object pixels in the image samples will all become 0, while the pixel values ​​of the target object pixels will not be 0, so that the target object will be highlighted in the fused image. The first fused image and the second fused image can be as follows Figure 4 As shown. Figure 4 It can be seen that the fused image can clearly show the position of the target object in the image sample, and can also reflect the recognition effect of the semantic segmentation model on the target object.

[0072] In the above-mentioned video conferencing, image processing and other scenarios, the semantic segmentation model needs to accurately identify the target object in the image, so the target object in the image sample is the focus of the training process. Figure 4 In the fused image shown, since the semantic information of the target object will be completely retained and the semantic information of the non-target object will be ignored, the image features extracted by the preset model contain rich semantic information of the target object, and the semantic information of the non-target object will not interfere with the model training. In other words, the focus is placed on the target object, thereby ensuring the effect of model training.

[0073] In addition, according to the description in step 103, the preset model for extracting image features can be a convolutional neural network of different structures, and the preset model usually includes multiple convolutional layers, and the multiple convolutional layers can output image features of multiple sizes. At this time, optionally, a second loss value can be calculated using each pair of first image features and second image features of the same size, and the model parameters can be adjusted according to the multiple second loss values. Among them, the number of second loss values ​​is the same as the number of convolutional layers in the preset model.

[0074] However, considering the amount of calculation in the model training process, optionally, image features of some sizes can be selected from image features of multiple sizes, and the second loss value can be calculated based on the image features of the selected sizes, so as to adjust the model parameters. Since the more times the convolution calculation is performed, the smaller the size of the image feature is, and the richer the semantic information contained in the image feature is, the smaller the image feature is, and therefore, the image feature of a smaller size can be selected to calculate the second loss value.

[0075] The model training methods provided in the above embodiments can be deployed on a service platform to provide model training services to users. The training device can be considered as a carrier of the service platform. As mentioned in the above embodiments, the training device can be a server. Figure 5 Flow chart of a service providing method provided by an embodiment of the present invention. The service providing method provided by an embodiment of the present invention can also be executed by a training device. Figure 5 As shown, the method may include the following steps:

[0076] S201, receiving a request for invoking a training service.

[0077] S202 , in response to an input operation of the user, input the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result.

[0078] S203: Fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image.

[0079] S204, adjusting the model parameters of the semantic segmentation model according to the image features and the prediction result of the fused image.

[0080] S205, outputting the adjusted model parameters.

[0081] Users can use different devices, such as their own terminal devices, tablet computers, etc., to generate training service requests and send the requests to the service platform (i.e., the server). The requests can include the acquired image samples.

[0082] Among them, the user may be a user with training needs. In the above-mentioned scenarios, live video broadcast, video conferencing or image or video processing is usually achieved with the help of an application (App li cat i on, referred to as APP), so the user may be the developer of the APP.

[0083] After receiving the training service request, the service platform can execute the above steps 201 to 205 to train the semantic segmentation model. Finally, the service platform can also feed back the model parameters to the user so that the user can obtain the trained semantic segmentation model. The trained semantic segmentation model can be deployed on the APP or on a server that has data interaction with the APP.

[0084] The specific implementation process of each step in this embodiment can be found in Figures 1 to 4 Related description of the illustrated embodiment. The technical effects that can be achieved by this embodiment can also be referred to the description of the above embodiments, which will not be repeated here.

[0085] For ease of understanding, the specific implementation process of the model training method and service provision method provided above is exemplified in combination with the following application scenarios. Figure 6 To understand.

[0086] In the image processing scenario, the developer of the image processing APP can collect image samples containing portraits through the Internet, and use terminal devices such as mobile phones and tablets to trigger training service requests to send training service requests containing image samples to the server.

[0087] After receiving the image sample, the server can input it into the semantic segmentation model to be trained, so that the semantic segmentation model can output the prediction result for the image sample. The prediction result is the classification result obtained after pixel-level classification of the image sample. At the same time, the developer can also manually annotate the collected image samples, that is, annotate the portrait part and non-portrait part (i.e. background part) in the image sample to obtain the annotation result.

[0088] Next, the prediction result and the image sample are fused to obtain a first fused image; the annotation result and the image sample are fused to obtain a second fused image. Figure 6 As shown in the content, the semantic information of the portrait can be completely retained in both fused images, and the semantic information of the non-portrait part can be filtered out. The convolutional neural network model pre-configured in the server can also perform feature extraction on the first fused image and the second fused image respectively, so that the extracted image features also contain the semantic information of the portrait.

[0089] On the one hand, the difference between the prediction result and the annotation result is used to adjust the model parameters of the semantic segmentation model, which takes into account the impact of the accuracy of the classification result of a single pixel on the model training. On the other hand, the similarity between the two image features is used to adjust the model parameters, which takes into account the impact of the semantic information of the target object on the model training. Finally, the adjusted model parameters can be fed back to the terminal device by the server so that the APP developer can obtain the trained semantic segmentation model.

[0090] After the semantic segmentation model is trained using the above method, the semantic segmentation model can be deployed on an image processing APP or a server that interacts with the APP. The APP user can input the image to be processed into the APP, and the semantic segmentation model can distinguish the portrait part and the background part in the image to be processed. Based on the output results of the semantic segmentation model, the background part in the image to be processed can be further replaced, so that the APP can realize the cutout of the image to be processed, that is, extract the portrait from the image to be processed, and then replace the background.

[0091] If the image to be processed input by the user is Figure 6 The image samples shown are the same, and the background is a street scene. For example, the portrait in the image to be processed can be retained, and the street scene in the image to be processed can be replaced with different backgrounds such as grasslands and art galleries, thereby creating an effect that the user is in different scenes.

[0092] The model training device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these model training devices can be configured using commercially available hardware components through the steps taught in this solution.

[0093] Figure 7 A schematic diagram of the structure of a model training device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device comprises:

[0094] The input module 11 is used to input the image sample into the semantic segmentation model so that the semantic segmentation model outputs a prediction result.

[0095] The fusion module 12 is used to fuse the prediction result and the annotation result corresponding to the image sample with the image sample to obtain a fused image.

[0096] The adjustment module 13 is used to adjust the model parameters of the semantic segmentation model according to the image features of the fused image and the prediction result.

[0097] Optionally, the fusion module 12 is specifically used for:

[0098] Performing image operation on the prediction result and the image sample to obtain a first fused image;

[0099] An image operation is performed on the annotation result and the image sample to obtain a second fused image.

[0100] Optionally, the input module 11 is further used to input the first fused image and the second fused image into a preset model, so that the preset model can perform feature extraction on the first fused image and the second fused image respectively.

[0101] Optionally, the adjustment module 13 is further used for:

[0102] Determining a first loss value according to a difference between the prediction result and the annotation result;

[0103] determining a second loss value according to a similarity between image features of the first fused image and the second fused image;

[0104] Adjust model parameters of the semantic segmentation model according to the first loss value and the second loss value.

[0105] Optionally, the adjustment module 13 is further used to adjust the model parameters according to the first loss value, the second loss value and their corresponding weight values.

[0106] The preset model includes multiple convolutional layers, and the image features include image features output by at least one convolutional layer among the multiple convolutional layers.

[0107] The semantic segmentation model is used to divide the objects in the image sample into target objects and non-target objects. The image sample includes a static image and / or an image frame in a video, and the target object includes a portrait.

[0108] Figure 7 The device shown can perform Figures 1 to 4 For the method of the embodiment shown in the figure, the part not described in detail in this embodiment can be referred to Figures 1 to 4 The implementation process and technical effects of this technical solution refer to Figures 1 to 4 The description in the illustrated embodiment will not be repeated here.

[0109] The above describes the internal functions and structure of the model training device. In a possible design, the structure of the model training device can be implemented as an electronic device, which can be a server, such as Figure 8 As shown, the electronic device may include: a processor 21 and a memory 22. The memory 22 is used to store the electronic device to perform the above Figures 1 to 4The program of the model training method provided in the illustrated embodiment, the processor 21 is configured to execute the program stored in the memory 22.

[0110] The program includes one or more computer instructions, wherein the one or more computer instructions can implement the following steps when executed by the processor 21:

[0111] Inputting the image sample into the semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0112] fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0113] According to the image features of the fused image and the prediction result, the model parameters of the semantic segmentation model are adjusted.

[0114] Optionally, the processor 21 is further configured to execute the aforementioned Figures 1 to 4 All or part of the steps in the illustrated embodiments.

[0115] The structure of the electronic device may further include a communication interface 23 for the electronic device to communicate with other devices or a communication network.

[0116] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by the above electronic device, which includes instructions for executing the above Figures 1 to 4 The procedures involved in the model training recognition method in the method embodiment shown.

[0117] The service providing apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these service providing apparatuses can be configured using commercially available hardware components through the steps taught in this solution.

[0118] Fig. 9 A schematic diagram of the structure of a service providing device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the device comprises:

[0119] The receiving module 31 is used to receive a request for invoking a training service.

[0120] The execution module 32 is used to execute the following steps according to the processing resources corresponding to the training service:

[0121] In response to an input operation by a user, inputting the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0122] fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0123] Adjusting model parameters of the semantic segmentation model according to the image features of the fused image and the prediction result;

[0124] Output the adjusted model parameters.

[0125] Fig. 9 The device shown can perform Figure 5 For the method of the embodiment shown in the figure, the part not described in detail in this embodiment can be referred to Figure 5 The implementation process and technical effects of this technical solution refer to Figure 5 The description in the illustrated embodiment will not be repeated here.

[0126] The above describes the internal functions and structure of the service provider. In a possible design, the structure of the service provider can be implemented as an electronic device, which can be a server, such as Fig.10 As shown, the electronic device may include: a processor 41 and a memory 42. The memory 42 is used to store the electronic device to perform the above Figure 5 The program of the service providing method provided in the illustrated embodiment, the processor 41 is configured to execute the program stored in the memory 42 .

[0127] The program includes one or more computer instructions, wherein the one or more computer instructions can implement the following steps when executed by the processor 41:

[0128] Receive a request to call a training service, and execute the following steps according to the processing resources corresponding to the training service:

[0129] In response to an input operation by a user, inputting the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result;

[0130] fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a fused image;

[0131] Adjusting model parameters of the semantic segmentation model according to the image features of the fused image and the prediction result;

[0132] Output the adjusted model parameters.

[0133] Optionally, the processor 41 is further configured to execute the aforementioned Figure 5 All or part of the steps in the illustrated embodiments.

[0134] The structure of the electronic device may further include a communication interface 43 for the electronic device to communicate with other devices or a communication network.

[0135] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by the above electronic device, which includes instructions for executing the above Figure 5 The procedures involved in the service providing method in the method embodiment shown.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that: include: Inputting the image sample into the semantic segmentation model so that the semantic segmentation model outputs a prediction result; Fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a first fused image and a second fused image; Determining a first loss value according to a difference between the prediction result and the annotation result; determining a second loss value according to a similarity between image features of the first fused image and the second fused image; Adjust model parameters of the semantic segmentation model according to the first loss value and the second loss value.

2. The method according to claim 1, characterized in that The step of fusing the prediction result and the annotation result corresponding to the image sample with the image sample to obtain a first fused image and a second fused image includes: Performing image operation on the prediction result and the image sample to obtain the first fused image; Perform image operation on the annotation result and the image sample to obtain the second fused image.

3. The method according to claim 2, characterized in that The method further comprises: The first fused image and the second fused image are input into a preset model so that the preset model performs feature extraction on the first fused image and the second fused image respectively.

4. The method according to claim 3, characterized in that The adjusting the model parameters of the semantic segmentation model according to the first loss value and the second loss value includes: Adjust the model parameters according to the first loss value, the second loss value and their respective corresponding weight values.

5. The method according to claim 3, characterized in that: The preset model includes multiple convolutional layers, and the image features include image features output by at least one convolutional layer among the multiple convolutional layers.

6. The method according to any one of claims 1 to 5, characterized in that The semantic segmentation model is used to divide objects in the image sample into target objects and non-target objects.

7. The method according to claim 6, characterized in that The image samples include static images and / or image frames in a video, and the target object includes a portrait.

8. A service providing method, characterized in that: include: Receive a request to call a training service, and execute the following steps according to the processing resources corresponding to the training service: In response to an input operation by a user, inputting the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result; Fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a first fused image and a second fused image; Determining a first loss value according to a difference between the prediction result and the annotation result; determining a second loss value according to a similarity between image features of the first fused image and the second fused image; Adjusting model parameters of the semantic segmentation model according to the first loss value and the second loss value; Output the adjusted model parameters.

9. A model training device, characterized in that: include: An input module, used for inputting an image sample into a semantic segmentation model so that the semantic segmentation model outputs a prediction result; A fusion module, used to fuse the prediction result and the annotation result corresponding to the image sample with the image sample respectively, so as to obtain a first fused image and a second fused image; An adjustment module, used to determine a first loss value according to a difference between the prediction result and the annotation result; A second loss value is determined according to the similarity between the image features of the first fused image and the second fused image; and a model parameter of the semantic segmentation model is adjusted according to the first loss value and the second loss value.

10. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the model training method as described in any one of claims 1 to 7.

11. A non-transitory machine-readable storage medium, characterized in that: The non-temporary machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the model training method as described in any one of claims 1 to 7.

12. A service providing device, characterized in that: include: A receiving module, used to receive a request to call a training service; An execution module is used to execute the following steps according to the processing resources corresponding to the training service: In response to an input operation by a user, inputting the image sample included in the request into a semantic segmentation model so that the semantic segmentation model outputs a prediction result; Fusing the prediction result and the annotation result corresponding to the image sample with the image sample respectively to obtain a first fused image and a second fused image; Determining a first loss value according to a difference between the prediction result and the annotation result; determining a second loss value according to a similarity between image features of the first fused image and the second fused image; Adjusting model parameters of the semantic segmentation model according to the first loss value and the second loss value; Output the adjusted model parameters.

13. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the service providing method as described in claim 8.

14. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the service providing method according to claim 8.

Images