Model training, sample generation method, electronic device and storage medium

Abstract

The embodiments of the present application provide a model training, sample generation method, electronic device and storage medium. In the model training method provided in the embodiments of the present application, a large number of training samples can be obtained based on image synthesis processing of a small number of target image samples and background image samples. In this embodiment, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of training samples, and is conducive to improving the generalization performance of the image segmentation model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a model training, sample generation method, electronic device and storage medium. Background Art

[0002] In some image processing application scenarios, there is a need for image segmentation, which usually indicates segmenting a certain image region in an image from the entire image, so as to perform other image processing operations based on the segmented image region.

[0003] In the prior art, image segmentation models are often used to achieve image segmentation. Before training the image segmentation model, a large number of training samples are often required to be prepared in advance. However, the existing method of obtaining training samples is not flexible enough, so that the training samples with limited capacity cannot cover as many image segmentation scenarios as possible. Therefore, a new solution needs to be proposed. Summary of the invention

[0004] Multiple aspects of the present application provide a model training, sample generation method, electronic device and storage medium to provide a flexible way to obtain training samples to effectively expand the sample capacity.

[0005] An embodiment of the present application provides a model training method, including: responding to a request for training an image segmentation model, performing image synthesis processing based on a first number of target image samples and a second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number; and training the image segmentation model based on the third number of training samples.

[0006] An embodiment of the present application also provides a sample generation method, including: obtaining a first number of target image samples and a second number of background image samples; performing image synthesis processing on the first number of target image samples and the second number of background image samples respectively to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number.

[0007] An embodiment of the present application also provides a sample generation method, comprising: acquiring a first number of target image samples, and reading a second number of background image samples from a background image sample database according to the reading probability of the background image; wherein the reading probability is adapted to the distribution characteristics of the result sample; performing image synthesis on the first number of target image samples and the second number of background image samples to obtain a third number of the result samples.

[0008] An embodiment of the present application also provides a sample generation method, comprising: obtaining a first number of target audio samples, and reading a second number of background audio samples from a background audio sample database according to the reading probability of the background audio samples; wherein the reading probability is adapted to the distribution characteristics of the result samples; and performing audio synthesis on the first number of target audio samples and the second number of background audio samples to obtain a third number of result samples.

[0009] The embodiment of the present application also provides a model training method, including: determining a neural network model, and inputting a training sample into the neural network model; outputting a predicted segmentation result obtained by segmenting the training sample at the output layer of the neural network model; the predicted segmentation result includes a predicted probability of a pixel on the training sample, and a distribution characteristic of a predicted boundary of a target image and a background image on the training sample; calculating a first loss function at the loss layer of the neural network model according to the predicted probability and a negative correlation between the predicted probability and a weight calculated by a loss function, and calculating a second loss function according to the distribution characteristic of the predicted boundary; and calculating a mixed loss function of the image segmentation model according to the first loss function and the second loss function, so as to iteratively train the image segmentation model according to the mixed loss function.

[0010] An embodiment of the present application also provides a model training device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to execute the steps in each method provided in the embodiment of the present application.

[0011] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, which can implement the steps of the methods provided in the embodiments of the present application when the computer program is executed.

[0012] In the model training method provided in the embodiment of the present application, a large number of training samples can be obtained based on image synthesis processing of a small number of target image samples and background image samples. In this implementation, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of training samples, and is conducive to improving the generalization performance of the image segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1A flowchart of a model training method provided for an exemplary embodiment of the present application;

[0015] Figure 2a A flowchart of a model training method provided for another exemplary embodiment of the present application;

[0016] Figure 2b A schematic diagram of multi-threaded parallel processing provided by an exemplary embodiment of the present application;

[0017] Figure 3 A flowchart of a model training method provided for another exemplary embodiment of the present application;

[0018] Figure 4a A schematic diagram of a flow chart of a sample generation method provided by an exemplary embodiment of the present application;

[0019] Figure 4b A schematic diagram of a flow chart of a sample generation method provided by an exemplary embodiment of the present application;

[0020] Figure 4c A schematic diagram of a flow chart of a sample generation method provided by an exemplary embodiment of the present application;

[0021] Figure 5 A flowchart of a model training method provided for an exemplary embodiment of the present application;

[0022] Figure 6 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application;

[0023] Figure 7 is a schematic structural diagram of an electronic device provided by another exemplary embodiment of the present application;

[0024] Figure 8 is a structural schematic diagram of an electronic device provided by another exemplary embodiment of the present application;

[0025] Fig. 9 is a structural schematic diagram of an electronic device provided by another exemplary embodiment of the present application;

[0026] Fig.10 It is a schematic structural diagram of an electronic device provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0028] Image segmentation based on visual saliency refers to extracting salient areas in an image by simulating the visual characteristics of the human eye through intelligent algorithms. Salient areas refer to areas of interest to humans in an image. For example, in some typical application scenarios, merchants can extract the main body of a product from a product poster and use it to make promotional images for other products.

[0029] In the prior art, image segmentation is usually achieved by using an image segmentation model, such as an image semantic recognition model based on a neural network. Before training the image segmentation model, it is often necessary to prepare a large number of training samples in advance. However, the existing method of obtaining training samples is not flexible enough, so that the training samples with limited capacity cannot cover as many image segmentation scenarios as possible. In view of this defect, a solution is provided in some embodiments of the present application. The technical solutions provided in each embodiment of the present application are described in detail below in conjunction with the accompanying drawings.

[0030] Figure 1 A flow chart of a model training method provided for an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0031] Step 101: Perform image synthesis processing based on a first number of target image samples and a second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number.

[0032] Step 102: train an image segmentation model based on the third number of training samples.

[0033] In this embodiment, the image segmentation model is used to implement an image segmentation operation based on visual saliency.

[0034] The target image refers to the image of the target object collected, which includes but is not limited to people, animals, plants, commodities, buildings, roads, bridges and other types of objects commonly seen in life.

[0035] An image to be processed usually contains the target object and the environment in which the target object is located. According to human visual characteristics, the target object constitutes the visually salient area in the image to be processed. In the image segmentation operation based on visual saliency, users can extract the target image of interest from the image to be processed through the image segmentation model. For example, from a product poster, the image area where the product is located is extracted; for another example, from a surveillance image, the image area where the person is located is extracted; for another example, from an aerial image of a road, the image area where the road is located is extracted.

[0036] Based on the above, the image segmentation model can be trained with training samples containing target images so that it can learn how to extract the target image from the image to be processed. In this embodiment, the target image samples are used to simulate the target images that may be encountered in the image segmentation scene. When collecting target image samples, images of various target objects that may be encountered in the image segmentation scene are covered as much as possible to improve the sample coverage. Continuing with the above example, the target image samples may include commodity image samples, portrait samples, building image samples, animal image samples, plant image samples, and the like. It should be understood that the target image samples described in this embodiment include target objects but do not include background information.

[0037] In practice, when the image to be processed shows a target object, a certain background will be used to set off the target object to achieve a better visual effect or aesthetics. In this embodiment, a background image sample can be used to simulate a background image used to set off the target object, so that the trained image segmentation model can recognize the background image on the image to be processed.

[0038] In this embodiment, the background image samples may include background image samples actually captured, and may also include artificially produced background image samples to improve the coverage of the training samples. The background image samples actually captured are used to enable the image segmentation model to learn the actual shooting scene of the target object, such as a park shooting scene, a scenic spot shooting scene, or a studio shooting scene; the artificially produced background image samples are used to enable the image segmentation model to learn the display background of the artificially produced target object, such as a gradient color background, a solid color background, etc., which will not be repeated.

[0039] In this embodiment, the model training method can be executed by an electronic device. The target image samples and the background image samples can be stored in the local storage space of the electronic device, or stored in a remote device, which is not limited in this embodiment. Upon receiving a request to train an image segmentation model, the electronic device can obtain a first number of target image samples and a second number of background image samples from the local storage or the remote device, add the first number of target image samples and the second number of background image samples to the memory, and perform image synthesis processing. The training samples obtained by the synthesis processing can be used for model training needs.

[0040] In this embodiment, a large number of training samples can be obtained by performing image synthesis processing on a small number of target image samples and background image samples. In this implementation, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of the training image segmentation model, and is conducive to improving the generalization performance of the image segmentation model.

[0041] The above embodiments record the implementation of synthesizing training samples based on target image samples and background image samples for training image segmentation models. In some exemplary embodiments, the synthesis operation of training samples can be performed offline, and the synthesized training samples can be stored in a designated storage space. When training the image segmentation model online, the offline synthesized training samples can be read from the designated storage space. In other exemplary embodiments, when training the image segmentation model online, the synthesis operation of training samples can be performed in real time online, which will be described in detail below.

[0042] In this embodiment, optionally, when the model training method is executed by an electronic device, the electronic device may start multiple threads to perform model training and synthesis operations of training samples online. In this embodiment, for ease of description, the thread that performs the synthesis operation of training samples is described as the first thread, and the thread that performs model training is described as the second thread. Among them, the first thread may include one or more threads, and the second thread may also include one or more threads, which is not limited in this embodiment. In some scenarios, the second thread may be implemented as the main thread started by the electronic device, and the first thread may be implemented as multiple sub-threads of the main thread.

[0043] Optionally, in response to a request to train an image segmentation model, the electronic device may perform image synthesis processing on a first number of target image samples and a second number of background image samples based on the first thread to obtain the third number of training samples, and store the third number of training samples in a sample queue. The electronic device may read at least one training sample from the sample queue based on the second thread, and train the image segmentation model based on the at least one training sample. It should be understood that in this process, based on the parallel processing of the first thread and the second thread, the electronic device can implement an efficient model training process of training the image segmentation model while synthesizing the training samples.

[0044] The sample queue is used to cache the synthesized training samples. As the second thread continuously reads training samples from the sample queue, the cache space of the sample queue is continuously released. Therefore, in general, the sample queue occupies less cache space to continuously store the training samples synthesized by the first thread.

[0045] Compared with the offline storage method of training samples, the online synthesis and online use of training samples provided in the embodiment of the present application, on the one hand, greatly reduces the occupancy rate of external storage space, and on the other hand, multi-threaded parallel processing effectively improves the training efficiency of the model.

[0046] Optionally, in some exemplary embodiments, the first thread may read the second number of background images from the background image sample database according to a random reading strategy. This background image selection method can increase the randomness of the training samples, thereby facilitating the universality of the image segmentation model for a variety of different segmentation scenarios.

[0047] Optionally, in other exemplary embodiments, the first thread may read a second number of background image samples from the background image sample database according to a set reading probability. This background image selection method can be trained to obtain an image segmentation model with better performance for a specific segmentation scene, which will be further described below in conjunction with the accompanying drawings.

[0048] Figure 2a A flow chart of a model training method provided for another exemplary embodiment of the present application is shown in FIG. Figure 2a As shown, the method includes:

[0049] Step 201, in response to a request for training an image segmentation model, based on a first thread, M target image samples are read from a target image sample database, and, according to a reading probability corresponding to the background image samples, N background image samples are read from a background image sample database.

[0050] Step 202: perform image synthesis processing on each of the M target image samples and the N background image samples to obtain M*N training samples, and store the M*N training samples in a sample queue.

[0051] Step 203: Perform image enhancement processing on the M*N training samples to optimize the M*N training samples.

[0052] Step 204: Read at least one training sample from the sample queue based on the second thread, and train an image segmentation model according to the at least one training sample.

[0053] In step 201, optionally, the target image samples and the background image samples may be stored in different databases for distinction. For example, the target image samples may be stored in a target image sample database, and the background image samples may be stored in a background image database.

[0054] In response to a request to train an image segmentation model, the first thread may read M target image samples from a target image sample database, and read N background image samples from the background image sample database according to the reading probabilities corresponding to the background image samples, where M and N are positive integers.

[0055] There are a large number of different background image samples in the background image database, and the reading probabilities corresponding to the background image samples can be used to directionally read the specified background image samples from the background image database.

[0056] Optionally, in this embodiment, the distribution characteristics of the training samples may be obtained in advance, and the reading probability corresponding to the background image samples in the background image sample database may be determined based on the distribution characteristics of the training samples. The distribution characteristics of the training samples may be represented by the frequency of occurrence of each type of training samples in the entire training samples. For example, the distribution characteristics of the training samples may be represented by the frequency of occurrence of the training samples containing the first background type in the entire training samples being 50%, the frequency of occurrence of the training samples containing the second background type in the entire training samples being 30%, and the frequency of occurrence of the training samples containing the third background type in the entire training samples being 10%.

[0057] Among them, the types of backgrounds can be divided according to the color distribution characteristics of the backgrounds, for example, the backgrounds can be divided into pure color types, mixed color types, gradient color types, etc. Alternatively, the types of backgrounds can be divided according to the generation channels of the backgrounds, for example, the backgrounds can be divided into studio shooting types, outdoor shooting types, artificial production types, etc. Alternatively, the types of backgrounds can be divided according to the proximity between the backgrounds and the target, for example, the backgrounds can be divided into types whose color difference with the target is less than a set color difference threshold and types whose color difference with the target is greater than or equal to a set color difference threshold.

[0058] In this embodiment, a target image sample can be synthesized with N background image samples, and the distribution probability of different background types in the background image samples can be adjusted, that is, the distribution probability of different types of training samples in the training samples can be adjusted to meet the distribution characteristics of the training samples.

[0059] Based on this, this embodiment can set the reading probability corresponding to the background image sample after determining the distribution characteristics of the training sample, so that the synthesized training sample meets the distribution characteristics. For example, following the above example, when the distribution characteristics of the training sample are as follows: the frequency of occurrence of the training sample containing the first background type in the entire training sample is 50%, the frequency of occurrence of the training sample containing the second background type in the entire training sample is 30%, and the frequency of occurrence of the training sample containing the third background type in the entire training sample is 10%, the reading probability of the background image sample matching the first background type can be set to 50%, the reading probability of the background image sample matching the second background type can be set to 40%, and the reading probability of the background image sample matching the third background type can be set to 10%.

[0060] The following is an explanation based on two typical application scenarios:

[0061] In some application scenarios, the trained image segmentation model is mainly used to segment the image to be processed under a specified background. For example, the image segmentation model is mainly used to segment product posters under a white background, or portraits under a white background. In this case, the image segmentation model can be trained with training samples under a specified background so that it learns how to segment the image to be processed under the specified background.

[0062] In order to enable the trained image segmentation model to segment the image to be processed under the specified background with higher segmentation accuracy, the number of training samples under the specified background can be appropriately increased. In this scenario, the distribution characteristics of the training samples can be expressed as: the frequency of occurrence of the training samples containing the specified background is greater than the set frequency threshold. At this time, when the first thread reads N background images, the reading probability of the background image samples matching the specified background can be increased. For example, continuing the above example, the reading probability of the white background image samples can be appropriately increased.

[0063] In other application scenarios, in order to improve the generalization ability of the image segmentation model so that the image segmentation model can better cope with a variety of different image segmentation scenarios, it can be determined that the frequency of occurrence of training samples containing complex backgrounds in the distribution characteristics of the training samples is greater than the set frequency threshold.

[0064] Based on training samples containing complex backgrounds, the image segmentation model's ability to analyze complex images can be enhanced, so as to ultimately achieve better segmentation accuracy for images to be processed with complex backgrounds. Complex background refers to backgrounds whose complexity is greater than a set complexity threshold. In this scenario, when the first thread reads N background images, the probability of reading background image samples whose complexity is greater than the set complexity threshold can be increased.

[0065] Based on the above implementation, in some optional embodiments, when the distribution characteristics of the training samples need to be readjusted during model training, the read probability of the background image samples can be adaptively adjusted according to the distribution characteristics of the new training samples, so that the training samples synthesized by the target image and the background image meet the distribution characteristics of the new training samples. Compared with the existing implementation of generating training samples offline, this step can adjust the distribution characteristics of the training samples by flexibly adjusting the read probability of the background sample images, greatly reducing the time cost and labor cost required to update the training samples.

[0066] In step 202, M and N are positive integers.

[0067] For example, when the first thread reads 10,000 target image samples from the target image sample database and 100 background image samples from the background image sample database, each of the 10,000 target image samples can be processed with the 100 background image samples to obtain 1 million training samples. Compared with the existing offline generation of training samples, this embodiment can greatly reduce the storage space occupied by training samples.

[0068] In step 203, optionally, image enhancement processing may be performed on each of the M*N training samples to optimize the training samples.

[0069] In some embodiments, the target image sample may include an alpha channel for recording transparency information, so as to facilitate image enhancement processing after image synthesis.

[0070] Optionally, the image enhancement process may include at least one of the following color optimization processes: brightness equalization process, contrast equalization process, edge blurring process, and color transformation process. Among them, the color optimization process can improve the visual effect of the training sample and reduce the distortion effect of the image synthesis operation on the training sample.

[0071] Optionally, the image enhancement process may include at least one of the following spatial transformation processes: mirror flipping, rotation, and local scaling. Through the spatial transformation process, training samples with diversified spatial distribution can be obtained to increase the randomness of the training samples.

[0072] In step 204, after the first thread stores the synthesized training samples into the sample queue, the second thread reads at least one training sample from the sample queue and trains the image segmentation model according to the at least one training sample.

[0073] In this embodiment, image synthesis processing is performed based on M target image samples and N background image samples to obtain N*M training samples. In this implementation, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of the training image segmentation model, and is conducive to improving the generalization performance of the image segmentation model.

[0074] An exemplary implementation of the above embodiment can be as follows Figure 2b As shown, the first thread is implemented as a plurality of sub-threads: sub-thread 1, sub-thread 2...sub-thread n, where n is a positive integer, and the second thread is implemented as the main thread. Sub-thread 1, sub-thread 2...sub-thread n can read the target image from the target image sample database respectively, and read the background image sample from the background image sample database according to the reading probability corresponding to each background type. Taking sub-thread 1 as an example, sub-thread 1 can read the background image sample from the background image sample database according to the reading probability corresponding to each background type while reading the shoe image from the target image database. Then, the shoe image and each background image sample read are synthesized respectively, and image enhancement processing is performed on the synthesized image to obtain a training sample, and the training sample is stored in the sample queue. The main thread reads the training sample from the sample queue to train the image segmentation model. At the same time, sub-thread 1 can continue to read other images and other background images from the target image database to continuously generate new training samples.

[0075] The following embodiments will further illustrate how the second thread trains the image segmentation model according to the training samples in conjunction with the accompanying drawings.

[0076] Figure 3 A flow chart of a model training method provided for another exemplary embodiment of the present application is as follows: Figure 3 As shown, the method includes:

[0077] Step 301, in response to a request to train an image segmentation model, perform image synthesis processing on a first number of target image samples and a second number of background image samples based on a first thread to obtain a third number of training samples, and store the third number of training samples in a sample queue.

[0078] Step 302: Based on the second thread, read at least one training sample from the sample queue.

[0079] Step 303: input the at least one training sample into the neural network model so that the output layer of the neural network model outputs the predicted segmentation result of the at least one training sample.

[0080] Step 304: Calculate a first loss function in the loss layer of the neural network model according to the predicted segmentation result, the pixel prediction probability on the at least one training sample, and the negative correlation between the predicted probability and the weight calculated by the loss function.

[0081] Step 305: Calculate a second loss function in the loss layer of the neural network model according to the distribution characteristics of the predicted boundaries of the target image samples and the background image samples provided by the predicted segmentation results.

[0082] Step 306: Calculate the mixed loss function of the image segmentation model based on the first loss function and the second loss function in the loss layer of the neural network model.

[0083] Step 307: determine whether the mixed loss function is greater than the set loss function threshold in the loss layer of the neural network model. If not, output the neural network model as an image segmentation model; if yes, adjust the parameters of the neural network model and execute step 302.

[0084] Among them, the optional implementation methods of step 301 can refer to the records of the aforementioned embodiments and will not be repeated here.

[0085] In step 302, optionally, the second thread may continue to read training samples from the sample queue and input them into the image segmentation model being trained when the sample queue is not empty until a training cycle is completed.

[0086] In step 303, optionally, in an embodiment of the present application, a neural network model (NNs) may be trained based on training samples to obtain an image segmentation model.

[0087] Optionally, in this step, the training sample can be input into the neural network model, and the input layer of the neural network model inputs the training sample into multiple subsequent computing layers (such as convolutional layers, pooling layers, etc.). After calculations by multiple computing layers, the output layer of the neural network model can output the predicted segmentation results for the training sample. Among them, the predicted segmentation results may include the predicted probability of each pixel on the training sample. Among them, the predicted probability may include the predicted probability that the pixel belongs to the target image or the predicted probability that it belongs to the background image.

[0088] Next, in step 304 and step 305, the predicted segmentation result output by the output layer of the neural network can be passed to the loss layer of the neural network model. The loss layer of the neural network model can calculate the loss function of the neural network model based on the predicted segmentation result and the actual distribution of pixels on the training sample and the actual distribution characteristics of the boundary. Among them, there is actually no order between step 304 and step 305, and the step numbers are only used for convenience of description. In practice, the second loss function can be calculated first, and then the first loss function can be calculated, or the first loss function and the second loss function can be calculated at the same time, which is not limited in this embodiment.

[0089] The calculation of the first loss function may be performed in combination with the prediction probability of the pixel and the loss function calculation weight corresponding to the prediction probability.

[0090] Generally, the higher the prediction probability of a pixel, the easier it is to classify the pixel. For example, when the prediction probability of a pixel belonging to the target image reaches 90%, it can be considered that the pixel is easily identified as a pixel in the target image.

[0091] However, in practice, some training samples or images to be processed contain many pixels that are difficult to segment. For example, pixels at the junction of the sole and the sole shadow in a black image, pixels at the armpit corner of a garment in a garment image, etc. In the process of training an image segmentation model, if the predicted probability of pixels that are easy to segment contributes more to the first loss function, then when there are more pixels that are easy to segment, the value of the first loss function will be smaller, thereby reducing the segmentation accuracy of the trained image segmentation model for pixels that are difficult to segment.

[0092] Based on this, in this embodiment, when calculating the first loss function, a loss function calculation weight is set for the predicted probability of the pixel to control the influence of the predicted probability of pixels that are easy to segment and pixels that are difficult to segment on the first loss function. Optionally, in this embodiment, the predicted probability and its loss function calculation weight can be set to be negatively correlated. In other words, the larger the value of the predicted probability, the smaller the loss function calculation weight corresponding to the predicted probability, and the smaller its contribution to the first loss function. Based on this, the first loss function can more accurately reflect the difference between the predicted value and the actual value of pixels that are difficult to segment, thereby enabling the trained image segmentation model to focus more on the areas on the image that are difficult to segment.

[0093] Optionally, in some embodiments, the first loss function may be implemented as a Focal Loss function, as shown in the following formula 1:

[0094]

[0095] in,

[0096] In Formula 1, N represents the total number of training samples, i represents the serial number of the training sample, j represents the serial number of the pixel; Ωi represents the image area on the i-th training sample, p ij represents the predicted probability of the jth pixel on the i-th training sample, y ij represents the actual value of the jth pixel on the i-th training sample, y ij ' represents the predicted value of the jth pixel on the ith training sample; α is the empirical coefficient, γ is the empirical index, and the values ​​thereof are not limited in this embodiment. In some embodiments, α may be 0.5 or 0.75.

[0097] Among them, (1-p ij ) γ Calculate the weight for the loss function corresponding to the predicted probability. Based on the above formula 1, for pixels that are easy to classify, their predicted values ​​are closer to the true values, so their corresponding p ij Smaller, 1-p ij Larger. According to the characteristics of the exponential function, 1-p ij When the value of is between (0,1), 1-p ij The larger the (1-p ij ) γ That is, (1-p ij ) γ This reduces the loss function calculation weights corresponding to pixels that are easy to classify, which is beneficial to improving the segmentation accuracy of the trained image segmentation model for pixels that are difficult to identify.

[0098] Wherein, when calculating the second loss function, the prediction region corresponding to the target image on the training sample can be determined according to the prediction probability of each pixel on the training sample provided by the prediction segmentation result. Then, the prediction boundary of the target image is determined according to the prediction region corresponding to the target image. After the prediction boundary of the target image is obtained, the overlap degree between the prediction boundary of the target image and the real boundary of the target image can be calculated as the distribution feature of the prediction boundary of the target image. Then, the second loss function is calculated according to the prediction boundary of the target image.

[0099] Alternatively, the degree of overlap of the real boundary of the background image on the training sample can be determined based on the predicted probability of each pixel on the training sample provided by the predicted segmentation result, as the distribution feature of the predicted boundary of the background image. Then, the second loss function is calculated based on the distribution feature of the predicted boundary of the background image.

[0100] The following embodiment will take the distribution characteristics of the predicted boundary of the target image as an example to illustrate the calculation method of the second loss function. Optionally, in this embodiment, the second loss function can be implemented as an IoU Loss function, as shown in the following formula 2:

[0101]

[0102] Where N is the total number of training samples, i is the serial number of the training sample, and C i represents the true boundary of the target image on the i-th training sample, Represents the predicted boundary of the target image on the i-th training sample. Based on the above formula 2, it can be seen that the greater the overlap between the predicted boundary and the true boundary of the target image on the i-th training sample, the smaller the value of the second loss function. Based on the second loss function, the trained image segmentation model can obtain global segmentation accuracy.

[0103] In step 306, optionally, after obtaining the first loss function and the second loss function, a mixed loss function of the neural network model may be calculated at the loss layer of the neural network model according to the first loss function and the second loss function.

[0104] Optionally, in some embodiments, the first loss function and the second loss function may be weighted and summed according to a set weighting coefficient to obtain a mixed loss function of the neural network model.

[0105] For example, w1 can be used as the weight of the Focal Loss function, w2 can be used as the weight of the IoU Loss function, and the weighted summation can be obtained. The mixed loss function of the neural network model can be shown in Formula 3:

[0106] Loss=w1*FocalLoss+w2*IouLoss Formula 3

[0107] In Formula 3, the values ​​of w1 and w2 can be adjusted according to actual needs. In some embodiments, w1 and w2 can be set to 1.

[0108] Training the neural network model based on the above-mentioned mixed loss function can enable the trained image segmentation model to pay attention to the global segmentation effect of the image while paying attention to pixels that are difficult to segment, which is beneficial to improving the segmentation accuracy of the image segmentation model.

[0109] In step 307, after obtaining the mixed loss function, it can be determined whether the mixed loss function is greater than the set loss function threshold; if yes, it is considered that the segmentation result of the neural network model is close to the actual distribution of the target image and the background image on the training sample, and the neural network model can be output as an image segmentation model. If no, the parameters of the neural network model can be adjusted, and step 302 is executed to iteratively train the neural network model. When adjusting the parameters of the neural network model, the back propagation algorithm can be used, which is not described here.

[0110] In this embodiment, when training a neural network model based on training samples and a neural network model, a first loss function is calculated based on the predicted probability of pixels on the training samples output by the neural network model and the negative correlation between the predicted probability and the weights calculated by the loss function; and a second loss function is calculated based on the distribution characteristics of the predicted boundaries of the target image or background image on the training samples output by the neural network model; then, a mixed loss function of the neural network model is calculated based on the first loss function and the second loss function, and the neural network model is iteratively trained based on the mixed loss function. Based on this implementation, the image segmentation model obtained by training the neural network model can pay attention to the global segmentation effect of the image while paying attention to pixels that are difficult to segment, which is conducive to improving the segmentation accuracy of the image segmentation model.

[0111] Figure 4a A flow chart of a training sample generation method provided by an exemplary embodiment of the present application is shown in FIG. Figure 4a As shown, the method includes:

[0112] Step 401a: In response to a sample generation request, a first number of target image samples and a second number of background image samples are acquired.

[0113] Step 402a: perform image synthesis processing on the first number of target image samples and the second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number.

[0114] In this embodiment, optionally, the target image samples and the background image samples may be stored in different databases for distinction. For example, the target image samples may be stored in a target image sample database, and the background image samples may be stored in a background image database.

[0115] Optionally, in some embodiments, each target image sample in the first number of target image samples may be subjected to image synthesis processing with the second number of background image samples. In this case, the third number may be the product of the first number and the second number, which will not be described in detail.

[0116] In some exemplary embodiments, when obtaining a second number of background image samples, the distribution characteristics of the training samples may be obtained, and then the reading probability of the background image samples in the background image sample database may be determined based on the distribution characteristics of the training samples, and the second number of background image samples may be obtained from the background image sample database according to the reading probability of the background image samples.

[0117] The distribution characteristics of the training samples can be represented by the frequency of occurrence of each type of training samples in the entire training samples. For example, the distribution characteristics of the training samples can be represented by the frequency of occurrence of the training samples containing the first background type in the entire training samples being 50%, the frequency of occurrence of the training samples containing the second background type in the entire training samples being 30%, and the frequency of occurrence of the training samples containing the third background type in the entire training samples being 10%.

[0118] Among them, the types of backgrounds can be divided according to the color distribution characteristics of the backgrounds, for example, the backgrounds can be divided into pure color types, mixed color types, gradient color types, etc. Alternatively, the types of backgrounds can be divided according to the generation channels of the backgrounds, for example, the backgrounds can be divided into studio shooting types, outdoor shooting types, artificial production types, etc. Alternatively, the types of backgrounds can be divided according to the proximity between the backgrounds and the target, for example, the backgrounds can be divided into types whose color difference with the target is less than a set color difference threshold and types whose color difference with the target is greater than or equal to a set color difference threshold.

[0119] In this embodiment, the training samples are synthesized based on the target image samples and the background image samples, and the distribution probabilities of different background types in the background image samples are adjusted, that is, the distribution probabilities of different types of training samples in the training samples can be adjusted to meet the distribution characteristics of the training samples.

[0120] Based on this, this embodiment can set the reading probability corresponding to the background image sample after determining the distribution characteristics of the training sample, so that the synthesized training sample meets the distribution characteristics. For example, following the above example, when the distribution characteristics of the training sample are as follows: the frequency of occurrence of the training sample containing the first background type in the entire training sample is 50%, the frequency of occurrence of the training sample containing the second background type in the entire training sample is 30%, and the frequency of occurrence of the training sample containing the third background type in the entire training sample is 10%, the reading probability of the background image sample matching the first background type can be set to 50%, the reading probability of the background image sample matching the second background type can be set to 40%, and the reading probability of the background image sample matching the third background type can be set to 10%.

[0121] In this embodiment, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of the training image segmentation model, and is conducive to improving the generalization performance of the image segmentation model.

[0122] Figure 4b A flow chart of a model training method provided for another exemplary embodiment of the present application is shown in FIG. Figure 4b As shown, the method includes:

[0123] Step 401b: Acquire a first number of target image samples, and read a second number of background image samples from a background image sample database according to the reading probability of the background image; wherein the reading probability is adapted to the distribution characteristics of the result samples.

[0124] Step 402b: perform image synthesis on the first number of target image samples and the second number of background image samples to obtain a third number of result samples.

[0125] The result sample refers to the sample image obtained by synthesizing the target image sample and the background image sample, which can be directly used for model training.

[0126] The distribution characteristics of the result samples refer to the distribution probability of each type of result samples in the result samples. In this embodiment, the distribution characteristics of the result samples can be represented by the occurrence frequency of each type of result samples in the entire result samples.

[0127] In this embodiment, the result samples are synthesized based on the target image samples and the background image samples, and the distribution probabilities of different background types in the background image samples are adjusted, that is, the distribution probabilities of different types of result samples in the result samples can be adjusted to meet the distribution characteristics of the result samples.

[0128] In some exemplary embodiments, when obtaining a second number of background image samples, the reading probability of the background image samples in the background image sample database can be determined based on the distribution characteristics of the result samples, and the second number of background image samples can be obtained from the background image sample database according to the reading probability of the background image samples.

[0129] Based on this, when synthesizing result samples, the reading probability of background image samples can be adaptively adjusted according to the distribution characteristics of the result samples, thereby increasing the flexibility of obtaining samples. On the one hand, it is beneficial to reduce the cost of obtaining result samples, and on the other hand, it is beneficial to improve the generalization performance of the image segmentation model.

[0130] Figure 4c A flow chart of a model training method provided for another exemplary embodiment of the present application is shown in FIG. Figure 4c As shown, the method includes:

[0131] Step 401c, obtaining a first number of target audio samples, and reading a second number of background audio samples from a background audio sample database according to the reading probability of the background audio samples; wherein the reading probability is adapted to the distribution characteristics of the result samples;

[0132] Step 402c: perform audio synthesis on the first number of target audio samples and the second number of background audio samples to obtain a third number of result samples.

[0133] The target audio refers to the information in the audio that the user is interested in, or the valid information in the audio. For example, a human voice or music in an audio mixed with background sound can be considered as the target audio.

[0134] Background audio refers to other sound data in the audio except the target audio. In some cases, background audio is artificially produced sound data used to set off the target audio, such as the background music used to set off the reading atmosphere in audio novels; in other cases, background audio is the background sound generated by the environment in which the audio is recorded, such as the background sound when recording in a recording studio, the background sound when recording in a park, etc.; in some other cases, background audio includes noise data such as equipment noise, transmission interference, etc. introduced during audio recording and transmission, which is not limited in this embodiment.

[0135] The result sample refers to the sample audio obtained by synthesizing the target audio sample and the background audio sample, which can be directly used for model training. For example, the result sample can be directly used to train a speech recognition model, a semantic segmentation model, etc.

[0136] The distribution characteristics of the result samples refer to the distribution probability of each type of result samples in the result samples. In this embodiment, the distribution characteristics of the result samples can be represented by the occurrence frequency of each type of result samples in the entire result samples.

[0137] In this embodiment, the result sample is synthesized based on the target audio sample and the background audio sample, and the distribution probability of different background types in the background audio sample is adjusted, so that the distribution probability of different types of result samples in the result sample can be adjusted to meet the distribution characteristics of the result sample.

[0138] In some exemplary embodiments, when obtaining a second number of background audio samples, the reading probability of the background audio samples in the background audio sample database can be determined based on the distribution characteristics of the result samples, and the second number of background audio samples can be obtained from the background audio sample database according to the reading probability of the background audio samples.

[0139] Based on this, when synthesizing result samples, the reading probability of background audio samples can be adaptively adjusted according to the distribution characteristics of the result samples, thereby increasing the flexibility of obtaining samples. On the one hand, it is beneficial to reduce the cost of obtaining result samples, and on the other hand, it is beneficial to improve the generalization performance of the audio segmentation model.

[0140] Figure 5 A flow chart of a model training method provided for an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes:

[0141] Step 501: Determine a neural network model.

[0142] Step 502: input the training sample into the neural network model so that the output layer of the neural network model outputs the predicted segmentation result obtained by segmenting the training sample; the predicted segmentation result includes the predicted probability of the pixels on the training sample and the distribution characteristics of the predicted boundaries of the target image and the background image on the training sample.

[0143] Step 503: Calculate the first loss function in the loss layer of the neural network model according to the predicted probability and the negative correlation between the predicted probability and the weight calculated by the loss function, and calculate the second loss function according to the distribution characteristics of the predicted boundary.

[0144] Step 504: Calculate a mixed loss function of the neural network model based on the first loss function and the second loss function at the loss layer of the neural network model, so as to iteratively train the neural network model based on the mixed loss function.

[0145] In this embodiment, when training the neural network model, the first loss function is calculated based on the predicted probability of the pixels on the training samples output by the neural network model and the negative correlation between the predicted probability and the weight calculated by the loss function; and the second loss function is calculated based on the distribution characteristics of the predicted boundary of the target image or background image on the training samples output by the neural network model; then, the mixed loss function of the neural network model is calculated based on the first loss function and the second loss function, and the neural network model is iteratively trained based on the mixed loss function. Based on this implementation, the neural network model can pay attention to the global segmentation effect of the image while paying attention to pixels that are difficult to segment, which is conducive to improving the segmentation accuracy of the neural network model.

[0146] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 201 to 204 can be device A; for another example, the execution subject of steps 201 and 202 can be device A, and the execution subject of step 203 can be device B; and so on.

[0147] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 201, 202, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0148] Figure 6 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a memory 601 and a processor 602 .

[0149] The memory 601 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0150] The processor 602 is coupled to the memory 601 and is used to execute the computer program in the memory 601, so as to: perform image synthesis processing according to a first number of target image samples and a second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number; and train an image segmentation model according to the third number of training samples.

[0151] Further optionally, when the processor 602 responds to a request to train an image segmentation model and performs image synthesis processing on a first number of target image samples and a second number of background image samples to obtain a third number of training samples, the processor 602 is specifically used to: respond to the request to train the image segmentation model, perform image synthesis processing on the first number of target image samples and the second number of background image samples based on the first thread to obtain the third number of training samples, and store the third number of training samples in a sample queue; when the processor 602 trains the image segmentation model based on the third number of training samples, the processor 602 is specifically used to: read at least one training sample from the sample queue based on the second thread, and train the image segmentation model based on the at least one training sample.

[0152] Further optionally, when the processor 602 performs image synthesis processing on the first number of target image samples and the second number of background image samples based on the first thread, it is specifically used to: read the first number of target image samples from the target image sample database based on the first thread; and, according to the reading probability corresponding to the background image samples, read the second number of background image samples from the background image sample database; and perform image synthesis processing on each target image sample in the first number of target image samples with the second number of background image samples.

[0153] Further optionally, the processor 602 is further configured to: obtain distribution characteristics of training samples; and determine, according to the distribution characteristics of the training samples, a reading probability corresponding to the background image samples in the background image sample database.

[0154] Further optionally, when the processor 602 determines the reading probability corresponding to the background image samples in the background image sample database based on the distribution characteristics of the training samples, it is specifically used to: increase the reading probability of the background image samples matching the specified background based on the distribution characteristics of the training samples; and / or increase the reading probability of the background image samples having a complexity greater than a set complexity threshold.

[0155] Further optionally, after performing image synthesis processing on the first number of target image samples and the second number of background image samples to obtain a third number of training samples, the processor 602 is further used to: perform image enhancement processing on the third number of training samples to optimize the third number of training samples.

[0156] Further optionally, when the processor 602 performs image enhancement processing on the third number of training samples, it is specifically used to: perform at least one of the following color optimization processing on the third number of training samples: brightness equalization processing, contrast equalization processing, edge blurring processing and color transformation processing; and / or, perform at least one of the following spatial transformation processing on the third number of training samples: mirror flipping processing, rotation processing and local scaling processing.

[0157] Further optionally, when the processor 602 reads at least one training sample from the sample queue based on the second thread and trains the image segmentation model according to the at least one training sample, it is specifically used to: read at least one training sample from the sample queue based on the second thread; input the at least one training sample into the neural network model so that the output layer of the neural network model outputs the predicted segmentation result of the at least one training sample; calculate a first loss function at the loss layer of the neural network model according to the predicted segmentation result, the predicted probability of pixels on the at least one training sample and the negative correlation between the predicted probability and the weight calculated by the loss function; and calculate a second loss function according to the distribution characteristics of the predicted boundary of the target image or background image provided by the predicted segmentation result; and calculate a mixed loss function of the neural network model according to the first loss function and the second loss function; and iteratively train the neural network model according to the mixed loss function to obtain the image segmentation model.

[0158] Further optionally, the first loss function is: Focal Loss function; the second loss function is IoULoss function; when the processor 602 calculates the mixed loss function of the image segmentation model according to the first loss function and the second loss function, it is specifically used to: perform weighted summation of the Focal Loss function and the IoU Loss function according to a set weighting coefficient to obtain the mixed loss function of the image segmentation model.

[0159] Further, if Figure 6 As shown, the electronic device also includes: a communication component 603, a display 604, a power component 605, an audio component 606 and other components. Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 Components shown.

[0160] In this embodiment, a large number of training samples can be obtained by performing image synthesis processing on a small number of target image samples and background image samples. In this implementation, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of the training image segmentation model, and is conducive to improving the generalization performance of the image segmentation model.

[0161] Figure 7 is a schematic diagram of the structure of an electronic device provided by another exemplary embodiment of the present application. Figure 7 As shown, the electronic device includes: a memory 701 and a processor 702.

[0162] The memory 701 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0163] The processor 702 is coupled to the memory 701 and is used to execute the computer program in the memory 701, so as to: obtain a first number of target image samples and a second number of background image samples; perform image synthesis processing on the first number of target image samples and the second number of background image samples respectively, to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number.

[0164] Further optionally, when acquiring the second number of background image samples, the processor 702 is specifically used to: respond to a sample generation request and acquire distribution characteristics of training samples; determine a reading probability of background image samples in a background image sample database based on the distribution characteristics of the training samples; and acquire the second number of background image samples from the background image sample database according to the reading probability of the background image samples.

[0165] Further, if Figure 7 As shown, the electronic device also includes: a communication component 703, a display 704, a power component 705, an audio component 706 and other components. Figure 7 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 7Components shown.

[0166] In this embodiment, different target image samples can be synthesized with different background image samples, which increases the flexibility of obtaining samples, effectively expands the sample capacity of the training image segmentation model, and is conducive to improving the generalization performance of the image segmentation model.

[0167] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0168] Figure 8 is a schematic diagram of the structure of an electronic device provided by another exemplary embodiment of the present application. Figure 8 As shown, the electronic device includes: a memory 801 and a processor 802.

[0169] The memory 801 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0170] The processor 802 is coupled to the memory 801 and is used to execute the computer program in the memory 801, so as to: determine a neural network model; input a training sample into the neural network model so that the output layer of the neural network model outputs a predicted segmentation result obtained by segmenting the training sample; the predicted segmentation result includes the predicted probability of the pixels on the training sample, and the distribution characteristics of the predicted boundaries of the target image and the background image on the training sample; calculate a first loss function in the loss layer of the neural network model according to the predicted probability and the negative correlation between the predicted probability and the weight calculated by the loss function, and calculate a second loss function according to the distribution characteristics of the predicted boundary; and calculate a mixed loss function of the neural network model according to the first loss function and the second loss function, so as to iteratively train the neural network model according to the mixed loss function.

[0171] Further, if Figure 8 As shown, the electronic device also includes: a communication component 803, a display 804, a power component 805, an audio component 806 and other components. Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.

[0172] In this embodiment, when training the neural network model, the first loss function is calculated based on the predicted probability of the pixels on the training samples output by the neural network model and the negative correlation between the predicted probability and the weight calculated by the loss function; and the second loss function is calculated based on the distribution characteristics of the predicted boundary of the target image or background image on the training samples output by the neural network model; then, the mixed loss function of the neural network model is calculated based on the first loss function and the second loss function, and the neural network model is iteratively trained based on the mixed loss function. Based on this implementation, the neural network model can pay attention to the global segmentation effect of the image while paying attention to pixels that are difficult to segment, which is conducive to improving the segmentation accuracy of the neural network model when segmenting images.

[0173] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0174] Fig. 9 is a schematic diagram of the structure of an electronic device provided by another exemplary embodiment of the present application. Fig. 9 As shown, the electronic device includes: a memory 901 and a processor 902 .

[0175] The memory 901 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0176] The processor 902 is coupled to the memory 901 and is used to execute the computer program in the memory 901, so as to: obtain a first number of target image samples, and read a second number of background image samples from a background image sample database according to the reading probability of the background image; wherein the reading probability is adapted to the distribution characteristics of the result samples; and perform image synthesis on the first number of target image samples and the second number of background image samples to obtain a third number of the result samples.

[0177] Further, if Fig. 9 As shown, the electronic device also includes: a communication component 903, a display 904, a power component 905, an audio component 906 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 9 Components shown.

[0178] In this embodiment, when synthesizing result samples, the reading probability of background image samples can be adjusted according to the distribution characteristics of the result samples, thereby increasing the flexibility of obtaining samples. On the one hand, it is beneficial to reduce the cost of obtaining result samples, and on the other hand, it is beneficial to improve the generalization performance of the image segmentation model.

[0179] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0180] Fig.10 is a schematic diagram of the structure of an electronic device provided by another exemplary embodiment of the present application. Fig.10 As shown, the electronic device includes: a memory 1001 and a processor 1002 .

[0181] The memory 1001 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0182] The processor 1002 is coupled to the memory 1001 and is used to execute the computer program in the memory 1001, so as to: obtain a first number of target audio samples, and read a second number of background audio samples from a background audio sample database according to the reading probability of the background audio samples; wherein the reading probability is adapted to the distribution characteristics of the result samples; and perform audio synthesis on the first number of target audio samples and the second number of background audio samples to obtain a third number of result samples.

[0183] Further, if Fig.10 As shown, the electronic device also includes: a communication component 1003, a display 1004, a power component 1005, an audio component 1006 and other components. Fig.10 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.10 Components shown.

[0184] In this embodiment, when synthesizing result samples, the reading probability of background image samples can be adjusted according to the distribution characteristics of the result samples, thereby increasing the flexibility of obtaining samples. On the one hand, it is beneficial to reduce the cost of obtaining result samples, and on the other hand, it is beneficial to improve the generalization performance of the image segmentation model.

[0185] Above Figure 6 , Figure 7 , Figure 8 , Fig. 9 and Fig.10The memory in can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0186] Above Figure 6 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0187] Above Figure 6 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0188] Above Figure 6 , Figure 7 , Figure 8 , Fig. 9 and Fig.10 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0189] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0194] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0195] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0196] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0197] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A model training method, characterized in that: include: Performing image synthesis processing on the first number of target image samples and the second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number; the second number of background image samples is obtained by: determining the reading probability corresponding to the background image samples in the background image sample database according to the distribution characteristics of the training samples; obtaining the second number of background image samples from the background image sample database according to the reading probability of the background image samples, wherein the distribution characteristics of the training samples are represented by the occurrence frequency of each type of training sample in the entire training samples; The image segmentation model is trained according to the third number of training samples.

2. The method according to claim 1, characterized in that In response to a request for training an image segmentation model, performing image synthesis processing according to a first number of target image samples and a second number of background image samples to obtain a third number of training samples includes: In response to the request of the training image segmentation model, performing image synthesis processing on the first number of target image samples and the second number of background image samples based on the first thread to obtain the third number of training samples, and storing the third number of training samples in a sample queue; Training the image segmentation model according to the third number of training samples includes: At least one training sample is read from the sample queue based on the second thread, and the image segmentation model is trained according to the at least one training sample.

3. The method according to claim 2, characterized in that Performing image synthesis processing on the first number of target image samples and the second number of background image samples based on the first thread includes: Reading the first number of target image samples from a target image sample database based on the first thread; and reading the second number of background image samples from the background image sample database according to the reading probability corresponding to the background image samples; Perform image synthesis processing on each target image sample in the first number of target image samples and the second number of background image samples.

4. The method according to claim 3, characterized in that Determining the reading probability corresponding to the background image sample in the background image sample database according to the distribution characteristics of the training sample includes: According to the distribution characteristics of the training samples, the reading probability of background image samples matching the specified background is increased; and / or the reading probability of background image samples having a complexity greater than a set complexity threshold is increased.

5. The method according to claim 1, characterized in that: After performing image synthesis processing according to the first number of target image samples and the second number of background image samples to obtain a third number of training samples, the method further includes: Perform image enhancement processing on the third number of training samples to optimize the third number of training samples.

6. The method according to claim 5, characterized in that Performing image enhancement processing on the third number of training samples includes: Performing at least one of the following color optimization processing on the third number of training samples: brightness equalization processing, contrast equalization processing, edge blurring processing, and color transformation processing; and / or, At least one of the following spatial transformation processes is performed on the third number of training samples: mirror flipping process, rotation process, and local scaling process.

7. The method according to claim 2, characterized in that: Reading at least one training sample from the sample queue based on the second thread, and training the image segmentation model according to the at least one training sample, including: Based on the second thread, reading at least one training sample from the sample queue; Inputting the at least one training sample into a neural network model so that an output layer of the neural network model outputs a predicted segmentation result of the at least one training sample; The loss layer of the neural network model calculates a first loss function according to the predicted probability of pixels on the at least one training sample in the predicted segmentation result and the negative correlation between the predicted probability and the weight calculated by the loss function; and calculates a second loss function according to the distribution characteristics of the predicted boundary of the target image or the background image provided by the predicted segmentation result; and calculates a mixed loss function of the neural network model according to the first loss function and the second loss function; According to the mixed loss function, the neural network model is iteratively trained to obtain the image segmentation model.

8. The method according to claim 7, characterized in that The first loss function is: Focal Loss function; the second loss function is IoU Loss function; Calculating a mixed loss function of the image segmentation model according to the first loss function and the second loss function includes: According to the set weighting coefficient, the Focal Loss function and the IoU Loss function are weighted and summed to obtain a mixed loss function of the image segmentation model.

9. A sample generation method, characterized in that: include: Acquire a first number of target image samples and respond to a sample generation request to acquire distribution characteristics of training samples; Determine the reading probability of background image samples in a background image sample database according to the distribution characteristics of the training samples; obtain a second number of background image samples from the background image sample database according to the reading probability of the background image samples, wherein the distribution characteristics of the training samples are represented by the occurrence frequency of each type of training samples in the entire training samples; The first number of target image samples are respectively subjected to image synthesis processing with the second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number.

10. A sample generation method, characterized in that: include: Acquire a first number of target image samples, and read a second number of background image samples from a background image sample database according to the reading probability of the background image; wherein the reading probability is adapted to the distribution characteristics of the result samples to acquire the distribution characteristics of the result samples; determine the reading probability of the background image samples in the background image sample database according to the distribution characteristics of the result samples; the distribution characteristics of the result samples are represented by the occurrence frequency of each type of result samples in the total number of result samples; Performing image synthesis on the first number of target image samples and the second number of background image samples to obtain a third number of result samples.

11. A sample generation method, characterized in that: include: Acquire a first number of target audio samples, and read a second number of background audio samples from a background audio sample database according to the reading probability of the background audio samples; wherein the reading probability is adapted to the distribution characteristics of the result samples to obtain the distribution characteristics of the result samples; determine the reading probability of the background audio samples in the background audio sample database according to the distribution characteristics of the result samples; the distribution characteristics of the result samples are represented by the occurrence frequency of each type of result samples in the entire result samples; Audio synthesis is performed on the first number of target audio samples and the second number of background audio samples to obtain a third number of result samples.

12. A model training method, characterized in that: include: Determine the neural network model; Inputting the training sample into the neural network model so that the output layer of the neural network model outputs a predicted segmentation result obtained by segmenting the training sample; the predicted segmentation result includes a predicted probability of a pixel on the training sample and a distribution feature of a predicted boundary of a target image and a background image on the training sample; Calculating a first loss function in the loss layer of the neural network model according to the predicted probability and the negative correlation between the predicted probability and the weight calculated by the loss function, and calculating a second loss function according to the distribution characteristics of the predicted boundary; and calculating a mixed loss function of the neural network model according to the first loss function and the second loss function, so as to iteratively train the neural network model according to the mixed loss function; Wherein, the method for obtaining the training samples is: performing image synthesis processing based on a first number of target image samples and a second number of background image samples to obtain a third number of training samples; wherein the third number is greater than the sum of the first number and the second number; the method for obtaining the second number of background image samples is: determining the reading probability corresponding to the background image samples in the background image sample database according to the distribution characteristics of the training samples; according to the reading probability of the background image samples, obtaining the second number of background image samples from the background image sample database, and the distribution characteristics of the training samples are represented by the frequency of occurrence of each type of training samples in the entire training samples.

13. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1-12.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps of the method described in any one of claims 1 to 12 can be realized.

Images