Image processing model training method, target extraction method and related device

Abstract

The invention relates to the technical field of image processing, in particular to an image processing model training and target extraction method and a related device. The image processing model training method comprises the steps of obtaining a training sample set; the training sample comprises a head-up sample and an angle sample; wherein the head-up sample represents a sample collected at a head-up angle, and the angle sample represents a sample collected at a non-head-up angle; respectively inputting the training samples into a to-be-trained image processing model, performing mask prediction training on the training samples, and outputting masks of the training samples to divide target objects and environment backgrounds in the training samples; and updating the image processing model by using the training loss. The method can adapt to images at different acquisition angles, so that the target extraction precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing model training method, a target extraction method, a computer program product, a computer device, and a computer-readable storage medium. Background Art

[0002] When extracting and assigning masks to target objects such as humans, small animals, and objects, the accurately labeled data in the public dataset is usually used as training samples to train the image processing model, which has a strong dependence on the public dataset. In scenarios such as video surveillance and human-computer interaction, the position of the target object has relatively significant feature changes compared to the front view, that is, the target object has a distribution offset under different acquisition angles, which can easily lead to unsatisfactory actual mask extraction results. Summary of the invention

[0003] Based on this, it is necessary to provide an image processing model training method, a target extraction method, a computer program product, a computer device and a computer-readable storage medium to address the above-mentioned technical problems, which can adapt to images acquired at different angles, thereby improving the accuracy of target extraction.

[0004] On the one hand, an image processing model training method is provided, and the image processing model training method includes: obtaining a training sample set; the training samples include level-view samples and angle samples; wherein the level-view samples represent samples collected at level-view angles, and the angle samples represent samples collected at non-level-view angles; respectively inputting the training samples into an image processing model to be trained for mask prediction training, and outputting the masks of the training samples to divide the target object and the environmental background in the training samples; and updating the image processing model using the training loss.

[0005] In one embodiment of the present application, the head-on samples and some angle samples carry mask annotations; inputting the training samples into the image processing model to be trained includes: in response to the training samples input for the forward adjacent first number training all carry mask annotations, inputting the angle samples without mask annotations into the image processing model; wherein the forward adjacent first number represents the first number of times occurring before the current moment and adjacent to the current moment; in response to the training samples input for the forward adjacent second number training all do not carry mask annotations, inputting the head-on samples or angle samples carrying mask annotations into the image processing model; wherein the forward adjacent second number represents the second number of times occurring before the current moment and adjacent to the current moment.

[0006] In one embodiment of the present application, before inputting the angle sample without mask annotation into the image processing model, it includes: evaluating the target object information in the angle sample without mask annotation; analyzing the confidence score of the target object information; in response to the confidence score being higher than the confidence threshold, allowing the angle sample without mask annotation to be input into the image processing model, and using the target object information as its virtual label.

[0007] In one embodiment of the present application, the annotation includes mask annotation and position box annotation of the target object; obtaining the training sample set includes: obtaining the head-on samples with mask annotations; obtaining angle samples without annotations, annotating the masks of the target objects in some angle samples, annotating the position boxes of the target objects in other angle samples, and making the angle samples with position box annotations less than the angle samples without mask annotations.

[0008] In one embodiment of the present application, performing mask prediction training includes: inputting the training sample into a feature extractor that matches its true category to output image features; wherein feature extraction weights are shared between feature extractors; inputting the image features into a category discriminator to output a discriminated category, and evaluating the discrimination loss based on the discriminated category and the true category; inputting the image features into a mask mapper that matches the true category to output a predicted mask that divides the target object and the environmental background, and evaluating the mask loss based on the predicted mask and the true value; and fusing the mask loss and the discrimination loss to obtain the training loss.

[0009] In one embodiment of the present application, the image processing model includes a first model and a second model, and the model parameter amount of the second model is less than the model parameter amount of the first model; the image processing model training method also includes: training the first model based on the training sample set; predicting the mask of the target object in the angle image by the trained first model to obtain a predicted mask; training the second model based on the angle image and the predicted mask.

[0010] On the other hand, a target extraction method is provided, which includes: acquiring an image to be extracted; extracting and labeling the target object in the image to be extracted using an image processing model; the image processing model is trained using an image processing model training method such as in any of the above embodiments.

[0011] On the other hand, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the image processing model training method in any of the above-mentioned embodiments; or, implements the steps of the computer program product in any of the above-mentioned embodiments.

[0012] On the other hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the image processing model training method in any one of the above-mentioned embodiments are implemented; or, the steps of the computer program product in any one of the above-mentioned embodiments are implemented.

[0013] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing model training method in any of the above-mentioned embodiments are implemented; or, the steps of the computer program product in any of the above-mentioned embodiments are implemented.

[0014] The above-mentioned image processing model training method, target extraction method, computer program product, computer device and computer-readable storage medium can form a training sample set using the head-up samples of the head-up acquisition angle and the angle samples of the non-head-up acquisition angle when training the image processing model, so as to improve the image processing model's adaptability to images acquired at the head-up acquisition angle and the non-head-up acquisition angle. In this way, when extracting the target object in the image to be extracted, the adaptability to multiple acquisition angles can be improved, the extraction accuracy of the target object can be improved, and the reliability of the target extraction result can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of an embodiment of the target extraction method of the present application;

[0016] Figure 2 It is a flowchart of an embodiment of the image processing model training method of the present application;

[0017] Figure 3 It is a flowchart of another embodiment of the image processing model training method of the present application;

[0018] Figure 4 It is a structural schematic diagram of an embodiment of an image processing model of the present application;

[0019] Figure 5 It is a flowchart of another embodiment of the image processing model training method of the present application;

[0020] Figure 6 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] In order to solve the technical problem of unsatisfactory target extraction effect in related technologies, the present application discloses an image processing model training method, a target extraction method, a computer program product, a computer device and a computer-readable storage medium. The image processing model training method includes: obtaining a training sample set; the training samples include level-view samples and angle samples; wherein the level-view samples represent samples collected at level-view angles, and the angle samples represent samples collected at non-level-view angles; respectively inputting the training samples into the image processing model to be trained for mask prediction training, outputting the masks of the training samples to divide the target objects and the environmental background in the training samples; and updating the image processing model using the training loss. The specific model structure and working principle of the present application are described in detail below.

[0023] For example, in a scenario such as an airport, surveillance devices are usually installed at the airport entrance security checkpoint, airport lobby, terminal hall, etc. to collect videos and images, so as to perform image processing such as target recognition, tracking, and detection on the collected video frames / images. When training the image processing model that image processing relies on, public data sets are usually used as training sample sets, which usually include images collected from a horizontal angle and related feature information.

[0024] It is easy to understand that the monitoring devices arranged at the aforementioned positions are generally used for shooting at a bird's-eye view angle. When shooting the same category or the same target object, there are obvious feature differences between the image captured at a bird's-eye view angle and the image captured at a head-on view angle. Taking the human body as an example, in the images formed by photographing the human body at security checkpoints, the human body's proportions, appearance, contours and other features are obviously different from the features of the images captured at a head-on view angle. When the image processing model trained with images captured at a head-on view angle is used to process images captured at airport security checkpoints, halls and other environments, the image processing effect is likely to be poor due to the large difference between the actual human features collected and the training samples, such as the human body not being recognized in the image, the human body extracted in the image being very different from the actual human body, and the human body being misidentified.

[0025] To this end, in the present application, when training the image processing model, the composition of the training sample set is adjusted, and the training sample set is explicitly constrained to include head-on samples collected at head-on angles and angle samples collected at non-head-on angles, so as to improve the processing performance of the image processing model for images collected at non-head-on angles through angle samples such as those collected at overhead angles and / or upward angles, thereby improving the reliability of the output results of the image processing model.

[0026] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the target extraction method of the present application.

[0027] S101: Acquire an image to be extracted.

[0028] In this embodiment, the image to be extracted represents the image of the target object and the environmental background to be divided.

[0029] The target object may be a human body, an animal of a specified or unspecified category, a vehicle, etc., which is not limited here. Taking the target object as a human as an example, the division of the target object and the environmental background may be to extract the unobstructed part of the human body, and the environmental background may be other parts of the image to be extracted except the unobstructed part of the human body, and a mask is output to distinguish the pixels of the unobstructed part of the human body from other pixels in the image.

[0030] S102: using an image processing model to annotate a target object in the image to be extracted; the image processing model is trained using an image processing model training method.

[0031] In this embodiment, the image processing model training method can be described in any of the following embodiments.

[0032] Among them, the image processing model training method may include obtaining a training sample set; the training samples include horizontal samples and angle samples; inputting the training samples into the image processing model to be trained for mask prediction training, outputting the mask of the training samples to divide the target object and the environmental background in the training samples; and updating the image processing model using the training loss.

[0033] As such, in this embodiment, the image to be extracted can be input into the trained image processing model, the image processing model can identify the target object in the image to be extracted, and output a mask or an image carrying a mask to extract the target object and separate the target object from the environmental background.

[0034] The specific working principle of the image processing model training process of this application is explained below.

[0035] See also Figure 2 , Figure 2 It is a flowchart of an embodiment of the image processing model training method of the present application. It should be noted that the image processing model in this embodiment can be applied to the target extraction method described above, and can also be applied to multi-target tracking, pedestrian re-identification, harmful animal detection in environments such as kitchens, and object status monitoring such as monitoring the opening and closing status of trash can lids, etc., without limitation here.

[0036] S201: Acquire a training sample set; the training samples include level-view samples and angle samples; wherein the level-view samples refer to samples collected at level-view angles, and the angle samples refer to samples collected at non-level-view angles.

[0037] In this embodiment, the training sample set is a collection of training samples, that is, it includes multiple training samples. The training samples are images used to train the image processing model.

[0038] The training samples may include horizontal samples and angle samples, wherein the horizontal samples are samples collected at horizontal angles, and the angle samples are samples collected at non-horizontal angles, for example, the camera, video recorder and other collection devices are at a downward angle and / or upward angle relative to the environment.

[0039] In layman's terms, the angle of acquisition of the horizontal view sample can be considered perpendicular to the direction of gravity. Images collected by mobile terminals such as face, eyes and other biometric recognition, time clocks, thumb cameras worn on the human body, etc. can be used as front-view samples. Images collected by monitoring equipment or video equipment at high or low places can be used as angle samples.

[0040] Optionally, the horizontal view samples and the angle samples may come from a public dataset and / or a self-built dataset, which is not limited here.

[0041] S202: respectively inputting the training samples into the image processing model to be trained for mask prediction training, and outputting the masks of the training samples to separate the target object and the environmental background in the training samples.

[0042] In this embodiment, when training the image processing model, the front view samples and angle samples in the training sample set are used as training samples and respectively input into the image processing model to be trained, and the image processing model is trained for mask prediction to improve the target object recognition ability of the image processing model, so as to output a mask suitable for the training sample and divide the target object and the environmental background in the training sample.

[0043] The mask of the target object pixel can be set to 0 or non-0 (such as 255, etc.), and the pixel values ​​of the environment background and the target object are different. For example, the target object pixel mask can be set to 0, and the environment background pixel value remains the real pixel value or the mask is set to non-0; or, the target object pixel mask can be set to non-0, and the environment background pixel value remains the real pixel value or the mask is set to 0, which is not limited here.

[0044] S203: Update the image processing model using the training loss.

[0045] In this embodiment, in the process of training the image processing model using the training samples, the loss of the training process can be calculated by the loss function, so as to update the image processing model using the training loss to improve the mask prediction performance of the image processing model. The number of training times or the loss threshold can be preset as a condition for completing the training, which will not be described in detail here.

[0046] It can be seen that in this embodiment, when training the image processing model, it is possible to take into account both the horizontal acquisition angle and the angle acquisition angle, and use the horizontal samples and the angle samples as training samples to train the image processing model to improve the image processing model's adaptability to images acquired at the horizontal acquisition angle and the non-horizontal acquisition angle, thereby optimizing the image processing model's adaptability to images at different acquisition angles and improving the accuracy of extracting the target object. In this way, when extracting the target object in the image to be extracted, it is possible to improve the adaptability to multiple acquisition angles, improve the extraction accuracy of the target object, and thus improve the reliability of the target extraction result.

[0047] Generally speaking, open source mask datasets of target objects such as humans, vehicles, and animals are usually images captured from a head-on acquisition angle. In situations such as video surveillance and human-computer interaction, the image processing models trained with such open source datasets are prone to unsatisfactory results when extracting human masks from overhead images, because the images captured by the camera device are usually obtained from a certain overhead angle. Similarly, unsatisfactory results are also likely to occur when identifying images taken from an upward angle. To this end, this application plans the categories and annotations of the training samples in the training sample set during the training process, the image processing model architecture, etc., to improve the mask extraction effect for images captured from non-head-on acquisition angles.

[0048] Specifically, see Figure 3 , Figure 3 It is a flowchart of another embodiment of the image processing model training method of the present application.

[0049] In one embodiment, the head-on samples and some angle samples in the training sample set carry mask annotations, and the positions and masks of the two viewing angles have a relatively corresponding distribution offset, so as to estimate the mask of the target object in the angle-captured image using the head-on samples and angle samples carrying mask annotations.

[0050] Optionally, the number of horizontal angles with mask annotations is greater than that of angle samples with mask annotations, so that the image processing model can be trained using a small amount of mask data of the target object at a top-down angle, thereby reducing the tedious workload of pre-annotating a large number of angle samples.

[0051] The training samples with mask annotations can be self-acquired open source target object mask data, and / or the training samples can be masked and annotated manually using a reliable model. Figure 3 As shown in the example, the target object can be a human, so the mask extraction can be performed on the human body that is not blocked in the image. In the current open source datasets, the human body mask data is usually collected at the head-on angle, so it can be used as a head-on sample with mask annotation.

[0052] Furthermore, the angle samples without mask annotations may be without any annotations, or may be with annotations of other categories such as position box annotations, contour annotations, etc., which are not limited here.

[0053] Taking the annotation including mask annotation and position box annotation of target object as an example, the angle sample without mask annotation can carry position box annotation, that is, the target object can be annotated in the angle sample by means of position box, etc., without specific mask annotation of the target object.

[0054] That is to say, in this embodiment, a head-on sample with a mask annotation can be obtained. Angle samples are obtained, and the masks of target objects in some angle samples are annotated, and the position boxes of target objects in other angle samples are annotated. In this way, in this embodiment, the unannotated sample data can be fully utilized, which is conducive to reducing the dependence on data annotation, thereby improving the utilization rate of sample data and reducing the cost of data annotation.

[0055] Furthermore, the angle samples with position box annotations can be more than the angle samples with mask annotations. In other words, the level-view samples with mask annotations and the angle samples with position box annotations are more than the angle samples with mask annotations. Further, some angle samples can be reserved without annotation or formed into unannotated derivative samples for testing image processing models, etc., which is not limited here.

[0056] It can be seen that the training sample set in this embodiment can be composed of at least three parts, namely, the horizontal samples with mask annotations, the angle samples with position box annotations, and the angle samples with mask annotations. Among them, the horizontal samples and angle samples with mask annotations can be used as prior knowledge containing relevant target objects to participate in the training of the image processing model.

[0057] Please continue reading Figure 3 In one embodiment, the image processing model may include a feature extractor, which, as the name implies, is used to extract image features of an input image (ie, a training sample, an image to be extracted, etc.). The number of feature extractors may be one or more.

[0058] In this embodiment, multiple feature extractors may be included, that is, the number of feature extractors is at least two, so that different feature extractors are used to extract image features of the horizontal sample and the angle sample. Optionally, model parameters / model weights between multiple feature extractors may be shared.

[0059] Specifically, the training sample can be input into a feature extractor that matches its true category to output the image features of the training sample. The true category is the sample category to which the training sample actually belongs, and the sample category can at least include a front-view sample category and an angle sample category. The angle sample category can also be further constrained to be a top-view angle sample category, an upward-view angle sample category, etc., which is not limited here.

[0060] That is, when the training sample is a normal view sample, that is, the training sample belongs to the normal view sample category, the training sample can be input into the feature extractor for extracting the image features of the formal sample. When the training sample is an angle sample, that is, the training sample belongs to the angle sample category, the training sample can be input into the feature extractor for extracting the image features of the angle sample.

[0061] Optionally, as described above, the straight-ahead samples and partial-angle samples carry mask annotations. Therefore, during the training of the image processing model, training samples carrying mask annotations and training samples without mask annotations (such as angle samples carrying position box annotations, etc.) can be alternately input into the image processing model to improve the credibility of the image processing model at different stages.

[0062] In layman's terms, after continuously inputting a first number of training samples with mask annotations for training, continuously inputting a second number of training samples without mask annotations to train the image processing model. The first number and the second number are at least one, and the first number and the second number can be the same or different, which is not limited here.

[0063] That is to say, when inputting training samples into the image processing model for training, it is possible to judge whether the training samples input several times in the previous direction carry mask annotations, so as to adaptively select the training samples currently input into the image processing model.

[0064] In other words, in response to the training samples inputted by the forward adjacent first number training all carry mask annotations, the angle samples without mask annotations can be inputted into the image processing model. The forward adjacent first number represents the first number of times that occurred before the current moment and are adjacent to the current moment. In response to the training samples inputted by the forward adjacent second number training all carry no mask annotations, the level-view samples or angle samples with mask annotations can be inputted into the image processing model. The forward adjacent second number represents the second number of times that occurred before the current moment and are adjacent to the current moment.

[0065] Optionally, the first quantity matches the first first quantity and the second quantity matches the second second quantity.

[0066] Please continue reading Figure 3In one embodiment, the image processing model may further include a category discriminator for determining the acquisition angle of the input image.

[0067] In response to obtaining the image features of the training samples, the image features of the training samples can be input into a category discriminator, so that the category discriminator evaluates the sample category to which the training samples belong, and the output category discriminator determines the discrimination category to which the training samples belong, so as to train the category discriminator's ability to discriminate the acquisition angle of the image, so that when performing mask extraction on the target object, the corresponding mask mapping method can be adaptively utilized based on the acquisition angle.

[0068] The discriminant category is the result of the category discriminator’s judgment on the sample category to which the training sample belongs, which may be consistent with or different from the true category of the training sample. Therefore, a loss function can be constructed through the true category and the discriminant category as the discriminant loss, so as to update the category discriminator using the discriminant loss to optimize its category discrimination ability. For example, the discriminant loss calculation formula of the category discriminator can be as follows:

[0069]

[0070] Among them, Loss ddl represents the discrimination loss; N represents the number of training samples; ω1 represents the weight factor; J i represents the i-th training sample (i.e. x i ) sample category, when x i For horizontal viewing samples, J i =0, when x i When J is the angle sample i =1; z i Indicates whether the i-th training sample has a mask annotation. When x i With mask annotation z i =0, when x i Without mask annotation i =1; as well as Represents the prediction result output by the category discriminator, such as x represents the output of the class discriminator i Identify the categories.

[0071] Please continue reading Figure 3 In one embodiment, the image processing model may further include a mask mapper for performing mask mapping processing on the training sample / image to be extracted.

[0072] Optionally, in this embodiment, multiple mask mappers may be included, that is, the number of mask mappers may be at least two, so that different mask mappers may be used to perform mask mapping processing on the horizontal samples and the angle samples. The mask mappers used for performing mask mapping on the same category of images may share model parameters / model weights.

[0073] Specifically, the image features of the training samples may be input into a mask mapper that matches the true category thereof, so that the mask mapper generates an adaptation mask based on the image features and may also map to the training samples.

[0074] Among them, during the training process of the image processing model, there may be errors in the mask prediction of the training samples. Therefore, a loss function can be constructed based on the predicted mask generated by the mask mapper and the true value as the mask loss. The mask loss can be used to update the mask mapper and the image processing model to update the mask extraction capability of the mask mapper.

[0075] For example, the loss formula of the mask extractor can be calculated as follows:

[0076]

[0077] Where MSE represents the mask loss; m represents the number of columns of pixels in the training sample; n represents the number of rows of pixels in the training sample; x represents the xth row pixel of the training sample; y represents the yth column pixel of the training sample; p xy Indicates the true mask of the pixel at row x and column y of the training sample. That is, if the training sample carries a mask annotation, this is the annotated mask value. If the training sample does not carry a mask annotation, this is the actual mask value to be assigned. It represents the mask assigned by the mask mapper to the pixel at row x and column y of the training sample, i.e., the predicted mask.

[0078] Furthermore, as described above, the training sample set includes horizontal samples and angle samples, and the target object extraction capability at different acquisition angles is trained based on the training sample set. Therefore, the specific loss value can be divided into the mask loss value MSR of angle acquisition v , the mask loss value MSE of the head-up acquisition h Mask loss value MSR v And the mask loss value MSE h Both of them can be constructed by using a preset mask and a real mask, and the calculation formulas can be as shown in Formula 1-2.

[0079] Therefore, the calculation formula for the mask loss of the target object can be as follows:

[0080]

[0081] Among them, Lmask represents the mask loss; N represents the number of training samples; ω2 represents the weighting factor, and ω2>1 can be constrained to ensure that there is a difference between the training of the image processing model under the angle acquisition state and the training under the level view state; M i represents the true mask of the training sample; T i Represents the predicted mask of the mask mapper for the training sample.

[0082] Optionally, the mask loss value MSR of the angle samples is calculated v Considering that most angle samples do not carry mask annotations, other relatively reliable models can be used to evaluate the target object information in the angle samples that do not carry mask annotations, such as identifying the position and contour of the target object, which is not limited here. And based on the confidence score of the evaluation result, relatively reliable angle samples can be selected to participate in the training of the image processing model, so as to further improve the effectiveness of the image processing model training.

[0083] Specifically, the target object information in the angle sample without mask annotation can be evaluated. The confidence score of the target object information is analyzed. In response to the confidence score being higher than the confidence threshold, the angle sample without mask annotation is allowed to be input into the image processing model, and the target object information is used as its virtual label. In other words, relatively unreliable angle samples can be filtered out through the confidence threshold to improve the effectiveness of image processing model training.

[0084] If so, the loss formulas for angle samples and head-on samples can be different, that is, the mask loss value MSR of all angle samples v (or the mask loss value MSR of the angle sample without mask annotation v ) can be calculated using a method different from the calculation method shown in formula 1-2. For example, the calculation formula for the total mask loss value of some or all angle samples can be as follows:

[0085]

[0086] in, Indicates the total mask loss value of the partial or all angle samples; Indicates whether the prediction of the i-th training sample under the current state φ of the image processing model is reliable; score(x i ) represents the training sample x i The confidence score at the current state φ; μ represents the confidence threshold.

[0087] Optionally, when the image processing model is updated, the current state φ and the confidence threshold μ may also be updated.

[0088] Furthermore, the discrimination loss and mask loss described above can be fused to obtain the training loss for updating the image processing model. The fusion can be performed by superposition, weighted fusion, etc., which are not limited here. For example, the calculation formula of the training loss can be as follows:

[0089] L=αLoss ddl +βL mask In formula 1-6, L represents the training loss; α and β represent weighting factors, which can constrain αβ<0.

[0090] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of the image processing model of the present application.

[0091] In one embodiment, the first model of the image processing model may include: a feature extractor 10, a category discriminator 20, and a mask mapper 30. The working principle of each module in the first model has been described above and will not be repeated here.

[0092] The following is an example of a detailed method for mask prediction of training samples using a specific image processing model architecture.

[0093] Furthermore, if Figure 4 As shown in the example, the feature extractor 10 may include a first feature extractor 11 and a second feature extractor 12. The first feature extractor 11 is used to perform feature extraction processing on the horizontal samples, and the second feature extractor 12 is used to perform feature extraction processing on the angle samples.

[0094] The mask mapper 30 may include a first mask mapper 31 and a second mask mapper 32. The first mask mapper 31 is used to perform mask mapping processing on the head-on samples, and the second mask mapper 32 is used to perform mask mapping processing on the angle samples.

[0095] In this way, when performing mask prediction training on training samples, the training samples can be input into the first feature extractor 11 or the second feature extractor 12 matched thereto to output the image features of the training samples. That is, when the training samples are frontal samples, they are input into the first feature extractor 11; when the training samples are angle samples, they are input into the second feature extractor 12. The model parameters / weights of the first feature extractor 11 and the second feature extractor 12 can be shared.

[0096] The image features are input into the category discriminator 20 to discriminate the sample category to which the training sample belongs and obtain the discrimination loss. The image features can be input into the first mask mapper 31 or the second mask mapper 32 matched therewith to output the prediction mask for dividing the target object and the environmental background in the training sample and obtain the mask prediction loss. In other words, when the training sample is a frontal sample, it is input into the first mask mapper 31; when the training sample is an angle sample, it is input into the second mask mapper 32.

[0097] The discriminative loss and the mask prediction loss can be fused to obtain the training loss for updating the first model.

[0098] For the specific definition of the image processing model, please refer to the definition of the image processing model training method and the target extraction method mentioned above, which will not be repeated here. Each module in the above image processing model can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0099] See also Figure 5 , Figure 5 It is a flowchart of another embodiment of the image processing model training method of the present application.

[0100] In this embodiment, the image processing model may include a first model and a second model, wherein the model size of the first model is larger than the model size of the second model. The model size may be expressed as a model parameter amount, that is, the model parameter amount of the second model is less than the model parameter amount of the first model.

[0101] In the image processing model training method described above, the first model in the image processing model may be trained based on a training sample set.

[0102] In response to the completion of the training of the first model, the first model can be miniaturized to obtain a second model with a smaller model size.

[0103] Specifically, the mask of the target object in the angle image can be predicted by the trained first model to obtain a predicted mask. The angle image is a sample for training the second model. The angle image and the training sample are the names of the samples used to relatively distinguish the first model from the second model training process. There may or may not be an intersection between the two, and no limitation is made here.

[0104] The second model can be trained based on the angle image and the predicted mask to reduce the model parameters of the target extraction process and reduce the model size of the model used in the actual target extraction process, which is conducive to realizing real-time sub-computation.

[0105] in, Figure 5 The example shown by the dashed arrow is a relatively simple process step of the target extraction process, which will not be repeated here.

[0106] In summary, the image processing model training and target extraction of the present application can take into account both real-time and high efficiency, and through strategies such as training sample set design, image processing model architecture design, and pseudo-mask design, the performance and efficiency of the image processing model can be improved, so as to improve the reliability and real-time performance of target object extraction from multiple acquisition angles, and enrich the application scenarios of the image processing model.

[0107] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the image processing model training method or the target extraction method described above are implemented, which will not be described in detail here.

[0108] It should be understood that although Figure 1-Figure 3 , Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 3 , Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0109] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer device of the present application.

[0110] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing model training method or a target extraction method is implemented.

[0111] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0113] S101: Acquire an image to be extracted.

[0114] S102: using an image processing model to annotate a target object in the image to be extracted; the image processing model is trained using an image processing model training method.

[0115] In this embodiment, the image processing model training method can be described in any of the following embodiments.

[0116] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0117] S201: Acquire a training sample set; the training samples include level-view samples and angle samples; wherein the level-view samples refer to samples collected at level-view angles, and the angle samples refer to samples collected at non-level-view angles.

[0118] S202: respectively inputting the training samples into the image processing model to be trained for mask prediction training, and outputting the masks of the training samples to separate the target object and the environmental background in the training samples.

[0119] S203: Update the image processing model using the training loss.

[0120] In one embodiment, the head-on samples and some angle samples carry mask annotations; when implementing step S202 in which the training samples are input into the image processing model to be trained, the processor further implements the following steps when executing the computer program: in response to the fact that the training samples input in the first number of forward adjacent training all carry mask annotations, the angle samples without mask annotations are input into the image processing model; in response to the fact that the training samples input in the second number of forward adjacent training do not carry mask annotations, the head-on samples or angle samples carrying mask annotations are input into the image processing model.

[0121] In one embodiment, before the angle samples without mask annotations are input into the image processing model, the processor further implements the following steps when executing the computer program: evaluating the target object information within the angle samples without mask annotations; analyzing the confidence score of the target object information; in response to the confidence score being higher than the confidence threshold, allowing the angle samples without mask annotations to be input into the image processing model, and using the target object information as its virtual label.

[0122] In one embodiment, the annotation includes mask annotation and position box annotation of the target object; when the training sample set is obtained in step S201, the processor further implements the following steps when executing the computer program: obtaining head-on samples with mask annotations; obtaining angle samples without annotations, annotating the masks of the target objects in some angle samples, annotating the position boxes of the target objects in other angle samples, and making the angle samples with position box annotations less than the angle samples without mask annotations.

[0123] In one embodiment, when performing mask prediction training in step S202, the processor further implements the following steps when executing the computer program: inputting the training sample into a feature extractor that matches its true category to output image features; wherein feature extraction weights are shared between feature extractors; inputting the image features into a category discriminator to output a discriminated category, and evaluating the discrimination loss based on the discriminated category and the true category; inputting the image features into a mask mapper that matches the true category to output a predicted mask that divides the target object and the environmental background, and evaluating the mask loss based on the predicted mask and the true value; and fusing the mask loss and the discrimination loss to obtain the training loss.

[0124] In one embodiment, the image processing model includes a first model and a second model, and the model parameter amount of the second model is less than the model parameter amount of the first model; when the processor executes the computer program, the following steps are also implemented: training the first model based on the training sample set; predicting the mask of the target object in the angle image by the trained first model to obtain a predicted mask; training the second model based on the angle image and the predicted mask.

[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0126] S101: Acquire an image to be extracted.

[0127] S102: using an image processing model to annotate a target object in the image to be extracted; the image processing model is trained using an image processing model training method.

[0128] In this embodiment, the image processing model training method can be described in any of the following embodiments.

[0129] In one embodiment, when the computer program is executed by a processor, the following steps are implemented:

[0130] S201: Acquire a training sample set; the training samples include level-view samples and angle samples; wherein the level-view samples refer to samples collected at level-view angles, and the angle samples refer to samples collected at non-level-view angles.

[0131] S202: respectively inputting the training samples into the image processing model to be trained for mask prediction training, and outputting the masks of the training samples to separate the target object and the environmental background in the training samples.

[0132] S203: Update the image processing model using the training loss.

[0133] In one embodiment, the head-on samples and some angle samples carry mask annotations; when the training samples are input into the image processing model to be trained in step S202, the computer program further implements the following steps when executed by the processor: in response to the training samples input in the first number of forward adjacent training all carrying mask annotations, the angle samples without mask annotations are input into the image processing model; in response to the training samples input in the second number of forward adjacent training all not carrying mask annotations, the head-on samples or angle samples carrying mask annotations are input into the image processing model.

[0134] In one embodiment, before the angle samples without mask annotations are input into the image processing model, the computer program, when executed by the processor, also implements the following steps: evaluating the target object information within the angle samples without mask annotations; analyzing the confidence score of the target object information; in response to the confidence score being higher than the confidence threshold, allowing the angle samples without mask annotations to be input into the image processing model, and using the target object information as its virtual label.

[0135] In one embodiment, the annotation includes mask annotation and position box annotation of the target object; when the training sample set is obtained in step S201, the computer program further implements the following steps when executed by the processor: obtaining head-on samples with mask annotations; obtaining angle samples without annotations, annotating the masks of the target objects in some angle samples, annotating the position boxes of the target objects in other angle samples, and making the angle samples with position box annotations less than the angle samples without mask annotations.

[0136] In one embodiment, when performing mask prediction training in step S202, the computer program further implements the following steps when executed by the processor: inputting the training sample into a feature extractor that matches its true category to output image features; wherein feature extraction weights are shared between feature extractors; inputting the image features into a category discriminator to output a discriminated category, and evaluating the discrimination loss based on the discriminated category and the true category; inputting the image features into a mask mapper that matches the true category to output a predicted mask that divides the target object and the environmental background, and evaluating the mask loss based on the predicted mask and the true value; and fusing the mask loss and the discrimination loss to obtain the training loss.

[0137] In one embodiment, the image processing model includes a first model and a second model, and the model parameter amount of the second model is less than the model parameter amount of the first model; when the computer program is executed by the processor, the following steps are also implemented: training the first model based on the training sample set; predicting the mask of the target object in the angle image by the trained first model to obtain a predicted mask; training the second model based on the angle image and the predicted mask.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The embodiments described above only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for training an image processing model, characterized in that: The image processing model training method comprises: Acquire a training sample set; the training samples include level-view samples and angle samples; wherein the level-view samples represent samples collected at level-view angles, and the angle samples represent samples collected at non-level-view angles; Inputting the training samples into the image processing model to be trained respectively for mask prediction training, and outputting the masks of the training samples to separate the target object and the environmental background in the training samples; The image processing model is updated using the training loss.

2. The image processing model training method according to claim 1, characterized in that: The head-on sample and part of the angle samples carry mask annotations; The step of inputting the training samples into the image processing model to be trained comprises: In response to the training samples inputted by the forward adjacent first number training all carrying mask annotations, the angle samples without mask annotations are inputted into the image processing model; wherein the forward adjacent first number represents the first number occurring before the current moment and adjacent to the current moment; In response to the fact that none of the training samples input for the forward adjacent second number training carry the mask annotation, the head-on sample or the angle sample carrying the mask annotation is input into the image processing model; wherein the forward adjacent second number represents the second number occurring before the current moment and adjacent to the current moment.

3. The image processing model training method according to claim 2, characterized in that: Before inputting the angle samples without mask annotation into the image processing model, the method includes: Evaluating target object information within the angle sample not carrying mask annotations; Analyzing the confidence score of the target object information; In response to the confidence score being higher than a confidence threshold, the angle sample without mask annotation is allowed to be input into the image processing model, and the target object information is used as its virtual label.

4. The image processing model training method according to claim 2, characterized in that: The annotation includes mask annotation and position box annotation of the target object; and the obtaining of the training sample set includes: Obtaining a head-up sample carrying the mask annotation; Angle samples are obtained, masks of target objects in some of the angle samples are marked, position boxes of target objects in other angle samples are marked, and the number of angle samples marked with the position boxes is greater than the number of angle samples marked with the masks.

5. The image processing model training method according to claim 1, characterized in that: The mask prediction training comprises: Input the training sample into a feature extractor that matches its true category to output image features; wherein feature extraction weights are shared between the feature extractors; Input the image features into a category discriminator to output a discriminated category, and evaluate a discriminative loss based on the discriminated category and the true category; Inputting the image features into a mask mapper matched with the true category to output a predicted mask for dividing the target object from the environmental background, and evaluating the mask loss based on the predicted mask and the true value; The mask loss and the discrimination loss are fused to obtain the training loss.

6. The image processing model training method according to claim 1, characterized in that: The image processing model includes a first model and a second model, wherein the model parameter amount of the second model is less than the model parameter amount of the first model; The image processing model training method also includes: Training the first model based on the training sample set; predicting the mask of the target object in the angle image by using the trained first model to obtain a predicted mask; The second model is trained based on the angle image and the predicted mask.

7. A target extraction method, characterized in that: The target extraction method comprises: Obtain the image to be extracted; The target object in the image to be extracted is extracted and labeled using an image processing model; the image processing model is trained using the image processing model training method according to any one of claims 1 to 6.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the steps of the image processing model training method described in any one of claims 1 to 6; or, it implements the steps of the computer program product described in any one of claim 7.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the image processing model training method described in any one of claims 1-6; or, it implements the steps of the computer program product described in any one of claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the image processing model training method described in any one of claims 1 to 6; or, it implements the steps of the computer program product described in any one of claim 7.

Images