Training method of blur detection model, blur detection method, device and equipment

Abstract

The application discloses a training method and device of a blur detection model, a blur detection method and device and equipment. The blur detection model comprises a virtual focus prediction network and a classification network. The virtual focus prediction network is trained first to obtain a trained virtual focus prediction network. Then, a first training target object image and a corresponding blur label are obtained, and the first training target object image is input into the trained virtual focus prediction network. The virtual focus prediction network can extract image features of the first training target object image, and the image features comprise relative depth features of the first training target object image. The obtained image features are input into the classification network to obtain a target object blur detection result. The classification network is trained by using a detection result loss generated by the target object blur detection result and the blur label. Whether an image is blurred is related to the relative depth of the image. The image features of the application are fused with the relative depth information of the image, and the detection effect of the classification network trained by using the image features is better.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a training method for a fuzz detection model, as well as a fuzz detection method, apparatus, and device. Background Technology

[0002] Typically, after acquiring an image, the objects within it may be blurry, which can affect subsequent processing. For example, if the product in a product image is blurry, it will be difficult to distinguish the product during recommendation, impacting the recommendation results. Therefore, blur detection is necessary to determine whether the objects in the image are blurry.

[0003] Current blur detection methods perform blur detection on every pixel in an image, but the blur detection results for each pixel do not accurately reflect whether the target object itself is blurry. Therefore, a better blur detection method is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a training method, apparatus, and device for a fuzzy detection model, as well as a fuzzy detection method, apparatus, and device that can train a high-performance fuzzy detection model and perform fuzzy detection of target objects in an image based on the fuzzy detection model, thereby obtaining accurate fuzzy detection results for the target object (which may be a commodity).

[0005] To achieve the above objectives, the technical solution provided in this application is as follows:

[0006] This application provides a training method for a fuzzy detection model, wherein the fuzzy detection model includes a defocus prediction network and a classification network, and the method includes:

[0007] Obtain the trained defocus prediction network; the trained defocus prediction network is used to extract image features of the target object image input to the defocus prediction network; the image features include the relative depth features of the target object image;

[0008] Obtain the first training target image and the corresponding fuzzy label;

[0009] The first training target image is input into the trained defocus prediction network to obtain the image features of the first training target image; the image features include the relative depth features of the first training target image;

[0010] The image features are input into a classification network to obtain the target object blur detection result output by the classification network;

[0011] Calculate the detection result loss based on the target object fuzzy detection result and the fuzzy label;

[0012] The classification network is trained based on the loss from the detection results.

[0013] Optionally, the training process of the defocus prediction network includes:

[0014] The second training target image is input into the defocus prediction network to obtain the image features of the second training target image extracted by the defocus prediction network and the predicted defocus image output by the defocus prediction network; the pixel values ​​of the pixels in the predicted defocus image are used to predict whether the pixels at the same position in the second training target image are within the depth of field corresponding to the image imager.

[0015] Based on the predicted out-of-focus image and the real out-of-focus image corresponding to the second training target image, the image loss is calculated; the pixel value of the pixel in the real out-of-focus image is used to indicate whether the pixel at the same position in the second training target image is within the depth of field corresponding to the image imager.

[0016] Obtain the relative depth features of the second training target image;

[0017] Calculate the feature loss based on the image features and the relative depth features;

[0018] The defocus prediction network is trained based on the image loss and the feature loss.

[0019] The image features of the target object image extracted by the trained defocus prediction network include the relative depth features of the target object image.

[0020] Optionally, obtaining the relative depth features of the second training target image includes:

[0021] The second training target image is input into the trained relative depth estimation network to obtain the relative depth features of the second training target image extracted by the relative depth estimation network.

[0022] The training process of the relative depth estimation network includes:

[0023] The third training target image is input into the relative depth estimation network to obtain the predicted relative depth features of the third training target image output by the relative depth estimation network.

[0024] The depth loss is calculated based on the predicted relative depth features and the actual relative depth features of the third training target image;

[0025] The relative depth estimation network is trained based on the depth loss.

[0026] Optionally, the defocus prediction network is a multi-scale fully convolutional network, which includes multiple first upsampling layers and multiple first downsampling layers, and the image features are multi-scale image features, including the image features output by each of the first upsampling layers.

[0027] The relative depth estimation network is a multi-scale fully convolutional network, which includes multiple second upsampling layers and multiple second downsampling layers. The relative depth features are multi-scale relative depth features, including the image features output by each second upsampling layer.

[0028] This application provides a fuzzy detection method, the method comprising:

[0029] Acquire at least one image of the target object;

[0030] The target image is input into a blur detection model to obtain the target blur detection result output by the blur detection model; the target blur detection result is used to indicate whether the target object in the target image is blurred.

[0031] The fuzzy detection model consists of a defocus prediction network and a classification network; the fuzzy detection model is trained according to the training method of the fuzzy detection model described above.

[0032] Optionally, acquiring at least one target image includes:

[0033] Obtain the video frames to be detected;

[0034] The video frame to be detected is input into the target object detection network to obtain the target object detection result of the video frame to be detected;

[0035] Based on the target object detection results, the video frame to be detected is cropped to obtain at least one target object image in the video frame to be detected.

[0036] This application provides a training device for a fuzzy detection model, the fuzzy detection model including a defocus prediction network and a classification network, the device comprising:

[0037] The first acquisition unit is used to acquire the trained defocus prediction network; the trained defocus prediction network is used to extract image features of the target image input to the defocus prediction network; the image features include the relative depth features of the target image;

[0038] The second acquisition unit is used to acquire the first training target image and the fuzzy label corresponding to the first training target image;

[0039] The first input unit is used to input the first training target image into the trained defocus prediction network to obtain the image features of the first training target image; the image features include the relative depth features of the first training target image.

[0040] The second input unit is used to input the image features into the classification network and obtain the target object blur detection result output by the classification network.

[0041] The calculation unit is used to calculate the detection result loss based on the fuzzy detection result of the target object and the fuzzy label;

[0042] An execution unit is used to train the classification network based on the detection results.

[0043] This application provides a fuzzy detection device, the device comprising:

[0044] Acquisition unit, used to acquire at least one image of a target object;

[0045] The input unit is used to input the target image into the blur detection model to obtain the target blur detection result output by the blur detection model; the target blur detection result is used to indicate whether the target object in the target image is blurred.

[0046] The fuzzy detection model consists of a defocus prediction network and a classification network; the fuzzy detection model is trained according to the training method of the fuzzy detection model described above.

[0047] This application provides an electronic device, including:

[0048] One or more processors;

[0049] Storage device, on which one or more programs are stored,

[0050] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the fuzz detection model as described above, or the fuzz detection method as described above.

[0051] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a fuzzy detection model as described above, or a fuzzy detection method as described above.

[0052] As can be seen from the above technical solution, this application has the following beneficial effects:

[0053] This application provides a training method for a blur detection model, which includes a defocus prediction network and a classification network. First, the defocus prediction network is trained to obtain a trained defocus prediction network. The trained defocus prediction network is used to extract image features from the target object image input to the defocus prediction network. These image features include the relative depth features of the target object image. Based on this, a first training target object image and its corresponding blur label are obtained to train the classification network. The blur label indicates whether the target object in the first training target object image is blurred. Specifically, the first training target object image is input into the trained defocus prediction network to obtain the image features extracted by the defocus prediction network. These image features include the relative depth features of the first training target object image. The relative depth features represent the relative depth information of the first training target object image. Further, the image features are input into the classification network to obtain the target object blur detection result. The classification network is trained using the target object blur detection result and the detection result loss generated by the blur label.

[0054] It is known that whether an image is blurry is related to the relative depth of the image. In this application, a defocus prediction network is pre-trained. The image features extracted by the trained defocus prediction network are fused with the relative depth information of the first training target image, thus introducing relative depth information into the blur detection process. Therefore, the classification network trained using these image features exhibits better detection performance, thereby improving the detection accuracy of the blur detection model. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a training method for a fuzzy detection model provided in an embodiment of this application;

[0057] Figure 2 A flowchart illustrating the training of a defocus prediction network provided in this application embodiment;

[0058] Figure 3 This is a schematic diagram of the structure of a fuzzy detection model provided in an embodiment of this application;

[0059] Figure 4 A flowchart of a fuzz detection method provided in an embodiment of this application;

[0060] Figure 5 A schematic diagram of the structure of a training device for a fuzzy detection model provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of a fuzzy detection device provided in an embodiment of this application. Detailed Implementation

[0062] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0063] To facilitate understanding and explanation of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be introduced first.

[0064] Typically, after acquiring an image, the objects within it may be blurry, which can affect subsequent processing. For example, when recommending similar products in movies or TV shows, if the video frames have background blur, the products (i.e., the objects in the video frames) may also be blurry. Recommending blurry products makes them difficult to distinguish, negatively impacting the recommendation results. Therefore, it is necessary to perform blur detection on the objects in the image to determine whether they are blurry.

[0065] Currently, blur detection methods in related technologies perform blur detection on each pixel in an image. However, the blur detection results for each pixel do not accurately reflect whether the object itself is blurry. Therefore, a new blur detection method is urgently needed.

[0066] Based on this, embodiments of this application provide a training method for a blur detection model, which includes a defocus prediction network and a classification network. First, the defocus prediction network is trained to obtain a trained defocus prediction network. The trained defocus prediction network is used to extract image features of the target object image input to the defocus prediction network, including the relative depth features of the target object image. Based on this, a first training target object image and its corresponding blur label are obtained, and the classification network is trained using the first training target object image and its corresponding blur label. The blur label indicates whether the target object in the first training target object image is blurred. Specifically, the first training target object image is input into the trained defocus prediction network to obtain the image features of the first training target object image extracted by the defocus prediction network, including the relative depth features of the first training target object image. The relative depth features represent the relative depth information of the first training target object image. Further, the image features are input into the classification network to obtain the target object blur detection result. The classification network is trained using the target object blur detection result and the detection result loss generated by the blur label.

[0067] It is known that whether an image is blurry is related to the relative depth of the image. In this application, a defocus prediction network is pre-trained. The image features extracted by the trained defocus prediction network are fused with the relative depth information of the first training target image, thus introducing relative depth information into the blur detection process. Therefore, the classification network trained using these image features exhibits better detection performance, thereby improving the detection accuracy of the blur detection model.

[0068] It is understandable that the shortcomings of the above solutions are the result of the applicant's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application below should be considered contributions made by the applicant to the embodiments of this application.

[0069] To facilitate understanding of this application, a training method for a fuzzy detection model provided in an embodiment of this application will be described below with reference to the accompanying drawings. For example, this training method for the fuzzy detection model can be implemented by a terminal device or a server.

[0070] See Figure 1 As shown, this figure is a flowchart of a training method for a fuzzy detection model provided in an embodiment of this application. Figure 1 As shown, the method may include S101-S106:

[0071] S101: Obtain the trained defocus prediction network; the trained defocus prediction network is used to extract image features of the target image input to the defocus prediction network; the image features include the relative depth features of the target image.

[0072] In this embodiment, the fuzzy detection model includes a defocus prediction network and a classification network. The training process of the fuzzy detection model includes training the defocus prediction network and training the classification network. Specifically, the defocus prediction network is trained first, and the trained defocus prediction network is obtained. Then, the classification network is trained, and after the classification network training is completed, the trained classification network is obtained. Thus, the model training is completed, and the trained fuzzy detection model includes the trained defocus prediction network and the trained classification network.

[0073] A defocus prediction network is a network that takes an image of a target object as input and outputs a defocused image of that target object. The target object image is an image that includes the target object. The output defocused image is a binary image, where the pixel values ​​of each pixel in the defocused image indicate whether pixels at the same location in the target object image are within the depth of field corresponding to the image imager. The image imager can be, for example, an image capturing device, and the target object image can be considered to have been acquired by the image imager, such as by taking a picture with the image imager. Depth of field refers to the range of distances in front of and behind the subject that can be imaged in sharp focus from the front edge of the image imager.

[0074] In some optional examples, the defocus prediction network can be a fully convolutional network. In a fully convolutional network, all layers are convolutional layers; the type of fully convolutional network is not limited here and can be selected according to actual needs.

[0075] In one possible implementation, this application provides a training method for a defocus prediction network, as detailed in S201-S205 below. During the training of the defocus prediction network, the relative depth features of the input image are used as supervision information. This ensures that the image features extracted by the trained defocus prediction network include the relative depth features of the input image. The specific training process of the defocus prediction network is detailed below. The input image is the image input to the defocus prediction network during training.

[0076] Based on this, by inputting the target image into the trained defocus prediction network, the image features of the target image extracted by the defocus prediction network can be obtained, and these image features include the relative depth features of the target image. This allows subsequent training of the classification network using image features including relative depth features, resulting in better training performance and ultimately higher detection accuracy of the trained blur detection model.

[0077] Image features refer to a set of attribute information that can characterize the features or content of an image. Image features can be represented by feature vectors or feature maps. Relative depth features can also be represented by feature vectors or feature maps, without limitation. Relative depth features are used to represent the relative depth information of an image. In this embodiment, depth refers to the distance of each pixel in the image relative to the shooting source (such as an image imager). Since it is impossible to accurately mark the absolute depth of a pixel from the shooting source manually, only pixels closer to the shooting source and pixels farther away from the shooting source can be marked. The quantity representing the distance closer to the shooting source and the distance farther away from the shooting source is the relative depth. For example, if the normalized relative depth range is [0,1], and the two relative depths are 0.5 and 0.6 respectively, then the first relative depth indicates that the pixel is closer to the shooting source, and the second relative depth indicates that the pixel is farther away from the shooting source.

[0078] As an optional example, the defocus prediction network includes a target network layer, which extracts image features of the target image input to the defocus prediction network. The target network layer is a network layer within the defocus prediction network. For details regarding the target network layer, please refer to the training method of the defocus prediction network in sections S201-S205 below.

[0079] S102: Obtain the first training target image and the corresponding fuzzy label of the first training target image.

[0080] The first training target image is an image that includes the target object, which is the object in the image that the user needs to determine whether it is blurry. For example, when the application scenario is product recommendation in a movie or TV show, the target object could be the product in a video frame of the movie or TV show.

[0081] The first training target image can be obtained by extracting video frames from movies or TV shows. As an alternative example, the video frames can be input into a trained target detection network, which then detects the targets in the video frames and extracts images containing the targets. The target detection network can be a neural network, used to detect targets in the video frames based on target detection algorithms.

[0082] In practical applications, when an object detection network detects an object in a video frame, it uses a pre-selection box to select the object. Based on this, the captured image containing the object (i.e., the object image) can be the image selected by the pre-selection box in the video frame, or it can be an image obtained by expanding the surrounding area based on the image selected by the pre-selection box; there is no limitation here.

[0083] As another alternative example, the first training target image can also be obtained manually from video frames of movies and TV shows.

[0084] The fuzzy label corresponding to the first training target image is used to indicate whether the target object in the first training target image is blurry. The fuzzy label corresponding to the first training target image can be obtained manually after acquisition.

[0085] For example, the blur label can be 1 or 0. A blur label of 1 indicates that the target object is blurry, and a blur label of 0 indicates that the target object is clear. It should be understood that the embodiments of this application do not limit the specific value of the blur label or the meaning of the value, and can be determined according to the actual situation.

[0086] Understandably, the first training target image and its corresponding fuzzy label are used to train the classification network in the fuzzy detection model.

[0087] S103: Input the first training target image into the trained defocus prediction network to obtain the image features of the first training target image; the image features include the relative depth features of the first training target image.

[0088] After the defocus prediction network is trained, the classification network can be trained. The classification network can be trained based on the first training target image and its corresponding blurred label.

[0089] Understandably, since the parameters of the defocus prediction network are optimal parameters obtained through training, these parameters can be frozen during the training of the classification network after the defocus prediction network is trained. "Freezing" means that during the training of the classification network, the defocus prediction network's parameters are used directly without retraining it. For example, in this step, the trained defocus prediction network can be used directly.

[0090] In practice, the first training target image is input into the trained defocus prediction network. Before the defocus prediction network outputs the defocused image corresponding to the first training target image, it performs feature extraction on the first training target image to obtain its image features. These image features include the relative depth features of the first training target image.

[0091] It is known that an image imager produces a clear image at depth of field, while the image before and after the depth of field is blurred. Depth of field refers to the relative depth range that makes an image clear. Therefore, there is a certain relationship between the relative depth of the pixels in an image and the blurriness of the target object in the image. Therefore, in this embodiment, the relative depth features of the image are incorporated into the image features. Subsequently, the image features including the relative depth features are used to train the classification network, which can improve the training effect of the classification network and thus improve the detection accuracy of the trained blur detection model.

[0092] S104: Input the image features into the classification network and obtain the target object blur detection result output by the classification network.

[0093] Image features, including the relative depth features of the first training target image, are input into the classification network to obtain the target blur detection result output by the classification network.

[0094] As an optional example, the object blur detection result is used to predict whether the object in the first training object image is blurry. For example, the object blur detection result may be 1 or 0. When the object blur detection result is 1, it indicates that the object is predicted to be blurry, and when the object blur detection result is 0, it indicates that the object is predicted to be sharp.

[0095] As another alternative example, the object blur detection result can also be used to represent the probability that an object in the first training object image is blurred, which can be represented by probability. Furthermore, a blur threshold can be set, and the blur threshold and the object blur detection result can be used to determine whether the object is blurred. For example, when the object blur detection result is greater than the blur threshold, the object is considered blurred; when the object blur detection result is less than or equal to the blur threshold, the object is considered sharp. For example, if the blur threshold is 0.6 and the object blur detection result is 0.8, the object can be considered blurred.

[0096] It is understood that the embodiments of this application do not limit the target object blur detection results, and can be determined according to the actual situation.

[0097] For example, the structure of the classification network in the embodiments of this application can be a neural network. The specific structure of the classification network is not limited here, as long as the classification network can perform the classification task.

[0098] S105: Calculate the detection result loss based on the fuzzy detection result of the target object and the fuzzy label.

[0099] Understandably, the target object fuzzy detection result is the predicted value, and the fuzzy label is the true value. Based on the target object fuzzy detection result and the fuzzy label, the detection result loss is calculated. The detection result loss is used to measure the difference between the target object fuzzy detection result and the fuzzy label.

[0100] In practice, a loss function can be constructed that includes the fuzzy detection results and fuzzy labels of the target object. After obtaining the fuzzy detection results and fuzzy labels of the target object, the fuzzy detection results and fuzzy labels of the target object are input into the loss function to calculate the loss value, which is the detection result loss.

[0101] It is understood that the embodiments of this application do not limit the specific formula of the loss function in this step, and can be set according to requirements.

[0102] S106: Train the classification network based on the loss of the detection results.

[0103] In practice, after obtaining the detection result loss, the classification network is trained using this loss, and the network parameters are adjusted. After adjusting the network parameters, it is determined whether a first preset condition has been met. If yes, the classification network training is complete, and a trained classification network is obtained. If not, the classification network training is not complete. In this case, the image features of the first training target image are re-input into the adjusted classification network to obtain the blurred detection result of the target object output by the classification network again. Then, based on the blurred detection result and blurred label of the target object output again, the detection result loss is recalculated, and the network parameters of the classification network are adjusted again based on the recalculated detection result loss until the first preset condition is met.

[0104] As an optional example, the first preset condition is reaching a first number of training iterations or the detection result loss reaching a first preset range. This application embodiment does not limit the first number of training iterations or the first preset range; they can be selected according to actual circumstances. When the first preset condition is reached, it indicates that the detection result loss approaches 0, the difference between the blurred detection result of the target object and the blurred label is sufficiently small, the blurred detection result of the target object output by the classification network approaches the blurred label, and the training of the classification network ends. The trained classification network is then combined with the trained defocus prediction network to obtain the trained blurred detection model.

[0105] As can be seen, the trained blur detection model is used as input to a target image and outputs a blur detection result for the target object. Based on the blur detection result, it can be determined whether the target object in the image is blurred. Because the image features input to the classification network during training include relative depth features, the training process of the classification network takes into account the relative depth information of image pixels, resulting in better training performance of the classification network and thus better detection performance of the obtained blur detection model.

[0106] Based on the above S101-S106, this application embodiment provides a training method for a blur detection model, which includes a defocus prediction network and a classification network. First, the defocus prediction network is trained to obtain a trained defocus prediction network. The trained defocus prediction network is used to extract image features of the target object image input to the defocus prediction network, including the relative depth features of the target object image. Based on this, a first training target object image and its corresponding blur label are obtained, and the classification network is trained using the first training target object image and its corresponding blur label. The blur label indicates whether the target object in the first training target object image is blurred. Specifically, the first training target object image is input into the trained defocus prediction network to obtain the image features of the first training target object image extracted by the defocus prediction network. These include the relative depth features of the first training target object image. The relative depth features represent the relative depth information of the first training target object image. Further, the image features are input into the classification network to obtain the target object blur detection result. The classification network is trained using the target object blur detection result and the detection result loss generated by the blur label.

[0107] It is known that whether an image is blurry is related to the relative depth of the image. In this application, a defocus prediction network is pre-trained. The image features extracted by the trained defocus prediction network are fused with the relative depth information of the first training target image, thus introducing relative depth information into the blur detection process. Therefore, the classification network trained using these image features exhibits better detection performance, improving the accuracy of the blur detection model.

[0108] To facilitate understanding of the defocus prediction network provided in the embodiments of this application, the training process of the defocus prediction network will be described below with reference to the accompanying drawings.

[0109] See Figure 2 , Figure 2 This is a flowchart illustrating a training network for predicting out-of-focus conditions, provided as an embodiment of this application. Figure 2 As shown, the method may include the following steps S201-S205:

[0110] S201: Input the second training target image into the defocus prediction network, obtain the image features of the second training target image extracted by the defocus prediction network and the predicted defocus image output by the defocus prediction network; the pixel values ​​of the pixels in the predicted defocus image are used to predict whether the pixels at the same position in the second training target image are within the depth of field corresponding to the image imager.

[0111] The second training target image is the input image used to train the defocus prediction network. It can be the same image as the first training target image or a different image; this is not a limitation here. The method for obtaining the second training target image is similar to that for the first training target image, and will not be described again here.

[0112] The second training target image is input into the defocus prediction network. This network extracts image features from the second training target image and outputs a defocused image. During training, the network outputs a predicted defocused image, which is the prediction result of the defocused image. It's understood that the size of the predicted defocused image corresponding to the second training target image is the same as the size of the second training target image. The pixel values ​​of pixels in the predicted defocused image are used to predict whether pixels at the same location in the second training target image are within the depth of field corresponding to the image imager. Here, "same location" refers to the same pixel position. For example, the pixel value of pixel A in the predicted defocused image is used to predict whether pixel a in the second training target image is within the depth of field corresponding to the image imager. The pixel position of pixel A in the predicted defocused image is the same as the pixel position of pixel a in the second training target image.

[0113] Understandably, an image imager produces a sharp image within the depth of field, while images appear blurred in front of and behind the depth of field. Depth of field includes foreground and background depth of field, with the point between these two depths being the focus point; therefore, depth of field can also be understood as the focal region. A defocused image is a binary image; for example, a pixel in a defocused image has a pixel value of either 1 or 0. A pixel value of 1 indicates that the pixel at the same location in the first training target image is within the depth of field (or focal region) corresponding to the image imager, and the pixel is sharp. A pixel value of 0 indicates that the pixel at the same location in the first training target image is not within the depth of field (or focal region) corresponding to the image imager, and the pixel is blurred.

[0114] As an optional example, the defocus prediction network includes a target network layer. After the second training target image is input into the defocus prediction network, the target network layer extracts image features from the second training target image. It is understood that the embodiments of this application do not limit the target network layer in the defocus prediction network; the target network layer in the defocus prediction network can be determined according to actual circumstances.

[0115] As an alternative example, the defocus prediction network can be a fully convolutional network, such as a multi-scale fully convolutional network. When the defocus prediction network is a multi-scale fully convolutional network, the image features of the second training target image are multi-scale image features.

[0116] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a fuzzy detection model provided in an embodiment of this application. Figure 2 The defocus prediction network shown is a multi-scale fully convolutional network. When training the defocus prediction network, Figure 2 The input image in the training is specifically the image of the second training target object.

[0117] In some optional examples, the multi-scale fully convolutional network includes multiple first upsampling layers and multiple first downsampling layers. In this case, the target network layer in the defocus prediction network may include multiple first upsampling layers.

[0118] Specifically, the processing of multi-scale fully convolutional networks includes downsampling and upsampling. First, downsampling is performed based on multiple first downsampling layers, followed by upsampling based on multiple first upsampling layers. Downsampling, or reducing the size of the input image, can be understood as shrinking the image. Upsampling, on the other hand, expands or enlarges the image or feature map so that the final output image is the same size as the second training target image. Upsampling is also known as image interpolation or image enlargement, and its main purpose is to enlarge the image.

[0119] like Figure 2 As shown, downsampling is implemented by four convolutional layers (i.e., four first downsampling layers), and upsampling is also implemented by four convolutional layers (i.e., four first upsampling layers). It is understood that the embodiments of this application do not limit the number of convolutional layers in the multi-scale fully convolutional network, nor do they limit the number of scales. Figure 2 This is just an example.

[0120] When the defocus prediction network is a multi-scale fully convolutional network, the image features include the image features output by each upsampling convolutional layer during the upsampling process, that is, the image features output by each first upsampling layer. It can be understood that the scale of the image features output by each first upsampling layer is different; therefore, the image features in this case are multi-scale image features. For example, Figure 2 The multi-scale image features in the model include image features at four scales of the second training target image output by the four upsampling convolutional layers during the upsampling process.

[0121] It can be seen that when the structure of the virtual focus prediction network is Figure 2 In the structure shown, after the defocus prediction network is trained, during the training of the classification network, the image features input to the classification network include image features at four scales of the first training target image. At this point, the image features input to the classification network are either the image features of these four scale images, or the image features of the four scale images are concatenated before being input into the classification network.

[0122] It is understandable that when the defocus prediction network is a multi-scale fully convolutional network, the extracted image features are multi-scale image features. Since image features at different scales represent different feature information, the more scales there are, the richer the feature information, which makes the subsequent classification network trained based on multi-scale image features perform better after training the defocus prediction network.

[0123] S202: Calculate the image loss based on the predicted defocused image and the real defocused image corresponding to the second training target image; the pixel value of the pixel in the real defocused image is used to indicate whether the pixel at the same position in the second training target image is within the depth of field corresponding to the image imager.

[0124] The true out-of-focus image serves as the label image. The pixel values ​​of the pixels in the true out-of-focus image are used to accurately represent whether pixels at the same location in the second training target image are within the depth of field corresponding to the image imager. In practical applications, the true out-of-focus image can be obtained through manual annotation.

[0125] After acquiring the predicted out-of-focus image, an image loss is calculated based on the true out-of-focus image corresponding to the predicted out-of-focus image and the second training target image. The image loss measures the difference between the predicted and true out-of-focus images. This image loss is used to train the out-of-focus prediction network.

[0126] As an optional example, an image loss function can be constructed first. After obtaining the predicted out-of-focus image and the true out-of-focus image, the predicted out-of-focus image and the true out-of-focus image are input into the image loss function to calculate the image loss. It is understood that the embodiments of this application do not limit the specific formula of the image loss function in this step, and can be set according to requirements.

[0127] S203: Obtain the relative depth features of the second training target image.

[0128] In one optional implementation, this application provides a specific implementation for obtaining the relative depth features of a second training target image, including:

[0129] The image of the second training target is input into the trained relative depth estimation network to obtain the relative depth features of the second training target image extracted by the relative depth estimation network.

[0130] The relative depth estimation network outputs relative depth features, which are then used to supervise the training of the defocus prediction network. The relative depth features of the second training target image represent the relative depth information of that image.

[0131] The relative depth estimation network is pre-trained and is only used in the training process of the blur prediction network. After the blur prediction network is trained, the relative depth estimation network is no longer needed in the training and application processes of the blur detection model.

[0132] It should be noted that when the defocus prediction network is a multi-scale fully convolutional network, the relative depth estimation network is also a multi-scale fully convolutional network, and the number of scales in the defocus prediction network and the relative depth estimation network are the same. As an optional example, the relative depth estimation network includes multiple second upsampling layers and multiple second downsampling layers. The number of second upsampling layers is the same as the number of first upsampling layers, and the number of second downsampling layers is the same as the number of first downsampling layers. Furthermore, when the relative depth estimation network is a multi-scale fully convolutional network, the relative depth features are multi-scale relative depth features, which are also the features output during the upsampling process of the relative depth estimation network. Specifically, when the multi-scale image features include the image features output by each first upsampling layer, the multi-scale relative depth features include the image features output by each second upsampling layer.

[0133] In one possible implementation, embodiments of this application provide a specific implementation for training a relative depth estimation network, including:

[0134] Input the third training target image into the relative depth estimation network, and obtain the predicted relative depth features of the third training target image output by the relative depth estimation network;

[0135] The depth loss is calculated based on the predicted relative depth features and the actual relative depth features of the third training target image.

[0136] A relative depth estimation network is trained based on depth loss.

[0137] It is understood that the third training target image may be the same as or different from the first training target image, the second training target image, etc., and this is not limited here. The method of obtaining the third training target image is similar to that of obtaining the first training target image, and will not be repeated here.

[0138] The predicted relative depth feature is the predicted value, and the actual relative depth feature is the label value. In practical applications, the actual relative depth feature can be obtained through manual annotation. Depth loss is used to measure the difference between the predicted and actual relative depth features. Specifically, a depth loss function can be constructed first. After obtaining the predicted and actual relative depth features, these features are input into the depth loss function to obtain the depth loss. It should be noted that the embodiments in this application do not limit the specific formula of the depth loss function; it can be set according to the actual situation.

[0139] Depth loss is used to train the relative depth estimation network and adjust its parameters. Then, it is determined whether a third preset condition has been met. If so, the training of the relative depth estimation network ends. If not, the process of inputting the third training target image into the relative depth estimation network, obtaining the predicted relative depth features output by the network, and subsequent steps is repeated until the third preset condition is met. The third preset condition is reaching a third training iteration or the detection result loss reaching a third preset range. This embodiment does not limit the third training iteration or the third preset range; they can be selected according to actual circumstances.

[0140] S204: Calculate the feature loss based on image features and relative depth features.

[0141] Image features can be represented using feature maps, and so can relative depth features. When image features are represented using feature maps, relative depth features also need to be represented using feature maps to ensure consistency in the representation of image features and relative depth features.

[0142] In this step, the image features specifically refer to the image features of the second training target image, which are the predicted values. To ensure that the relative depth features serve as supervisory information to oversee the training process of the defocus prediction network, this embodiment uses the relative depth features output by the relative depth estimation network as the label values. During the training of the defocus prediction network, the image features, or a subset of the image features, are made to approximate the relative depth features. Here, the selection of a subset of image features is not limited to any specific feature; it can be chosen according to actual needs.

[0143] Specifically, feature loss is calculated based on image features and relative depth features. When it is necessary to make image features approximate relative depth features, the feature loss measures the difference between the image features and the relative depth features. When it is necessary to make a portion of the image features approximate relative depth features, the feature loss measures the difference between that portion of the image features and the relative depth features. Therefore, by training the defocus prediction network and adjusting its network parameters, the magnitude of the feature loss can be controlled, thereby achieving the goal of making image features or a portion of image features approximate relative depth features.

[0144] S205: Train the defocus prediction network based on image loss and feature loss.

[0145] After obtaining the image loss and feature loss, a defocus prediction network is trained based on the image loss and feature loss. The network parameters of the defocus prediction network are adjusted, and it is determined whether a second preset condition is met. If yes, the defocus prediction network training ends, and the trained defocus prediction network is obtained. If not, S201 and subsequent steps are repeated until the second preset condition is met, and the trained defocus prediction network is obtained. The second preset condition is either reaching a second number of training iterations or the detection result loss reaching a second preset range. This embodiment does not limit the second number of training iterations or the second preset range; they can be selected according to actual circumstances.

[0146] Understandably, training a defocus prediction network using image loss can lead to a more accurate output of defocused images, resulting in more accurate feature extraction. Using feature loss in training ensures that the extracted target image features include relative depth features, thus fusing relative depth information and improving the performance of subsequent classification networks. Specifically, if the target network layer of the defocus prediction network extracts image features for a second training target image, then the feature extraction for the target image will be performed by the target network layer.

[0147] Based on the above S201-S205, the second training target image is input into the defocus prediction network to obtain the image features of the second training target image extracted by the defocus prediction network and the predicted defocus image output by the defocus prediction network. Image loss is calculated based on the predicted defocus image and the corresponding real defocus image of the second training target image. The second training target image is input into the trained relative depth estimation network to obtain the relative depth features of the second training target image extracted by the relative depth estimation network. Feature loss is calculated based on the image features and relative depth features. The defocus prediction network is trained based on the image loss and feature loss. In this embodiment, a relative depth estimation network is introduced during the training of the defocus prediction network. The relative depth features extracted by the relative depth estimation network are used as supervisory information to supervise the training of the defocus prediction network, enabling the image features extracted by the trained defocus prediction network to fuse with the relative depth information of the input image, facilitating subsequent training of the classification network.

[0148] Based on the above method embodiments, after training the blur detection model, the blur detection model is applied to perform blur detection of target objects in video frames. See also... Figure 4 , Figure 4 A flowchart of a fuzz detection method provided in an embodiment of this application is shown below. Figure 4As shown, the method may include S401-S402:

[0149] S401: Acquire at least one image of a target object.

[0150] In one possible implementation, this application provides a specific method for acquiring an image of at least one target object, including:

[0151] Obtain the video frames to be detected;

[0152] Input the video frame to be detected into the target object detection network to obtain the target object detection result of the video frame to be detected;

[0153] Based on the target object detection results, image cropping is performed on the video frame to be detected to obtain at least one target object image in the video frame to be detected.

[0154] The process involves inputting the video frame to be detected into a target object detection network to obtain the target object detection result. This target object detection result can be a pre-selected bounding box marked in the video frame, with the target object enclosed within the pre-selected box. Then, based on the target object detection result, image cropping is performed on the video frame to obtain at least one target object image. Specifically, the cropped target object image can be the image enclosed by the pre-selected box, or an image obtained by expanding the surrounding area of ​​the image enclosed by the pre-selected box.

[0155] The video frames to be detected can be one or more video frames from a movie or TV show. The target object detection network can be a product detection network, where the target object to be detected is the product in the video frame, and the acquired target object image is the product image.

[0156] S402: Input the target image into the fuzzy detection model to obtain the target fuzzy detection result output by the fuzzy detection model; the target fuzzy detection result is used to indicate whether the target in the target image is blurred.

[0157] The fuzzy detection model consists of a defocus prediction network and a classification network; the fuzzy detection model is trained according to the training method of the fuzzy detection model described in any of the above embodiments.

[0158] In practice, the target image is first input into a defocus detection network to obtain image features extracted by the network. Then, these image features are input into a classification network to obtain the target blur detection result output by the classification network.

[0159] When the target object is detected as blurry, the target object in the video frame to be detected is marked as blurry; when the target object is detected as clear, the target object in the video frame to be detected is marked as clear. When the target object is a product, the video frame can be processed according to the target object blur detection result to reduce the occurrence of blurry products in the video frame, so that product recommendations can meet user needs and improve user experience.

[0160] It is understood that the technical implementation of the virtual focus prediction network and the classification network can be found in the above-mentioned method implementation examples, and will not be repeated here.

[0161] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0162] Based on the fuzzy detection model training method provided in the above-described embodiments, this application also provides a fuzzy detection model training device. The fuzzy detection model training device will be described below with reference to the accompanying drawings. Since the principle of the device in this disclosure for solving the problem is similar to the fuzzy detection model training method described above in this application, the implementation of the device can refer to the implementation of the method, and repeated details will not be elaborated further.

[0163] See Figure 5 The figure is a schematic diagram of the structure of a training device for a fuzzy detection model provided in an embodiment of this application. The fuzzy detection model includes a defocus prediction network and a classification network. Figure 5 As shown, the training device for the fuzz detection model includes:

[0164] The first acquisition unit 501 is used to acquire the trained defocus prediction network; the trained defocus prediction network is used to extract image features of the target image input to the defocus prediction network; the image features include the relative depth features of the target image;

[0165] The second acquisition unit 502 is used to acquire the first training target image and the fuzzy label corresponding to the first training target image;

[0166] The first input unit 503 is used to input the first training target image into the trained defocus prediction network to obtain the image features of the first training target image; the image features include the relative depth features of the first training target image.

[0167] The second input unit 504 is used to input the image features into the classification network and obtain the target object blur detection result output by the classification network.

[0168] The calculation unit 505 is used to calculate the detection result loss based on the target object fuzzy detection result and the fuzzy label;

[0169] Execution unit 506 is used to train the classification network based on the detection results.

[0170] In one optional implementation, the apparatus further includes: a defocus prediction network training unit;

[0171] The defocus prediction network training unit includes:

[0172] The first input subunit is used to input the second training target image into the defocus prediction network, obtain the image features of the second training target image extracted by the defocus prediction network and the predicted defocus image output by the defocus prediction network; the pixel values ​​of the pixels in the predicted defocus image are used to predict whether the pixels at the same position in the second training target image are within the depth of field corresponding to the image imager.

[0173] The first calculation subunit is used to calculate the image loss based on the predicted defocused image and the real defocused image corresponding to the second training target image; the pixel value of the pixel in the real defocused image is used to indicate whether the pixel at the same position in the second training target image is within the depth of field corresponding to the image imager.

[0174] The first acquisition subunit is used to acquire the relative depth features of the second training target image;

[0175] The second calculation subunit is used to calculate the feature loss based on the image features and the relative depth features;

[0176] The first training subunit is used to train the defocus prediction network based on the image loss and the feature loss;

[0177] The image features of the target object image extracted by the trained defocus prediction network include the relative depth features of the target object image.

[0178] In one optional implementation, the first acquisition subunit includes:

[0179] The second input subunit is used to input the second training target image into the trained relative depth estimation network to obtain the relative depth features of the second training target image extracted by the relative depth estimation network.

[0180] The device further includes: a relative depth estimation network training unit;

[0181] The relative depth estimation network training unit includes:

[0182] The second acquisition subunit is used to input the third training target image into the relative depth estimation network and acquire the predicted relative depth features of the third training target image output by the relative depth estimation network.

[0183] The third calculation subunit is used to calculate the depth loss based on the predicted relative depth features and the actual relative depth features of the third training target image;

[0184] The second training subunit is used to train the relative depth estimation network based on the depth loss.

[0185] In one optional implementation, the defocus prediction network is a multi-scale fully convolutional network, which includes multiple first upsampling layers and multiple first downsampling layers, and the image features are multi-scale image features, including the image features output by each of the first upsampling layers.

[0186] The relative depth estimation network is a multi-scale fully convolutional network, which includes multiple second upsampling layers and multiple second downsampling layers. The relative depth features are multi-scale relative depth features, including the image features output by each second upsampling layer.

[0187] Based on the fuzzy detection method provided in the above embodiments, this application also provides a fuzzy detection device, which will be described below with reference to the accompanying drawings. Since the principle by which the device in this disclosure solves the problem is similar to the fuzzy detection method described above in this application, the implementation of the device can refer to the implementation of the method, and repeated details will not be described again.

[0188] See Figure 6 , Figure 6 This is a schematic diagram of a fuzzy detection device provided in an embodiment of this application. Figure 6 As shown, the fuzzy detection device includes:

[0189] Acquisition unit 601 is used to acquire at least one image of a target object;

[0190] The input unit 602 is used to input the target image into the blur detection model to obtain the target blur detection result output by the blur detection model; the target blur detection result is used to indicate whether the target object in the target image is blurred;

[0191] The fuzzy detection model consists of a defocus prediction network and a classification network; the fuzzy detection model is trained according to the training method of the fuzzy detection model described above.

[0192] In one optional implementation, the acquisition unit 601 includes:

[0193] The first acquisition subunit is used to acquire the video frame to be detected;

[0194] The second acquisition subunit is used to input the video frame to be detected into the target object detection network and acquire the target object detection result of the video frame to be detected.

[0195] The cropping subunit is used to crop the video frame to be detected based on the target object detection result, and obtain at least one target object image in the video frame to be detected.

[0196] Additionally, embodiments of this application provide an electronic device, including:

[0197] One or more processors;

[0198] Storage device, on which one or more programs are stored,

[0199] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the fuzzy detection model as described in any of the above embodiments, or the fuzzy detection method as described in any of the above embodiments.

[0200] In addition, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the training method for the fuzzy detection model as described in any of the above embodiments, or the fuzzy detection method as described in any of the above embodiments.

[0201] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0202] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Regarding the methods disclosed in the embodiments, since they correspond to the systems disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the system section description.

[0203] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0204] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a fuzzy detection model, characterized in that, The fuzz detection model includes a defocus prediction network and a classification network, and the method includes: Obtain the trained defocus prediction network; the trained defocus prediction network is used to extract image features of the target object image input to the defocus prediction network; the image features include the relative depth features of the target object image; The defocus prediction network is a multi-scale fully convolutional network, which includes multiple first upsampling layers and multiple first downsampling layers. The image features are multi-scale image features, including the image features output by each of the first upsampling layers. The relative depth feature is used to represent the amount of distance closer to the shooting source and distance farther from the shooting source; Obtain the first training target image and the corresponding fuzzy label; The first training target image is input into the trained defocus prediction network to obtain the image features of the first training target image; the image features include the relative depth features of the first training target image; The image features are input into a classification network to obtain the target object blur detection result output by the classification network; Calculate the detection result loss based on the target object fuzzy detection result and the fuzzy label; The classification network is trained based on the loss from the detection results.

2. The method according to claim 1, characterized in that, The training process of the defocus prediction network includes: The second training target image is input into the defocus prediction network to obtain the image features of the second training target image extracted by the defocus prediction network and the predicted defocus image output by the defocus prediction network; the pixel values ​​of the pixels in the predicted defocus image are used to predict whether the pixels at the same position in the second training target image are within the depth of field corresponding to the image imager. Based on the predicted out-of-focus image and the real out-of-focus image corresponding to the second training target image, the image loss is calculated; the pixel value of the pixel in the real out-of-focus image is used to indicate whether the pixel at the same position in the second training target image is within the depth of field corresponding to the image imager. Obtain the relative depth features of the second training target image; Calculate the feature loss based on the image features and the relative depth features; The defocus prediction network is trained based on the image loss and the feature loss. The image features of the target object image extracted by the trained defocus prediction network include the relative depth features of the target object image.

3. The method according to claim 2, characterized in that, The step of obtaining the relative depth features of the second training target image includes: The second training target image is input into the trained relative depth estimation network to obtain the relative depth features of the second training target image extracted by the relative depth estimation network. The training process of the relative depth estimation network includes: The third training target image is input into the relative depth estimation network to obtain the predicted relative depth features of the third training target image output by the relative depth estimation network. The depth loss is calculated based on the predicted relative depth features and the actual relative depth features of the third training target image; The relative depth estimation network is trained based on the depth loss.

4. The method according to claim 3, characterized in that, The relative depth estimation network is a multi-scale fully convolutional network, which includes multiple second upsampling layers and multiple second downsampling layers. The relative depth features are multi-scale relative depth features, including the image features output by each second upsampling layer.

5. A fuzzy detection method, characterized in that, The method includes: Acquire at least one image of the target object; The target image is input into a blur detection model to obtain the target blur detection result output by the blur detection model; the target blur detection result is used to indicate whether the target object in the target image is blurred. The fuzzy detection model consists of a defocus prediction network and a classification network; the fuzzy detection model is trained using the training method for the fuzzy detection model according to any one of claims 1-4.

6. The method according to claim 5, characterized in that, The acquisition of at least one target image includes: Obtain the video frames to be detected; The video frame to be detected is input into the target object detection network to obtain the target object detection result of the video frame to be detected; Based on the target object detection results, the video frame to be detected is cropped to obtain at least one target object image in the video frame to be detected.

7. A training device for a fuzzy detection model, characterized in that, The blur detection model includes a defocus prediction network and a classification network, and the device includes: The first acquisition unit is used to acquire the trained defocus prediction network; the trained defocus prediction network is used to extract image features of the target image input to the defocus prediction network; the image features include the relative depth features of the target image; The defocus prediction network is a multi-scale fully convolutional network, which includes multiple first upsampling layers and multiple first downsampling layers. The image features are multi-scale image features, including the image features output by each of the first upsampling layers. The relative depth feature is used to represent the amount of distance closer to the shooting source and distance farther from the shooting source; The second acquisition unit is used to acquire the first training target image and the fuzzy label corresponding to the first training target image; The first input unit is used to input the first training target image into the trained defocus prediction network to obtain the image features of the first training target image; the image features include the relative depth features of the first training target image. The second input unit is used to input the image features into the classification network and obtain the target object blur detection result output by the classification network. The calculation unit is used to calculate the detection result loss based on the fuzzy detection result of the target object and the fuzzy label; An execution unit is used to train the classification network based on the detection results.

8. A fuzzy detection device, characterized in that, The device includes: Acquisition unit, used to acquire at least one image of a target object; The input unit is used to input the target image into the blur detection model to obtain the target blur detection result output by the blur detection model; the target blur detection result is used to indicate whether the target object in the target image is blurred. The fuzzy detection model consists of a defocus prediction network and a classification network; the fuzzy detection model is trained using the training method for the fuzzy detection model according to any one of claims 1-4.

9. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the fuzzy detection model as described in any one of claims 1-4, or the fuzzy detection method as described in any one of claims 5-6.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the training method for the fuzzy detection model as described in any one of claims 1-4, or the fuzzy detection method as described in any one of claims 5-6.

Images