A method, device, electronic device and storage medium for determining scene type

Abstract

The embodiment of the present invention discloses a method, device, electronic device and storage medium for judging the scene type; a depth estimation model obtained by training based on an unlabeled sample image and a depth estimation sample image corresponding to the sample image can be obtained, the depth estimation sample image is obtained by reconstructing the sample image after the depth estimation model extracts a sample depth feature map from the sample image, and the depth feature map of the image to be judged is extracted through the depth estimation model, the depth feature map includes the depth information of each pixel point in the image to be judged, based on the depth information of each pixel point in the depth feature map, the target depth information of the image to be judged is calculated, and based on the preset correspondence between the depth information and the scene type, and the target depth information, the target scene type corresponding to the image to be judged is determined; therefore, the method of the embodiment of the present invention can be based on artificial intelligence technology, use unlabeled sample images for training, reduce the dependence on manually labeled data, and save human resources.

Description

Technical Field

[0001] The present invention relates to the field of image technology, and in particular to a scene type determination method, device, electronic device and storage medium. Background Art

[0002] In recent years, with the widespread adoption of artificial intelligence (AI), automated cataloging of video and image material has attracted increasing attention. Shot type, as a label, is a crucial element in the cataloging process. Shot type recognition aims to identify a specific frame in an image or video as a close-up, medium shot, long shot, panoramic shot, or close-up.

[0003] Currently, the primary method for determining the scene type of a video or image is to manually label the images or video frames as close-up, medium shot, long shot, panoramic shot, and close-up, and then perform machine learning through classification tasks. However, this method relies on the quality of the labeled data. Because the boundaries in the standard specifications for scene types are difficult to define, especially the boundaries between close-up and medium shot, and between medium shot and long shot, the consistency of manual labeling is relatively low. Furthermore, different TV stations have different standards, requiring different labeled data for different scenes. If the standards change, the data needs to be re-labeled, which consumes a lot of manpower. Therefore, the classification task is not very effective and places a high demand on human resources. Summary of the Invention

[0004] The embodiments of the present invention provide a method, device, electronic device, and storage medium for determining scene type, which can reduce reliance on manually labeled data, save human resources, and improve the accuracy of determining scene type.

[0005] An embodiment of the present invention provides a method for determining a scene type, comprising:

[0006] Obtaining a depth estimation model, where the depth estimation model is trained based on an unlabeled sample image and a depth estimation sample image corresponding to the sample image, where the depth estimation model extracts a sample depth feature map from the sample image and reconstructs the sample image based on the sample depth feature map, where the sample depth feature map includes depth information of each pixel in the sample image;

[0007] Extracting a depth feature map of the image to be judged by using the depth estimation model, wherein the depth feature map includes depth information of each pixel in the image to be judged;

[0008] Calculating target depth information of the image to be judged based on the depth information of each pixel in the depth feature map;

[0009] Based on the preset correspondence between depth of field information and scene type, and the target depth of field information, the target scene type corresponding to the image to be determined is determined.

[0010] Accordingly, an embodiment of the present invention further provides a device for determining a scene type, comprising:

[0011] a model acquisition unit, configured to acquire a depth estimation model, the depth estimation model being trained based on an unlabeled sample image and a depth estimation sample image corresponding to the sample image, the depth estimation sample image being obtained by extracting a sample depth feature map from the sample image by the depth estimation model and then reconstructing the image based on the sample depth feature map, the sample depth feature map including depth information of each pixel in the sample image;

[0012] A feature extraction unit, configured to extract a depth feature map of the image to be judged using the depth estimation model, wherein the depth feature map includes depth information of each pixel in the image to be judged;

[0013] A depth of field estimation unit, configured to calculate target depth of field information of the image to be judged based on the depth of field information of each pixel point in the depth feature map;

[0014] The type determination unit is configured to determine a target scene type corresponding to the image to be determined based on a preset correspondence between depth of field information and scene type and the target depth of field information.

[0015] In an optional example, the image to be judged is a video frame in a video to be judged, and the video to be judged contains at least two video frames. The feature extraction module is used to use each video frame in the video to be judged as an image to be judged, and extract a depth feature map of each image to be judged through the depth of field estimation model;

[0016] After the type judgment unit, a video type judgment unit is also included, which is used to calculate the similarity between the images to be judged in the video to be judged;

[0017] Determining, based on the similarity, at least one group of video frames in the video to be judged, wherein the similarity between the video frames in each group of video frames is greater than a preset similarity threshold;

[0018] If, in each group of video frames, there are continuous video frames in the video to be determined, determining the continuous video frames as a group of target video frames;

[0019] Determining the scene type of each group of target video frames according to the target scene type corresponding to each of the to-be-judged images in each group of target video frames;

[0020] According to the scene type of each group of target video frames, a corresponding scene type label is added to each group of target video frames.

[0021] In an optional example, the video type determination unit may be configured to determine a preset number of video frames as central video frames of the cluster center points from the video frames of the video to be determined;

[0022] Obtaining similarities between the video frames in the video to be judged and each of the central video frames;

[0023] Classifying the video frames in the video to be judged into the cluster where the corresponding target center video frame is located, wherein the similarity between the video frame and the corresponding target center video frame is not less than a preset similarity threshold;

[0024] Based on the video frames in each of the clusters, a new central video frame is selected for each of the clusters, and the step of obtaining similarities between the video frames in the video to be determined and each of the central video frames is returned to execution until a clustering end condition is met;

[0025] The video frames in each of the clusters are respectively determined as a group of video frames.

[0026] In an optional example, the type determination unit further includes an area calculation unit, configured to determine a target object in the image to be determined and calculate a target area ratio of the target object in the image to be determined;

[0027] The determining the target scene type corresponding to the image to be determined based on the preset correspondence between the depth of field information and the scene type and the target depth of field information includes:

[0028] Based on the preset depth of field information and area ratio, the corresponding relationship with the scene type, and the target depth of field information and the target area ratio of the image to be determined, the target scene type corresponding to the image to be determined is determined.

[0029] In an optional example, the area calculation unit may be used to determine the photographed object in the image to be determined that belongs to a preset object type;

[0030] If it is determined that the number of the photographed object is one, the photographed object is determined as a target photographed object, and a target area ratio of the target photographed object in the image to be determined is calculated;

[0031] If it is determined that the number of the photographed objects is at least two, calculate the area ratio of each of the photographed objects in the image to be judged, determine the target photographed object from the photographed objects based on the area ratio, and determine the area ratio corresponding to the target photographed object as the target area ratio.

[0032] In an optional example, the model acquisition unit further includes a model training module for acquiring a depth estimation model to be trained and the sample image, wherein the sample image does not have a label indicating the scene type of the sample image, and the depth estimation model includes a feature extraction layer and an image reconstruction layer;

[0033] Performing feature extraction and depth analysis on the pixels in the sample image through the feature extraction layer to obtain a sample depth feature map corresponding to the sample image, wherein the sample depth feature map includes feature information and depth information of each pixel in the sample image;

[0034] Performing image reconstruction based on the feature information of each pixel in the sample depth feature map through the image reconstruction layer to obtain a reconstructed image corresponding to the sample image;

[0035] determining a loss of the depth estimation model according to the sample image and the reconstructed image;

[0036] Parameters of the depth estimation model are adjusted based on the loss to obtain a trained depth estimation model.

[0037] In an optional example, the feature extraction layer includes a depth of field analysis module and at least two sequentially connected feature extraction modules;

[0038] The model training unit may be configured to perform feature extraction on pixels in the sample image through the feature extraction module to obtain at least two intermediate feature maps of the sample image, wherein the intermediate feature maps include feature information of the pixels;

[0039] Determining, by the depth of field analysis module, the depth of field information of the pixel points based on the feature information of the pixel points in the intermediate feature map, and obtaining a sample depth feature map corresponding to the sample image based on the depth of field information and the intermediate feature map;

[0040] The image reconstruction layer is used to perform image reconstruction based on the feature information of the pixels in the intermediate feature map and the sample depth feature map to obtain a reconstructed image corresponding to the sample image.

[0041] In an optional example, the model training unit may be configured to perform a pose transformation on the photographed object in the sample image based on the pose transformation parameters to obtain a first pose estimation image corresponding to the sample image;

[0042] Performing pose restoration on the first pose estimation image based on the pose transformation parameters and the reconstructed image to obtain a restored image;

[0043] Calculating image loss based on an image difference between the sample image and the restored image;

[0044] Based on the image loss, a loss of the depth estimation model is determined.

[0045] In an optional example, the model training unit may be configured to perform edge detection on each photographic object in the sample image and the depth estimation sample image to obtain a first edge of each photographic object in the sample image and a second edge of each photographic object in the depth estimation sample image;

[0046] An edge loss of the depth of field estimation model is determined according to a difference between the first edge and the second edge corresponding to the same photographed object.

[0047] In an optional example, the feature extraction unit may be configured to perform feature extraction and depth analysis on pixel points in the sample image through a feature extraction layer of the depth estimation model to obtain a sample depth feature map corresponding to the sample image.

[0048] Accordingly, an embodiment of the present invention further provides an electronic device comprising a memory and a processor; the memory stores an application, and the processor is configured to run the application in the memory to execute the steps of any one of the scene type determination methods provided in the embodiments of the present invention.

[0049] In addition, an embodiment of the present invention further provides a storage medium storing a plurality of instructions suitable for loading by a processor to execute the steps of any one of the scene type determination methods provided in the embodiment of the present invention.

[0050] By adopting the scheme of the embodiment of the present invention, a depth of field estimation model can be obtained. The depth of field estimation model is trained based on an unlabeled sample image and a depth of field estimation sample image corresponding to the sample image. The depth of field estimation sample image is obtained by extracting a sample depth feature map from the sample image by the depth of field estimation model, and then reconstructing the image based on the sample depth feature map. The sample depth feature map includes the depth of field information of each pixel point in the sample image. Through the depth of field estimation model, a depth feature map of the image to be judged is extracted. The depth feature map includes the depth of field information of each pixel point in the image to be judged. Based on the depth of field information of each pixel point in the depth feature map , calculate the target depth of field information of the image to be judged, and determine the target scene type corresponding to the image to be judged based on the preset correspondence between the depth of field information and the scene type, as well as the target depth of field information; since the depth of field estimation model used in the embodiment of the present invention first extracts a sample depth feature map from an unlabeled sample image, performs image reconstruction based on the sample depth feature map to obtain a depth of field estimation sample image, and then trains based on the depth of field estimation sample image and the sample image, the method of the embodiment of the present invention can use unlabeled sample images for training, reduce dependence on manually labeled data, and save human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 Schematic diagram of a scene of a method for determining a scene type provided by an embodiment of the present invention;

[0053] Figure 2 is a flow chart of a method for determining scene type provided by an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of an image reconstruction process using a scene estimation model provided by an embodiment of the present invention;

[0055] Figure 4 is a flow chart of a method for determining scene type provided by an embodiment of the present invention;

[0056] Figure 5 2 is a schematic structural diagram of a device for determining scene type according to an embodiment of the present invention;

[0057] Figure 6 is another structural diagram of the scene type determination device provided by an embodiment of the present invention;

[0058] Figure 7It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. The solutions provided in the embodiments of this application involve technologies such as computer vision and machine learning of artificial intelligence. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the present invention.

[0060] In the embodiment of the present invention, the process of constructing the depth of field estimation model specifically involves machine learning (ML) in artificial intelligence. Machine learning is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0061] The embodiments of the present invention provide a method, device, electronic device and storage medium for determining a scene type. Specifically, the embodiments of the present invention provide a method for determining a scene type applicable to a scene type determining device, which can be integrated into an electronic device.

[0062] The electronic device can be a terminal or other device, including but not limited to a mobile terminal and a fixed terminal. For example, a mobile terminal includes but is not limited to a smart phone, a smart watch, a tablet computer, a laptop computer, a smart car terminal, etc., wherein a fixed terminal includes but is not limited to a desktop computer, a smart TV, etc.

[0063] The electronic device can also be a server or other device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, but is not limited to these.

[0064] The scene type determination method according to the embodiment of the present invention may be implemented by a server, or may be implemented by a terminal and a server together.

[0065] The following describes the method by taking the example of a terminal and a server jointly implementing the scene type determination method.

[0066] like Figure 1 As shown, the scene type judgment system provided by the embodiment of the present invention includes a terminal 10 and a server 20, etc.; the terminal 10 and the server 20 are connected through a network, for example, through a wired or wireless network connection, etc., wherein the terminal 10 can exist as a terminal that sends the image to be judged to the server 20.

[0067] The server 20 can be used to obtain a depth of field estimation model, extract a depth feature map of the image to be judged through the depth of field estimation model, calculate the target depth of field information of the image to be judged based on the depth of field information of each pixel point in the depth feature map, and determine the target scene type corresponding to the image to be judged based on the preset correspondence between the depth of field information and the scene type, as well as the target depth of field information.

[0068] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0069] The embodiment of the present invention will be described from the perspective of a shot type determination device, which can be integrated into a server or a terminal.

[0070] like Figure 2 As shown, the specific process of the scene type determination method of this embodiment can be as follows:

[0071] 201. Obtain a depth of field estimation model. The depth of field estimation model is trained based on an unlabeled sample image and a depth of field estimation sample image corresponding to the sample image. The depth of field estimation sample image is obtained by extracting a sample depth feature map from the sample image by the depth estimation model and reconstructing the image based on the sample depth feature map. The sample depth feature map includes depth information of each pixel point in the sample image.

[0072] Among them, the sample image can be an image without depth of field information and / or a label for indicating the scene type of the sample image. The content in the sample image can be people, plants, etc., and the scene of the image can also be indoors or outdoors. The embodiment of the present invention does not limit factors such as the content and scene of the sample image.

[0073] To enable the depth estimation model to estimate depth in images, computer vision (CV) technology was also applied during the construction process. Computer vision is the science of enabling machines to "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, detecting, and measuring objects. Further image processing is performed to transform the computer-generated images into images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. Common biometric recognition technologies include facial recognition and fingerprint recognition.

[0074] The depth estimation model is obtained through pre-training. Through the pre-training process, the parameters of the depth estimation model can be adjusted so that the depth estimation model can achieve better depth estimation performance. Specifically, before step 201, the following steps may also be included:

[0075] Obtaining a depth of field estimation model to be trained and a sample image, where the sample image does not have a label indicating the scene type of the sample image, and the depth of field estimation model includes a feature extraction layer and an image reconstruction layer;

[0076] Through the feature extraction layer, feature extraction and depth analysis are performed on the pixels in the sample image to obtain a sample depth feature map corresponding to the sample image. The sample depth feature map includes feature information and depth information of each pixel in the sample image;

[0077] Through the image reconstruction layer, image reconstruction is performed based on the feature information of each pixel in the sample depth feature map to obtain a reconstructed image corresponding to the sample image;

[0078] Determining the loss of the depth estimation model based on the sample image and the reconstructed image;

[0079] The parameters of the depth estimation model are adjusted based on the loss to obtain a trained depth estimation model.

[0080] The feature extraction layer may specifically include the number of feature extraction layers used to extract feature information in the depth estimation model, the number of input channels of the feature extraction layer, etc. For example, if the feature extraction layer of the depth estimation model is a convolutional layer, the feature extraction layer of the depth estimation model may include the number of convolutional layers, the size of the convolution kernel in the convolutional layer, and / or the number of input channels corresponding to each convolutional layer, etc.

[0081] It is understood that the depth estimation model can have only one feature extraction layer for feature extraction, or it can have multiple feature extraction layers to extract features from the sample image. When the depth estimation model has multiple feature extraction layers, as the number of extraction operations of the feature extraction layer increases, the sample depth feature map extracted by each feature extraction layer will contain less content information about the sample image and more feature information.

[0082] In one example, in order to make the depth estimation sample image obtained after pixel reconstruction closer to the sample image and reduce the error in determining the loss of the depth estimation model, the pixel reconstruction can be performed in combination with the depth feature maps corresponding to different feature extraction modules. Optionally, the feature extraction layer includes a depth analysis module and at least two sequentially connected feature extraction modules. The step of "performing feature extraction and depth analysis on the pixels in the sample image through the feature extraction layer to obtain a sample depth feature map corresponding to the sample image" may include:

[0083] Performing feature extraction on pixels in the sample image through a feature extraction module to obtain at least two intermediate feature maps of the sample image, where the intermediate feature maps include feature information of the pixels;

[0084] Through the depth of field analysis module, the depth of field information of the pixel point is determined based on the feature information of the pixel point in the intermediate feature map, and the sample depth feature map corresponding to the sample image is obtained based on the depth of field information and the intermediate feature map.

[0085] Accordingly, the step of "reconstructing the image based on the feature information of each pixel in the sample depth feature map through the image reconstruction layer to obtain a reconstructed image corresponding to the sample image" includes:

[0086] Through the image reconstruction layer, image reconstruction is performed based on the feature information of the pixels in the intermediate feature map and the sample depth feature map to obtain a reconstructed image corresponding to the sample image.

[0087] Among them, the image reconstruction operation can specifically be the inverse operation of feature extraction on the sample image. For example, if the feature extraction layer extracts features from the sample image by convolution, and then synthesizes the sample depth feature map based on the features, then the inverse operation can be to decompose the features in the sample depth feature map, and then perform deconvolution to restore the image; for example, if the feature extraction layer extracts features from the sample image by upsampling, and then synthesizes the sample depth feature map based on the features, then the inverse operation can be to decompose the features in the sample depth feature map, and then perform downsampling to restore the sample image to obtain the depth of field estimation sample image, and so on.

[0088] Among them, the image reconstruction process of the pixel points by combining the intermediate depth feature maps corresponding to different feature extraction modules can be as follows: Figure 3When performing image reconstruction, each image reconstruction layer will realize image reconstruction based on the intermediate feature layer of the corresponding feature extraction module and the intermediate reconstructed image output by the previous image reconstruction layer.

[0089] In actual applications, the sample image can be a single image taken by a monocular camera, or a single image in a series of photos or videos taken by a monocular camera or a binocular camera. When the sample image is taken by a monocular camera or a single image, it is necessary to perform a pose transformation on the subject in the sample image to predict the relative pose and then solve the model loss based on the relative pose. Optionally, the step of "determining the loss of the depth estimation model based on the sample image and the reconstructed image" may include:

[0090] Performing a pose transformation on the object in the sample image based on the pose transformation parameters to obtain a first pose estimation image corresponding to the sample image;

[0091] Based on the pose transformation parameters and the reconstructed image, the pose of the first pose estimation image is restored to obtain the restored image;

[0092] Calculate the image loss based on the image difference between the sample image and the restored image;

[0093] Based on the image loss, the loss of the depth estimation model is determined.

[0094] Among them, when the pose of the subject in the sample image is transformed, the pose of the subject in the sample image can be randomly transformed by a pose transformation algorithm or a pose transformation network, or a reference image of the subject in the sample image taken from another arbitrary angle can be obtained, and the pose of the subject in the sample image and the reference image can be estimated.

[0095] Specifically, the process of calculating the pose estimation error can be expressed as follows:

[0096]

[0097] Among them, L p represents the pose estimation error, I t Represents the sample image, I t′→t The calculation process is as follows:

[0098] I t′→t =I t′ <proj(D t ,T t→t′ ,K)>

[0099] Among them, D tRepresents the reconstructed image, and K represents the preset camera intrinsic parameter. For simplicity, we assume that the intrinsic parameter K calculated in advance for all sample images is the same. The camera intrinsic parameter may be different from the camera intrinsic parameter when the sample image is actually taken. The technician can set it according to the usage of the depth of field estimation model. The present invention does not limit this. t′ Represents the first pose estimation image, T t→t′ Represents the pose transformation parameters between the sample image and the first pose estimation image. For example, the pose transformation parameters can be a pose transformation matrix representing the pose change of the photographed object, etc. t′→t Can represent I t′ Based on the depth image D t and I t′ Relative to I t The pose transformation parameter T t→t′ Reconstructed I t , that is, the restored image.

[0100] Among them, pe is an error calculation function, and the specific calculation of pe can be implemented according to the structural similarity algorithm (SSIM) and / or Manhattan distance (L1 norm); proj() is in the 2D coordinate system, in I t′ Upper projection depth D t The result below, <> is a sampling operation.

[0101] In another embodiment, to improve the accuracy of the depth information obtained by the depth estimation model, edge loss may be calculated for the sample image and the subject in the depth estimation sample image, and the loss of the depth estimation model may be calculated based on the edge loss. Optionally, the step of "determining the loss of the depth estimation model based on the sample image and the reconstructed image" may include:

[0102] Performing edge detection on each photographic object in the sample image and the depth-of-field estimation sample image to obtain a first edge of each photographic object in the sample image and a second edge of each photographic object in the depth-of-field estimation sample image;

[0103] An edge loss of a depth estimation model is determined according to a difference between a first edge and a second edge corresponding to the same photographed object.

[0104] Specifically, the calculation process of edge loss can be expressed by the following formula:

[0105]

[0106] Among them, L s represents the edge loss.

[0107] The step of “performing edge detection on each photographed object in the sample image and the depth of field estimation sample image” may include:

[0108] Performing filtering processing on the sample image and the depth estimation sample image so as to make the sample image and the depth estimation sample image become smooth images;

[0109] For the smoothed sample image and depth estimation sample image, the finite difference of the first-order partial derivative is used to calculate the magnitude and direction of the gradient;

[0110] Perform non-maximum suppression on the gradient amplitude;

[0111] A threshold algorithm is used to detect and connect the edges of the sample image and the depth estimation sample image.

[0112] In another example, the edge detection process may also be implemented in combination with an edge detection algorithm, such as a Canny edge detection algorithm, a Marr-Hildreth algorithm, etc. The implementation method of edge detection is not limited and is not limited in the embodiment of the present invention.

[0113] Optionally, to improve the accuracy of the depth estimation model, the loss of the depth estimation model can be calculated by combining edge loss and image loss. For example, the loss calculation method of the depth estimation model can be expressed as follows:

[0114] L=μL p +λL s

[0115] Among them, μ and λ are the coefficients of image loss and edge loss respectively, and these coefficients can be continuously corrected during the model training process.

[0116] 202. Extract a depth feature map of the image to be judged through a depth estimation model, where the depth feature map includes depth information of each pixel in the image to be judged.

[0117] In one embodiment, the depth feature map may be an image generated by the depth estimation model based on the extracted feature information of the image to be judged after the feature extraction is performed on the image to be judged.

[0118] Among them, the features extracted by the depth of field estimation model for the image to be judged can specifically be local features of the image, such as contrast, detail clarity, shadows and highlights, etc. The local features of the image can reflect the local particularity of the image; they can also be global features of the image, such as line features, color features, texture features and structural features, etc.

[0119] Specifically, a convolutional network can be set in the depth estimation model to extract image features. The step of "extracting a depth feature map of the image to be judged through the depth estimation model" may include:

[0120] The convolutional network based on the depth estimation model extracts local features and global features of the image to be judged;

[0121] Feature synthesis is performed based on local features and global features to obtain a depth feature map of the image to be judged.

[0122] Among them, the convolutional network is a network structure that can extract the features of an image. For example, the convolutional network may include a convolutional layer, and the convolutional layer can extract the features of an image through a convolution operation.

[0123] In an optional example, a convolutional network may include multiple convolutional layers, each of which has at least one convolutional unit. Different convolutional units can extract different features. When performing feature extraction through the convolutional layer, the image to be judged is scanned through the convolutional unit, and different convolution kernels are used to learn different features to extract the original image features.

[0124] In another example, the depth estimation model includes a feature extraction layer, and step 202 may include: performing feature extraction and depth analysis on pixel points in the image to be judged through the feature extraction layer of the depth estimation model to obtain a sample depth feature map corresponding to the image to be judged.

[0125] The depth of field analysis process may be to convert the characteristic values ​​of the pixel points to obtain the relative depth of field information of the image to be judged.

[0126] It is understandable that if the depth estimation model is trained using sample images labeled with actual depth values, the target depth information output by the depth estimation model may be the actual depth value of the image.

[0127] In some examples, the step of "extracting a depth feature map of the image to be judged through a depth estimation model" can also be applying a scale-invariant feature transform (SIFT) feature extraction method, a histogram of oriented gradients (HOG) feature extraction method, etc. in the depth estimation model.

[0128] 203. Calculate target depth information of the image to be judged based on the depth information of each pixel in the depth feature map.

[0129] When calculating the target depth information, the depth information of each pixel in the image to be judged can be directly summed up.

[0130] In another example, according to the number of pixels in the image to be determined, the depth information of each pixel in the image to be determined may be directly added together, and then the average value may be calculated as the target depth information.

[0131] Optionally, different calculation weights can be set for the objects in the image to be judged, and then the depth of field information corresponding to different pixel points can be weightedly calculated. For example, the pixel weight of the target object in the image to be judged can be set to the highest, and the pixel weights of other objects can be adjusted according to the area ratio of other objects, and so on.

[0132] 204. Determine a target scene type corresponding to the image to be determined based on a preset correspondence between depth of field information and scene type, and target depth of field information.

[0133] Among them, the correspondence between depth of field information and scene type can be freely set by technicians according to actual application scenarios.

[0134] In actual applications, if it is necessary to judge the scene type of a certain video, the scene type of each video frame in the video can be judged first. At the same time, in order to make the result on the video more continuous and smooth, the scene type of the video can be determined based on the scene type of each video frame in the video. Specifically, the step of "extracting the depth feature map of the image to be judged through the depth of field estimation model" may include:

[0135] Each video frame in the video to be judged is used as an image to be judged, and a depth feature map of each image to be judged is extracted through a depth estimation model;

[0136] Accordingly, the embodiment of the present invention further includes:

[0137] Calculate the similarity between each image to be judged in the video to be judged;

[0138] Determine, based on the similarity, at least one group of video frames in the video to be judged, wherein the similarity between each video frame in each group of video frames is greater than a preset similarity threshold;

[0139] If, in each group of video frames, there are continuous video frames in the video to be judged, then the continuous video frames are determined to be a group of target video frames;

[0140] Determine the scene type of each group of target video frames according to the target scene type corresponding to each to-be-judged image in each group of target video frames;

[0141] According to the scene type of each group of target video frames, a corresponding scene type label is added to each group of target video frames.

[0142] Wherein, each video frame in the video to be judged may have time information, such as a timestamp, etc. When determining consecutive video frames, it is possible to directly determine whether two adjacent video frames are consecutive based on the timestamps of the video frames themselves;

[0143] Alternatively, each video frame in the video to be determined may have a corresponding sequential number. When determining consecutive video frames, the numbers of each video frame may be used. If the numbers of two adjacent video frames are consecutive, the video frames are considered to be consecutive.

[0144] Alternatively, the judgment may be made directly based on the similarity between video frames. If the similarity between adjacent video frames is greater than a judgment threshold, such as greater than 99%, the two video frames may be considered to be adjacent.

[0145] In one example, when determining at least one group of video frames in a video to be judged, two adjacent video frames in the video to be judged can be compared in sequence. If the similarity between the two video frames is greater than a preset similarity threshold, the two video frames are classified as the same group of video frames. If the time similarity between the two video frames is not greater than the preset similarity threshold, the two video frames are respectively classified as video frames in different groups.

[0146] For example, if there are five video frames A, B, C, D, and E in the video to be judged, and the similarity between A and B is greater than a preset similarity threshold, A and B are grouped together, and the similarity between B and C is then calculated. If the similarity between B and C is greater than the preset similarity threshold, C is also grouped together. If the similarity between C and D is not greater than the preset similarity threshold, D is grouped together. If the similarity between D and E is not greater than the preset similarity threshold, E is grouped together. After the grouping is completed, the similarity of the video frames in groups one, two, and three is recalculated until the grouping of the video frames no longer changes.

[0147] In another example, in order to improve computing efficiency and save computing resources, the video frames in the video to be judged may be grouped by clustering. Specifically, the step of "determining at least one group of video frames in the video to be judged based on similarity" may include:

[0148] Determining a preset number of video frames as central video frames of cluster centers from each video frame of the video to be judged;

[0149] Obtaining the similarity between the video frame in the video to be judged and each central video frame;

[0150] Divide the video frames in the video to be judged into the cluster where the corresponding target center video frame is located, wherein the similarity between the video frame and the corresponding target center video frame is not less than a preset similarity threshold;

[0151] Based on the video frames in each cluster, a new central video frame is selected for each cluster, and the step of obtaining similarities between the video frames in the video to be judged and each central video frame is returned to execute until the clustering end condition is met;

[0152] The video frames in each cluster are respectively determined as a group of video frames.

[0153] The cluster is a set of data objects generated based on a clustering process. In an embodiment of the present invention, the cluster is a set of video frames generated based on a clustering process.

[0154] The central video frame is the cluster center determined when clustering video frames in the video to be judged. The number of central video frames can be a fixed number that is preset. The central video frame selected during the first clustering is generally randomly selected. Each time a new central video frame is selected within a cluster, another video frame can be selected within each cluster as the new central video frame.

[0155] In another example, the clustering process may be implemented in the form of a clustering model. The clustering model may cluster the videos to be judged by adjusting the number of central video frames multiple times to determine the most accurate number of central video frames.

[0156] Optionally, the clustering end condition may be that the video frames in each cluster no longer change, or that the central video frame corresponding to each cluster no longer changes, or that the number of times the clustering process returns to execute the step of obtaining the similarity between the video frames in the video to be judged and the central video frames, and so on.

[0157] In practical applications, if the sample images contain a single scene or content, the depth estimation model may not be able to learn other types of scenes or content well. If images containing other scenes or content are presented, the depth estimation model may make incorrect predictions about these images. To address this issue, the sample images can be expanded to include all possible scenes and content. However, this solution significantly increases the number of sample images required, and the time and computing resources required to train the depth estimation model will also increase significantly.

[0158] In one example, the area ratio of the target object in the image to be judged can be calculated, and the area ratio and the target depth of field information can be combined to determine the scene type of the image to be judged. With this solution, no additional sample images are required to train the depth of field estimation model, and the accuracy of depth of field type determination can be further improved. Figure 3 As shown, before step 204, the following steps may also be included:

[0159] Determine the target object in the image to be judged, and calculate the target area ratio of the target object in the image to be judged.

[0160] The target subject is the primary subject among all subjects in the image to be determined. For example, the target subject may be the subject located at the center of the image, or the subject located at the focal point of the image, or the subject with the largest area in the image to be determined, etc.

[0161] Accordingly, after determining the target area ratio of the target photographed object, the step of "determining the target scene type corresponding to the image to be determined based on the preset correspondence between depth of field information and scene type, and the target depth of field information" may include:

[0162] Based on the preset depth of field information and area ratio, the corresponding relationship with the scene type, and the target depth of field information and target area ratio of the image to be judged, the target scene type corresponding to the image to be judged is determined.

[0163] The correspondence between depth of field information, area ratio, and shot type can be customized by the developer. Developers can adjust the correspondence between depth of field information, area ratio, and shot type based on different shot type classification rules. This customizable correspondence ensures that the depth of field estimation model is applicable to different shot type classification rules.

[0164] The step of “determining the target object in the image to be determined and calculating the target area ratio of the target object in the image to be determined” may specifically include:

[0165] Determining a photographed object in the image to be determined to be of a preset object type;

[0166] If it is determined that the number of the photographed object is one, the photographed object is determined as the target photographed object, and the target area ratio of the target photographed object in the image to be judged is calculated;

[0167] If it is determined that the number of photographed objects is at least two, calculate the area ratio of each photographed object in the image to be judged, determine the target photographed object from the photographed objects based on the area ratio, and determine the area ratio corresponding to the target photographed object as the target area ratio.

[0168] Among them, the preset object type can be pre-set by the user based on factors such as the shooting purpose or shooting scene of the image to be judged or the video to be judged. For example, if the video to be judged is an interview with a news figure, the preset object type can be set to a person; if the video to be judged is a nature documentary, the preset object type can be set to a plant, and so on.

[0169] In another example, if there is no photographed object belonging to a preset object type in the image to be determined, the target area ratio may be determined to be zero.

[0170] Among them, different recognition models can be used to determine whether the photographed object in the image to be judged belongs to a preset object type. For example, a face recognition model can be used to identify the face in the image to be judged. Specifically, the face model can be a ret i naface face detector, an MT-CNN face detection network, etc.

[0171] As can be seen from the above, the embodiment of the present invention uses unlabeled sample images for training, which reduces the dependence on manually labeled data and saves human resources.

[0172] According to the method described in the previous embodiment, the following will be further described in detail by taking the image to be determined as a video frame in the video to be determined as an example.

[0173] In this embodiment, the Figure 1 system is described.

[0174] like Figure 4 As shown, the scene type determination method of this embodiment may have the following specific process:

[0175] 401. The server pre-trains the depth estimation model to be trained based on the sample image.

[0176] The pre-training process may specifically include:

[0177] Obtaining a depth estimation model to be trained and a sample image, where the sample image does not have a label indicating the scene type of the sample image, and the depth estimation model includes a feature extraction layer and an image reconstruction layer;

[0178] Through the feature extraction layer, feature extraction and depth analysis are performed on the pixels in the sample image to obtain a sample depth feature map corresponding to the sample image. The sample depth feature map includes feature information and depth information of each pixel in the sample image;

[0179] Through the image reconstruction layer, image reconstruction is performed based on the feature information of each pixel in the sample depth feature map to obtain a reconstructed image corresponding to the sample image;

[0180] Determining the loss of the depth estimation model based on the sample image and the reconstructed image;

[0181] The parameters of the depth estimation model are adjusted based on the loss to obtain a trained depth estimation model.

[0182] 402. The server obtains the video to be judged sent by the terminal and obtains the depth estimation model

[0183] Among them, after generating the video to be judged, the terminal can send the video to be judged to the server via wired means, or the terminal can send the video to be judged to the cloud server wirelessly. When the server needs to perform depth of field type judgment, it can obtain the video to be judged from the cloud server.

[0184] 403. The server extracts a depth feature map of the image to be judged in the video to be judged by using a depth estimation model. The depth feature map includes depth information of each pixel in the image to be judged.

[0185] The depth feature map may be an image generated based on feature information of the image to be judged after the depth of field estimation model performs feature extraction on the image to be judged.

[0186] It is understood that the depth information in the depth feature map is converted based on the feature values ​​of the pixels to obtain the relative depth information of the image to be judged. If the depth estimation model is trained using sample images labeled with actual depth values, the target depth information output by the depth estimation model can be the actual depth value of the image.

[0187] 404. The server calculates target depth information of the image to be judged based on the depth information of each pixel in the depth feature map.

[0188] When calculating the target depth information, the depth information of each pixel in the image to be judged can be directly summed up.

[0189] In another example, according to the number of pixels in the image to be determined, the depth information of each pixel in the image to be determined may be directly added together, and then the average value may be calculated as the target depth information.

[0190] Optionally, different calculation weights can be set for the objects in the image to be judged, and then the depth of field information corresponding to different pixel points can be weightedly calculated. For example, the pixel weight of the target object in the image to be judged can be set to the highest, and the pixel weights of other objects can be adjusted according to the area ratio of other objects, and so on.

[0191] The specific calculation method of the depth of field information is not limited, and technicians can set the calculation method of the target depth of field information according to actual application conditions.

[0192] 405. The server determines the target subject in the image to be determined, and calculates the target area ratio of the target subject in the image to be determined.

[0193] In actual applications, if the scene or content of the sample image is relatively simple, the depth estimation model may not be able to learn other types of scenes or content well. If images containing other scenes or content appear, the depth estimation model may make incorrect predictions for these images.

[0194] In one example, the image's scene type can be determined by calculating the area ratio of the target subject in the image to be judged and combining this area ratio with the target's depth of field information. This solution eliminates the need for additional sample images to train the depth estimation model and further improves the accuracy of depth of field type determination.

[0195] The step of “determining the target object in the image to be determined and calculating the target area ratio of the target object in the image to be determined” may specifically include:

[0196] Determining a photographed object in the image to be determined to be of a preset object type;

[0197] If it is determined that the number of the photographed object is one, the photographed object is determined as the target photographed object, and the target area ratio of the target photographed object in the image to be judged is calculated;

[0198] If it is determined that the number of photographed objects is at least two, calculate the area ratio of each photographed object in the image to be judged, determine the target photographed object from the photographed objects based on the area ratio, and determine the area ratio corresponding to the target photographed object as the target area ratio.

[0199] 406. The server determines the target scene type corresponding to the image to be determined based on the preset depth of field information and area ratio, the correspondence between the scene type and the scene type, and the target depth of field information and target area ratio of the image to be determined.

[0200] For example, the preset depth of field information and area ratio correspond to the shot type as follows: when the depth of field information is greater than 0.09 and the area ratio is less than 0.1, it is a medium shot; when the depth of field information is greater than 0.1 and less than 0.5 and the area ratio is greater than 0.3 and less than 0.5, it is a close shot, and so on.

[0201] In an example, if the target depth information and target area ratio of the image to be determined are 0.3 and 0.4 respectively, the target scene type of the image to be determined is close-up.

[0202] 407. The server calculates the similarity between the images to be judged in the video to be judged, and determines, based on the similarity, at least one group of video frames in the video to be judged, wherein the similarity between the video frames in each group of video frames is greater than a preset similarity threshold.

[0203] Optionally, the step of “determining at least one group of video frames in the video to be judged based on the similarity” may include: determining a preset number of video frames as central video frames of the cluster centers from the video frames of the video to be judged;

[0204] Obtaining the similarity between the video frame in the video to be judged and each central video frame;

[0205] Divide the video frames in the video to be judged into the cluster where the corresponding target center video frame is located, wherein the similarity between the video frame and the corresponding target center video frame is not less than a preset similarity threshold;

[0206] Based on the video frames in each cluster, a new central video frame is selected for each cluster, and the step of obtaining similarities between the video frames in the video to be judged and each central video frame is returned to execute until the clustering end condition is met;

[0207] The video frames in each cluster are respectively determined as a group of video frames.

[0208] 408. The server determines the scene type of each group of target video frames according to the target scene type corresponding to each to-be-judged image in each group of target video frames.

[0209] In one example, the target scene types corresponding to the images to be determined in each group of target video frames may be counted, and the scene type with the most corresponding images to be determined is used as the scene type of the group of target video frames.

[0210] For example, in a group of target video frames, there are 3 images to be judged whose corresponding target scene type is medium shot, 7 images to be judged whose corresponding target scene type is close shot, and 4 images to be judged whose corresponding target scene type is long shot. Then the scene type of this group of target video frames is long shot.

[0211] 409. The server adds a corresponding scene type tag to each group of target video frames according to the scene type of each group of target video frames.

[0212] For example, if the scene type of the target video frame is a distant view, a distant view scene type tag is set for the entire target video frame. When a distant view video frame is needed, the entire target video frame can be directly obtained to ensure the continuity of the obtained target video frames.

[0213] In another example, in order to make the results of each frame more robust, after determining the scene type of each group of target video frames, if the scene type corresponding to the image to be judged in the target video frame is different from the scene type of the target video frame, the scene type corresponding to the image to be judged is modified to the scene type of the target video frame.

[0214] As can be seen from the above, the embodiment of the present invention uses unlabeled sample images for training, which reduces the dependence on manually labeled data and saves human resources.

[0215] In order to better implement the above method, an embodiment of the present invention also provides a scene type determination device.

[0216] refer to Figure 5 , the device comprises:

[0217] A model acquisition unit 501 is configured to acquire a depth estimation model. The depth estimation model is trained based on an unlabeled sample image and a depth estimation sample image corresponding to the sample image. The depth estimation sample image is obtained by extracting a sample depth feature map from the sample image and then reconstructing the image based on the sample depth feature map. The sample depth feature map includes depth information of each pixel in the sample image.

[0218] The feature extraction unit 502 is used to extract a depth feature map of the image to be judged using a depth estimation model, where the depth feature map includes depth information of each pixel in the image to be judged;

[0219] A depth estimation unit 503 is configured to calculate target depth information of the image to be judged based on the depth information of each pixel in the depth feature map;

[0220] The type determination unit 504 is configured to determine the target scene type corresponding to the image to be determined based on the preset correspondence between depth of field information and scene type and target depth of field information.

[0221] In an optional example, the image to be judged is a video frame in the video to be judged, and the video to be judged contains at least two video frames. The feature extraction module is used to use each video frame in the video to be judged as the image to be judged, and extract a depth feature map of each image to be judged through a depth of field estimation model;

[0222] like Figure 6 As shown, after the type judgment unit 504, a video type judgment unit 505 is also included, which is used to calculate the similarity between each image to be judged in the video to be judged;

[0223] Determine, based on the similarity, at least one group of video frames in the video to be judged, wherein the similarity between each video frame in each group of video frames is greater than a preset similarity threshold;

[0224] If, in each group of video frames, there are continuous video frames in the video to be judged, then the continuous video frames are determined to be a group of target video frames;

[0225] Determine the scene type of each group of target video frames according to the target scene type corresponding to each to-be-judged image in each group of target video frames;

[0226] According to the scene type of each group of target video frames, a corresponding scene type label is added to each group of target video frames.

[0227] In an optional example, the video type determination unit 505 may be configured to determine a preset number of video frames as central video frames of the cluster centers from the video frames of the video to be determined;

[0228] Obtaining the similarity between the video frame in the video to be judged and each central video frame;

[0229] Divide the video frames in the video to be judged into the cluster where the corresponding target center video frame is located, wherein the similarity between the video frame and the corresponding target center video frame is not less than a preset similarity threshold;

[0230] Based on the video frames in each cluster, a new central video frame is selected for each cluster, and the step of obtaining similarities between the video frames in the video to be judged and each central video frame is returned to execute until the clustering end condition is met;

[0231] The video frames in each cluster are respectively determined as a group of video frames.

[0232] In an optional example, before the type determination unit 504, an area calculation unit 506 is further included, which is used to determine the target object in the image to be determined and calculate the target area ratio of the target object in the image to be determined;

[0233] Based on the preset correspondence between depth of field information and scene type, and the target depth of field information, determining the target scene type corresponding to the image to be judged includes:

[0234] Based on the preset depth of field information and area ratio, the corresponding relationship with the scene type, and the target depth of field information and target area ratio of the image to be judged, the target scene type corresponding to the image to be judged is determined.

[0235] In an optional example, the area calculation unit 506 may be used to determine the photographed object in the image to be determined that belongs to a preset object type;

[0236] If it is determined that the number of the photographed object is one, the photographed object is determined as the target photographed object, and the target area ratio of the target photographed object in the image to be judged is calculated;

[0237] If it is determined that the number of photographed objects is at least two, calculate the area ratio of each photographed object in the image to be judged, determine the target photographed object from the photographed objects based on the area ratio, and determine the area ratio corresponding to the target photographed object as the target area ratio.

[0238] In an optional example, before the model acquisition unit 501, a model training unit 507 is further included, which is used to obtain a depth estimation model to be trained and a sample image, where the sample image does not have a label for indicating the scene type of the sample image, and the depth estimation model includes a feature extraction layer and an image reconstruction layer;

[0239] Through the feature extraction layer, feature extraction and depth analysis are performed on the pixels in the sample image to obtain a sample depth feature map corresponding to the sample image. The sample depth feature map includes feature information and depth information of each pixel in the sample image;

[0240] Through the image reconstruction layer, image reconstruction is performed based on the feature information of each pixel in the sample depth feature map to obtain a reconstructed image corresponding to the sample image;

[0241] Determining the loss of the depth estimation model based on the sample image and the reconstructed image;

[0242] The parameters of the depth estimation model are adjusted based on the loss to obtain a trained depth estimation model.

[0243] In an optional example, the feature extraction layer includes a depth analysis module and at least two sequentially connected feature extraction modules;

[0244] The model training unit 507 may be configured to perform feature extraction on pixels in the sample image through a feature extraction module to obtain at least two intermediate feature maps of the sample image, where the intermediate feature maps include feature information of the pixels;

[0245] Determine the depth of field information of the pixel points based on the feature information of the pixel points in the intermediate feature map through the depth of field analysis module, and obtain a sample depth feature map corresponding to the sample image based on the depth of field information and the intermediate feature map;

[0246] Through the image reconstruction layer, image reconstruction is performed based on the feature information of the pixels in the intermediate feature map and the sample depth feature map to obtain the reconstructed image corresponding to the sample image.

[0247] In an optional example, the model training unit 507 may be configured to perform a pose transformation on the photographed object in the sample image based on the pose transformation parameters to obtain a first pose estimation image corresponding to the sample image;

[0248] Based on the pose transformation parameters and the reconstructed image, the pose of the first pose estimation image is restored to obtain the restored image;

[0249] Calculate the image loss based on the image difference between the sample image and the restored image;

[0250] Based on the image loss, the loss of the depth estimation model is determined.

[0251] In an optional example, the model training unit 507 may be configured to perform edge detection on each object in the sample image and the depth estimation sample image to obtain a first edge of each object in the sample image and a second edge of each object in the depth estimation sample image.

[0252] An edge loss of a depth estimation model is determined according to a difference between a first edge and a second edge corresponding to the same photographed object.

[0253] In an optional example, the feature extraction unit 502 may be configured to perform feature extraction and depth analysis on pixel points in the sample image through a feature extraction layer of a depth estimation model to obtain a sample depth feature map corresponding to the sample image.

[0254] From the above, it can be seen that through the scene type judgment device, unlabeled sample images can be used for training, reducing the dependence on manually labeled data and saving human resources.

[0255] In addition, an embodiment of the present invention further provides an electronic device, which may be a terminal or a server, etc. Figure 7 , which shows a schematic structural diagram of an electronic device involved in an embodiment of the present invention, specifically:

[0256] The electronic device may include components such as a radio frequency (RF) circuit 701, a memory 702 including one or more computer-readable storage media, an input unit 703, a display unit 704, a sensor 705, an audio circuit 706, a wireless fidelity (WiFi) module 707, a processor 708 including one or more processing cores, and a power supply 709. It will be understood by those skilled in the art that Figure 7 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0257] The RF circuit 701 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is handed over to one or more processors 708 for processing; in addition, uplink data is sent to the base station. Generally, the RF circuit 701 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 701 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0258] The memory 702 can be used to store software programs and modules. The processor 708 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, a phone book, etc.). In addition, the memory 702 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 708 and the input unit 703 with access to the memory 702.

[0259] The input unit 703 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, in one embodiment, the input unit 703 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can detect user touch operations on or near it (for example, operations performed by a user using a finger, stylus, or any other suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connected devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch direction and detects signals generated by the touch operation, transmitting the signals to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 708. It can also receive and execute commands from the processor 708. In addition, touch-sensitive surfaces can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface, the input unit 703 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0260] The display unit 704 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 704 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 708 to determine the type of touch event. The processor 708 then provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 7 In the embodiment, the touch-sensitive surface and the display panel are used as two independent components to realize input and output functions, but in some embodiments, the touch-sensitive surface and the display panel can be integrated to realize input and output functions.

[0261] The terminal may also include at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the terminal, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0262] Audio circuit 706, a speaker, and a microphone provide an audio interface between the user and the terminal. Audio circuit 706 converts received audio data into electrical signals and transmits them to the speaker, which then converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 706 and converted into audio data. The audio data is then processed by processor 708 and transmitted via RF circuit 701 to, for example, another terminal. Alternatively, the audio data is output to memory 702 for further processing. Audio circuit 706 may also include an earphone jack to allow communication between an external headset and the terminal.

[0263] WiFi is a short-range wireless transmission technology. The terminal can help users send and receive emails, browse web pages and access streaming media through the WiFi module 707. It provides users with wireless broadband Internet access. Figure 7 A WiFi module 707 is shown, but it is understandable that it is not an essential component of the terminal and can be omitted as needed without changing the essence of the invention.

[0264] Processor 708 is the terminal's control center, connecting all components of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 702 and accessing data stored in memory 702, it executes various terminal functions and processes data, thereby performing overall phone testing. Optionally, processor 708 may include one or more processing cores; preferably, processor 708 may integrate an application processor and a modem processor, with the application processor primarily handling the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 708.

[0265] The terminal also includes a power supply 709 (e.g., a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 708 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 709 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0266] Although not shown, the terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 708 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 702 according to the following instructions, and the processor 708 will run the applications stored in the memory 702 to implement various functions as follows:

[0267] Obtain a depth estimation model, where the depth estimation model is trained based on an unlabeled sample image and a depth estimation sample image corresponding to the sample image. The depth estimation sample image is obtained by extracting a sample depth feature map from the sample image and then reconstructing the image based on the sample depth feature map. The sample depth feature map includes depth information of each pixel in the sample image.

[0268] Extracting a depth feature map of the image to be judged through a depth estimation model. The depth feature map includes the depth information of each pixel in the image to be judged.

[0269] Based on the depth information of each pixel in the depth feature map, the target depth information of the image to be judged is calculated;

[0270] Based on the preset correspondence between depth of field information and scene type, and target depth of field information, the target scene type corresponding to the image to be determined is determined.

[0271] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0272] To this end, an embodiment of the present invention provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the scene type determination methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0273] Obtain a depth estimation model, where the depth estimation model is trained based on an unlabeled sample image and a depth estimation sample image corresponding to the sample image. The depth estimation sample image is obtained by extracting a sample depth feature map from the sample image and then reconstructing the image based on the sample depth feature map. The sample depth feature map includes depth information of each pixel in the sample image.

[0274] Through the depth estimation model, a depth feature map of the image to be judged is extracted, and the depth feature map includes the depth information of each pixel in the image to be judged;

[0275] Based on the depth information of each pixel in the depth feature map, the target depth information of the image to be judged is calculated;

[0276] Based on the preset correspondence between depth of field information and scene type, and target depth of field information, the target scene type corresponding to the image to be determined is determined.

[0277] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0278] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0279] Since the instructions stored in the storage medium can execute the steps in any one of the scene type determination methods provided in the embodiments of the present invention, the beneficial effects that can be achieved by any one of the scene type determination methods provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0280] According to one aspect of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the above embodiments.

[0281] The above is a detailed introduction to a scene type determination method, device, electronic device and storage medium provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for determining scene type, characterized in that: include: Obtain a depth estimation model; Extracting a depth feature map of the image to be judged by using the depth estimation model, wherein the depth feature map includes depth information of each pixel in the image to be judged; Based on the depth information of each pixel in the depth feature map, the target depth information of the image to be judged is calculated; Based on the preset correspondence between the depth of field information and the scene type, and the target depth of field information, determining the target scene type corresponding to the image to be determined; The training process of the depth estimation model includes: Acquire a depth estimation model to be trained and an unlabeled sample image, wherein the sample image does not have a label for indicating the type of the sample image, and the depth estimation model includes a feature extraction layer and an image reconstruction layer; Through the feature extraction layer, feature extraction and depth analysis are performed on the pixel points in the sample image to obtain a sample depth feature map corresponding to the sample image, wherein the sample depth feature map includes feature information and depth information of each pixel point in the sample image; Through the image reconstruction layer, image reconstruction is performed based on the feature information of each pixel point in the sample depth feature map to obtain a reconstructed image corresponding to the sample image; Determining a loss of the depth estimation model according to the sample image and the reconstructed image; Parameters of the depth estimation model are adjusted based on the loss to obtain a trained depth estimation model.

2. The scene type determination method according to claim 1, characterized in that: The image to be judged is a video frame in the video to be judged, and the video to be judged contains at least two of the video frames; The step of extracting a depth feature map of the image to be determined by using the depth estimation model includes: Taking each video frame in the video to be judged as an image to be judged, and extracting a depth feature map of each image to be judged by using the depth of field estimation model; After determining the target scene type corresponding to the image to be determined based on the preset correspondence between the depth of field information and the scene type, and the target depth of field information, the method further includes: Calculating the similarity between the images to be judged in the video to be judged; Determine, according to the similarity, at least one group of video frames in the video to be judged, wherein in each group of video frames, the similarity between the video frames is greater than a preset similarity threshold; If, in each group of video frames, there are continuous video frames in the video to be determined, determining the continuous video frames as a group of target video frames; Determine the scene type of each group of target video frames according to the target scene type corresponding to each of the to-be-determined images in each group of target video frames; According to the scene type of each group of the target video frames, a corresponding scene type label is added to each group of the target video frames.

3. The scene type determination method according to claim 2, characterized in that: The step of determining at least one group of video frames in the video to be judged according to the similarity comprises: Determining a preset number of video frames as central video frames of cluster center points from each video frame of the video to be determined; Obtaining the similarity between the video frame in the video to be judged and each of the central video frames; Dividing the video frames in the video to be judged into the cluster where the corresponding target center video frame is located, wherein the similarity between the video frame and the corresponding target center video frame is not less than a preset similarity threshold; Based on the video frames in each of the clusters, a new central video frame of each of the clusters is selected, and the step of obtaining the similarity between the video frames in the video to be determined and each of the central video frames is returned to execute until a clustering end condition is met; The video frames in each of the clusters are respectively determined as a group of video frames.

4. The scene type determination method according to claim 1, characterized in that: Before determining the target scene type corresponding to the image to be determined based on the preset correspondence between the depth of field information and the scene type and the target depth of field information, the method further includes: Determine the target photographed object in the image to be determined, and calculate the target area ratio of the target photographed object in the image to be determined; The determining the target scene type corresponding to the image to be determined based on the preset correspondence between the depth of field information and the scene type, and the target depth of field information, comprises: Based on the preset depth of field information and area ratio, the corresponding relationship with the scene type, and the target depth of field information and the target area ratio of the image to be determined, the target scene type corresponding to the image to be determined is determined.

5. The scene type determination method according to claim 4, characterized in that: The step of determining the target photographed object in the image to be determined and calculating the target area ratio of the target photographed object in the image to be determined includes: Determining a photographed object in the image to be determined that belongs to a preset object type; If it is determined that the number of the photographed object is one, the photographed object is determined as a target photographed object, and a target area ratio of the target photographed object in the image to be determined is calculated; If it is determined that the number of the photographed objects is at least two, calculate the area ratio of each of the photographed objects in the image to be judged, determine the target photographed object from the photographed objects based on the area ratio, and determine the area ratio corresponding to the target photographed object as the target area ratio.

6. The scene type determination method according to claim 1, characterized in that: The feature extraction layer includes a depth analysis module and at least two sequentially connected feature extraction modules; The feature extraction and depth analysis of the pixels in the sample image are performed through the feature extraction layer to obtain a sample depth feature map corresponding to the sample image, including: By means of the feature extraction module, feature extraction is performed on the pixel points in the sample image to obtain at least two intermediate feature maps of the sample image, wherein the intermediate feature maps include feature information of the pixel points; Determine the depth information of the pixel points based on the feature information of the pixel points in the intermediate feature map through the depth analysis module, and obtain a sample depth feature map corresponding to the sample image based on the depth information and the intermediate feature map; The image reconstruction layer is used to reconstruct the image based on the feature information of each pixel in the sample depth feature map to obtain a reconstructed image corresponding to the sample image, including: The image reconstruction layer is used to perform image reconstruction based on the feature information of the pixels in the intermediate feature map and the sample depth feature map to obtain a reconstructed image corresponding to the sample image.

7. The scene type determination method according to claim 1, characterized in that: The step of determining the loss of the depth estimation model according to the sample image and the reconstructed image comprises: Performing a posture transformation on the photographed object in the sample image based on the posture transformation parameters to obtain a first posture estimation image corresponding to the sample image; Based on the posture transformation parameters and the reconstructed image, performing posture restoration on the first posture estimation image to obtain a restored image; Calculating image loss based on an image difference between the sample image and the restored image; Based on the image loss, a loss of the depth estimation model is determined.

8. The scene type determination method according to claim 1, characterized in that: The step of determining the loss of the depth estimation model according to the sample image and the reconstructed image comprises: Performing edge detection on each photographed object in the sample image and the reconstructed image to obtain a first edge of each photographed object in the sample image and a second edge of each photographed object in the reconstructed image; An edge loss of the depth of field estimation model is determined according to a difference between the first edge and the second edge corresponding to the same photographed object.

9. The scene type determination method according to claim 1, characterized in that: The step of extracting a depth feature map of the image to be determined by using the depth estimation model includes: The feature extraction layer of the depth estimation model performs feature extraction and depth analysis on the pixel points in the image to be judged, so as to obtain a sample depth feature map corresponding to the image to be judged.

10. A device for determining scene type, characterized in that: include: A model acquisition unit is used to acquire a depth of field estimation model, wherein the training process of the depth of field estimation model includes: acquiring a depth of field estimation model to be trained and an unlabeled sample image, wherein the sample image does not have a label for indicating the type of scene of the sample image, and the depth of field estimation model includes a feature extraction layer and an image reconstruction layer; through the feature extraction layer, performing feature extraction and depth of field analysis on pixel points in the sample image to obtain a sample depth feature map corresponding to the sample image, wherein the sample depth feature map includes feature information and depth of field information of each pixel point in the sample image; through the image reconstruction layer, performing image reconstruction based on the feature information of each pixel point in the sample depth feature map to obtain a reconstructed image corresponding to the sample image; determining the loss of the depth of field estimation model according to the sample image and the reconstructed image; and adjusting the parameters of the depth of field estimation model based on the loss to obtain a trained depth of field estimation model; A feature extraction unit, configured to extract a depth feature map of the image to be determined by using the depth estimation model, wherein the depth feature map includes depth information of each pixel in the image to be determined; A depth estimation unit, configured to calculate target depth information of the image to be judged based on the depth information of each pixel in the depth feature map; The type determination unit is used to determine the target scene type corresponding to the image to be determined based on the preset correspondence between the depth of field information and the scene type and the target depth of field information.

11. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the scene type determination method according to any one of claims 1 to 10.

12. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the scene type determination method according to any one of claims 1 to 10.

Images