Image processing method, device, electronic device and computer storage medium

Abstract

Embodiments of the present invention provide an image processing method, apparatus, electronic device, and computer storage medium. The image processing method includes: performing subject detection processing on a set of images to be cropped based on multiple subject categories to obtain multiple subject bounding box candidate sets; determining a target subject bounding box candidate set from the multiple subject bounding box candidate sets based on the subject category priority order; and determining at least one target subject bounding box from the target subject bounding box candidate set to crop the target image. Because the subject detection processing can detect multiple subject categories in the set of images to be cropped, it can also calibrate subject bounding boxes for different subject categories in the target image, and then determine at least one target subject bounding box from the target subject bounding box candidate set, thereby improving calibration accuracy.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of communication technologies, and in particular to an image processing method, apparatus, electronic device, and computer storage medium. Background Art

[0002] Inserting images into web pages such as search results pages or product browsing pages, and displaying web resources to users in a graphic and textual manner, reflects the product development trend in the Internet image reading era.

[0003] Considering that images displayed on web pages often vary in shape and size, displaying them directly without resizing them can make the entire webpage appear cluttered, impacting the user's reading experience and sometimes even creating a poor visual experience. Therefore, to maintain a consistent presentation style across images on a webpage, proper image cropping is necessary to achieve a good display experience. However, existing technologies fail to deliver satisfactory results in diverse target scenarios. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image processing method, apparatus, electronic device, and computer storage medium to solve or alleviate the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present invention, there is provided an image processing method, comprising: performing subject detection processing on a set of images to be cropped according to multiple subject categories to obtain multiple subject bounding box candidate sets; determining, for a target image in the set of images to be cropped, a target subject bounding box candidate set from the multiple subject bounding box candidate sets; and determining at least one target subject bounding box in the target subject bounding box candidate set to crop the target image.

[0006] According to a second aspect of an embodiment of the present invention, there is provided an image processing method, comprising: performing subject detection processing on a plurality of video frame sets corresponding to a plurality of videos, respectively, to obtain a plurality of subject bounding box candidate sets corresponding to a plurality of subject categories, respectively; for a target video frame set in the plurality of video frame sets, determining a target subject bounding box candidate set from the plurality of subject bounding box candidate sets; and determining at least one target subject bounding box in the target subject bounding box candidate set to crop a cover of the target video from the target video frame set.

[0007] According to a third aspect of an embodiment of the present invention, there is provided an image processing method, comprising: presenting an operation interface to a user, wherein a plurality of subject categories are presented in the operation interface; in response to a selection instruction input by the user in the operation interface, determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on a to-be-cropped picture set according to the plurality of subject categories; and determining at least one target subject bounding box in the target subject bounding box candidate set to crop the target picture.

[0008] According to a fourth aspect of an embodiment of the present invention, there is provided an image processing apparatus, comprising: a presentation module, configured to present an operation interface to a user, wherein a plurality of subject categories are presented in the operation interface; a first determination module, configured to respond to a selection instruction input by a user in the operation interface, and to determine a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on a to-be-cropped picture set according to the plurality of subject categories; and a second determination module, configured to determine at least one target subject bounding box in the target subject bounding box candidate set to crop the target picture.

[0009] According to a fifth aspect of an embodiment of the present invention, an image processing device is provided, comprising: a subject detection processing module, which performs subject detection processing on multiple video frame sets corresponding to multiple videos, respectively, to obtain multiple subject bounding box candidate sets corresponding to multiple subject categories, respectively; a first determination module, which determines, for a target video frame set in the multiple video frame sets, a target subject bounding box candidate set from the multiple subject bounding box candidate sets; and a second determination module, which determines at least one target subject bounding box in the target subject bounding box candidate set to crop a cover of the target video from the target video frame set.

[0010] According to a sixth aspect of an embodiment of the present invention, an image processing device is provided, comprising: a subject detection processing module, which performs subject detection processing on a set of pictures to be cropped according to multiple subject categories, to obtain multiple subject bounding box candidate sets; a first determination module, which determines, for a target picture in the set of pictures to be cropped, a target subject bounding box candidate set from the multiple subject bounding box candidate sets; and a second determination module, which determines at least one target subject bounding box in the target subject bounding box candidate set to crop the target picture.

[0011] According to the seventh aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; and a computer-readable medium configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect or the second aspect.

[0012] According to an eighth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect or the second aspect is implemented.

[0013] The solution of the embodiments of the present invention can perform subject detection on a set of images to be cropped based on multiple subject categories, obtaining multiple subject bounding box candidate sets. For a target image in the set of images to be cropped, a target subject bounding box candidate set is determined from the multiple subject bounding box candidate sets. At least one target subject bounding box from the target subject bounding box candidate set is determined to crop the target image. Because the subject detection process can detect multiple subject categories in the set of images to be cropped, subject bounding box calibration can be performed for different subject categories in the target image. Finally, at least one target subject bounding box is determined from the target subject bounding box candidate set, thereby improving calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0015] Figure 1 A schematic diagram of an exemplary network architecture applicable to an image processing method according to an embodiment of the present invention;

[0016] Figure 2A is a schematic block diagram of an image processing method according to another embodiment of the present invention;

[0017] Figure 2B A schematic block diagram of a subject bounding box selection process of an example of an image processing method according to another embodiment of the present invention;

[0018] Figure 2C A schematic block diagram of a subject bounding box selection process of another example of an image processing method according to another embodiment of the present invention;

[0019] Figure 2DA schematic block diagram of a subject bounding box selection process of another example of an image processing method according to another embodiment of the present invention;

[0020] Figure 3A A schematic diagram of a subject bounding box of an example of an image processing method according to another embodiment of the present invention;

[0021] Figure 3B A schematic block diagram of image blur detection is an example of an image processing method according to another embodiment of the present invention;

[0022] Figure 4 is a schematic block diagram of an image processing method according to another embodiment of the present invention;

[0023] Figure 5 is a schematic block diagram of an image processing method according to another embodiment of the present invention;

[0024] Figure 6 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention;

[0025] Figure 7 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention;

[0026] Figure 8 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention;

[0027] Figure 9 is a schematic structural diagram of an electronic device according to another embodiment of the present invention;

[0028] Figure 10 This is a hardware structure of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0030] The specific implementation of the embodiment of the present invention is further described below with reference to the accompanying drawings of the embodiment of the present invention. Figure 1 FIG1 is a schematic diagram of an exemplary network architecture 100 applicable to an image processing method according to an embodiment of the present invention. The network architecture 100 includes a client system 120, a tailoring server system 160, and an application server system 170 connected to each other via a network 110. It should be understood that Figure 1 The network architecture shown is only exemplary, and the embodiments of the present invention are also applicable to other network architectures. Figure 1 The network architecture shown is applicable to all examples of the embodiments of the present invention. All network architectures shown are solutions to facilitate understanding of the embodiments of the present invention and should not be understood as limiting the application scenarios, interaction methods, and various communication relationships of the embodiments of the present invention.

[0031] For example, by way of example and not limitation, Figure 1 As shown, application server 170 can be connected to cropping server system 160 directly or indirectly via network 110, or by bypassing network 100. Image data storage 174 can store images to be cropped. Image data storage 164 can store a collection of images to be cropped obtained from image data storage 174, as well as a collection of cropped images. For example, the application server system can provide various services to client system 120. These services include, but are not limited to, website access services, video services, streaming media services, etc. For example, application server system 170 can be a short video server. For example, the client system can be installed with a short video application.

[0032] In one specific implementation, client system 120 can be an electronic device comprising hardware, software, or embedded logic components, or a combination of two or more such components, and capable of performing appropriate functions implemented or supported by client system 120. By way of example and not limitation, client system 120 can include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smartphone, other suitable electronic devices, or any suitable combination thereof. Embodiments of the present invention contemplate any suitable client system 120. Client system 120 can enable network users at client system 120 to access network 110. Client system 120 can enable its users to communicate with other users at other client systems 120.

[0033] Figure 2A FIG. 4 is a schematic block diagram of an image processing method according to another embodiment of the present invention. Figure 2A The image processing method comprises:

[0034] 210: Perform subject detection on the cropped image set according to multiple subject categories to obtain multiple subject bounding box candidate sets.

[0035] It should be understood that the subject detection process described herein can be object detection. It can also be saliency detection. Object detection, as a crucial component of image understanding, aims to locate all objects of interest in an image and determine their locations and sizes. Because various objects have varying appearances, shapes, and postures, coupled with interference from factors such as illumination and occlusion during imaging, deep learning can be applied to object detection. For example, subject detection in embodiments of the present invention may include a combination of object detection and saliency detection. For example, when performing saliency detection, the subject edit box described above represents a salient region. For example, in a specific scene, the primary object is a known object type, such as a person, face, or car. For example, using a person as an example, object detection can be performed on an image of the person to calculate the object's location and category. For example, the location is represented by the minimum bounding box surrounding the object. Figure 3A FIG. 1 is a schematic diagram of a subject boundary box of an example of an image processing method according to another embodiment of the present invention. Figure 3A As shown, objects of different subject categories may have corresponding bounding boxes (minimum boxes). It should be understood that the processing of detecting multiple subject categories in the embodiment of the present invention may be implemented in the same detector or in multiple detectors respectively. For example, when performing target detection, the above-mentioned subject bounding box is the target subject bounding box. It should also be understood that although the image processing in the embodiment of the present invention is used for processing including image cropping, the image processing in the embodiment of the present invention may be for processing two-dimensional images or three-dimensional images. For example, in performing three-dimensional image processing, the above-mentioned outer bounding box may be a cylindrical outer bounding box, a rectangular outer bounding box, a polyhedron outer bounding box, a sphere or ellipsoid outer bounding box, etc. For example, deep learning is used to perform the above-mentioned subject detection processing. For example, different neural networks and their combinations are used to perform the above-mentioned subject detection processing. For example, the shape and number of the bounding boxes in the subject detection processing may be arbitrary. For example, the shape of the bounding box may be a rectangle, a circle, or a polygon with any number of sides such as a triangle. As Figure 2CAs shown in , the shapes of the subject bounding boxes can be different. For example, subject bounding boxes for the same category can be different or the same. For example, subject bounding boxes for different categories can be the same or different. In other words, the shape of the subject bounding box can be independent of the subject category, that is, they can be unrelated. As an example, the shape of the subject bounding box is associated with the subject category. For example, the subject bounding boxes can be identified by different colors. For example, for an image to be cropped with multiple and complex subject categories, different subject bounding box candidate sets in this article can be distinguished by different colors. Color parameters facilitate distinguishing different candidate sets, thereby improving image processing efficiency. It should be understood that color does not need to be used to distinguish candidate sets. For example, different subject categories have different parameters, feature vectors, or structured feature information, and one or more of these information can be used to distinguish different subject bounding box candidate sets. It should be understood that the multiple subject bounding box candidate sets in this article can be obtained in any order. For example, they can be obtained according to the subject category priority order, thereby corresponding to the subject priority order in the next step. This is due to the simplicity of the program configuration. For example, subject detection processing is performed on the cropped picture set according to multiple subject categories to obtain multiple subject bounding box candidate sets. Subject detection processing can be performed on the cropped picture set based on the first subject category priority order to obtain multiple subject bounding box candidate sets.

[0036] 220: For the target picture in the to-be-cropped picture set, determine a target subject bounding box candidate set from a plurality of subject bounding box candidate sets.

[0037] It should be understood that the same or similar subject category priority order as used in the previous step can be used. For example, the target image is the image to be cropped from the set of images to be cropped. In one example, the subject category priority order is exactly the same as that used in the previous step. This allows for the establishment of a unified data partitioning standard. In other words, when developing an object-oriented image processing method according to an embodiment of the present invention, the same class template or method template can be used, thereby improving computational efficiency. As another example, a different subject category priority order from that used in the previous step can be used. For example, in the previous step, a first subject category priority order is used, while a second subject category priority order is used when determining the target subject bounding box candidate set. For example, the second subject category priority order can be an adjustment to the first subject category priority order. For example, the image processing method according to an embodiment of the present invention can be suitable for batch processing of a large number of image crops. For example, the first subject category priority order is set for the entire dataset. For example, the second subject category priority order is set for a specific image or subset of images. For example, the second subject category priority order is a subset of the first subject category priority order. For example, the second subject category priority order is a portion intercepted from the first subject category priority order. For example, the first subject category priority includes categories that are not included in the second subject category. For example, the aforementioned missing categories are all categories with lower priorities, then the aforementioned first subject category priority order can be directly intercepted to obtain the second subject category priority order. For example, a subject category priority sorting method can be configured, and before executing the first step of an embodiment of the present invention, a first sorting process is used to obtain the first subject category priority order. For example, a second sorting process can be set for the target image to be cropped. For example, the second sorting process can be a filter for the result of the first sorting process. For example, in the first and second sorting processes for the same multiple subject categories, the first sorting is different from the second sorting. In other words, the first sorting is re-sorted to obtain the second sorting. For example, the first sorting and the second sorting are for different multiple subject categories. In other words, the first sorting is filtered and re-sorted.

[0038] It should also be understood that the subject category priority order can be a pre-set priority order. It can also be obtained through experimentation or set for specific image types or image application scenarios. For example, in short video processing scenarios, such as when determining a short video cover, people typically have the highest priority among the subject categories. Therefore, people can be set to the highest priority in the above order. For example, in a pet social scene, pets are the most important subject category in the image, so pets can be set to the highest priority. For example, if people are set to the second highest priority. For example, if the intention is to present the interaction between people and pets in the short video cover, people can be set to the second highest priority. For example, people can also be set to the highest priority and pets to the second highest priority. For example, in a scenario where a target video is determined and a video cover is determined, multiple feature video frames in the target video can be determined, and a subject bounding box candidate set can be determined for each feature frame. For example, the correlation between each video frame can be compared. For example, similarity features between adjacent video frames can be compared. For example, similarity features between different video frames with a frame distance less than a predetermined value can be compared. For example, the relevance of the video to the subject category is determined based on the similarity feature. The priority of the subject category is determined based on the aforementioned relevance. For example, the feature video frame contains the subject element of the video. For example, the feature video frame is associated with the subject element of the video. For example, the feature video frame includes the subject element of the video. For example, the feature video includes the tagged element of the video.

[0039] 230: Determine at least one target subject bounding box in the target subject bounding box candidate set to crop the target image.

[0040] It should be understood that when determining at least one target subject bounding box in the set of candidate target subject bounding boxes, a single target subject bounding box can be determined. For example, the target subject bounding box includes the center of the image. In other words, the center of the target image is within the range of the target subject bounding box. For example, the size of the target subject bounding box is greater than a predetermined value. For example, the size of the target subject bounding box is greater than a predetermined ratio of the target image. For example, the size of the target subject bounding box has a first ratio within the cropping area. For example, the cropping area has a second ratio within the target image. For example, the first ratio is determined based on the second ratio. Alternatively, the second ratio is determined based on the first ratio. This ensures that the target subject bounding box is prominently presented in the cropping area. Furthermore, the integrity of the content within the target subject bounding box is ensured. For example, the first ratio can be greater than the second ratio. Preferably, the first ratio is approximately equal to the second ratio. For example, the second ratio can be determined during pattern recognition. After pattern recognition is performed, a set of images to be cropped is obtained. For example, after obtaining the set of images to be cropped, pattern recognition is performed on the images to be cropped. The embodiment of the present invention is applicable to any processing flow, and the order of the above-mentioned various processing is not limited.

[0041] Since the subject detection process can detect multiple subject categories in the cropped image set, it can then calibrate the subject bounding boxes of different subject categories in the target image, and then determine at least one target subject bounding box from the target subject bounding box candidate set, thereby improving the accuracy of the calibration.

[0042] For example, as an example, Figure 2BAs shown, the initial set of images to be cropped includes roughly three rectangular shapes, corresponding to three subject categories: people, plants, and objects. The first figure shows all subject bounding boxes for all images in the set of images to be cropped in the three aforementioned categories, i.e., multiple subject bounding box candidate sets. The next figure shows the target subject bounding box candidate set for people as the subject category. It should be understood that in this example, the subject category with the highest priority can be people, plants, or objects. For example, people as the subject category have the highest priority, so people are selected as the target subject bounding box candidate set. For example, people as the subject category may not have the highest priority, but plants or objects do. However, since these subject categories correspond to empty candidate sets, based on the subject category priority order, the subject category corresponding to people is selected as the target subject bounding box candidate set. In other words, people can be the first non-empty candidate set. It should be understood that the above screening method is merely exemplary, and the first non-empty candidate set may not be selected as the target subject bounding box candidate set. Following the process arrows shown in the figure, it can be seen that the target subject bounding box is selected. For example, although a single target subject bounding box is schematically illustrated, it should be understood that embodiments of the present invention may also select multiple target subject bounding boxes. For example, cropping may be performed based on the target subject bounding box. For example, a cropping region may be determined based on the target subject bounding box. For example, cropping may be performed based on the cropping region to obtain a cropped image.

[0043] In another example, Figure 2DThis figure illustrates subject bounding box selection processing in another example of an image processing method according to another embodiment of the present invention. The first figure in the image processing flow shows that a target image includes two subject categories: people and houses. The second figure shows that, for this target image, a person bounding box candidate set including three person bounding boxes is determined (selected) from two candidate sets: a person bounding box candidate set and a house bounding box candidate set. The third figure illustrates the determination of a target person bounding box (the second bounding box) from the three person bounding boxes. For example, the determination can be based on the person bounding box's position in the image, its size, or a combination of the two. It should be understood that other methods can also be used to determine the target person bounding box, including but not limited to the various methods in this example and other examples described below. The fourth figure illustrates the result of cropping the target image based on the target person bounding box. It should be understood that, in this example, only two specific subject categories are shown, but in other examples, more or fewer subject categories may be included. Furthermore, the number of subjects in each category can be arbitrary. It should also be understood that in this example, a process for cropping a target image is shown. However, in other examples, the operation can be performed based on a set of images to be cropped, including multiple images. The multiple images can be continuous video frames in a specific video, discontinuous video frames in a specific video, discontinuous video frames in multiple videos, or multiple images that are related or unrelated to each other.

[0044] In another implementation of the present invention, for a target image in the image set to be cropped, determining a target subject bounding box candidate set from multiple subject bounding box candidate sets includes: determining a first non-empty candidate set for the target image in the image set to be cropped based on a subject category priority order. For example, multiple non-empty candidate sets may also be determined for the target image in the image set to be cropped based on a subject category priority order. For example, determining a target subject bounding box candidate set from the multiple non-empty candidate sets. In one example, a preset candidate set in the non-empty candidate set is determined as the target subject bounding box candidate set. For example, a user sets a preferred candidate set based on historical statistical results or experience, and this candidate set is determined as the target subject bounding box candidate set. For example, for the target image in the image set to be cropped, determining a target subject bounding box candidate set from multiple subject bounding box candidate sets includes: determining multiple non-empty candidate sets for the target image in the image set to be cropped; and determining the target subject bounding box candidate set from the multiple non-empty candidate sets based on a subject category priority order.

[0045] For the target picture in the to-be-cropped picture set, determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets, including: for the target picture in the to-be-cropped picture set, based on the subject category priority order, determining whether a current subject bounding box candidate set corresponding to a current subject category priority is an empty candidate set; if so, updating the current subject category priority with a next subject category priority based on the subject category priority order.

[0046] It should be understood that when performing priority update processing, for example, a pointer variable is used to shift the priority of the next subject category from the current subject category in the subject category queue. For example, the subject category queue can be updated. For example, the aforementioned pointer variable shift and subject category update processing can be performed in parallel. For example, the queue can be updated before or after the pointer variable shift. For example, the queue can be updated before shifting to a specific subject category.

[0047] As an example, we set category priorities for objects in an image. This priority can also be user-defined. For example, we can set the first-level priority category to faces, the second-level priority categories to vehicles, dogs, balls, and so on, and the third-level priority categories to trees. After subject detection in the image, the algorithm considers object categories from priority one to priority three in sequence. First, objects at priority one are added to the candidate set. If no objects are found at that priority level, the detection results for objects at the next priority level are added to the candidate set, and the calibration module is run, and so on. Only when all levels are clear of the target object does the algorithm return the center point of the saliency detection.

[0048] Because multi-level subject target calibration or crop shot selection does not require repeated execution of the aforementioned subject detection algorithm, requiring only selection or filtering, or pointer variable processing or queue update processing such as described above, the additional computational overhead is minimal. In other words, there is no significant efficiency degradation compared to a single-level approach, thereby improving computational efficiency.

[0049] In another implementation of the present invention, if the current subject bounding box candidate set is a non-empty candidate set, the current subject bounding box candidate set is determined as the target subject bounding box candidate set.

[0050] In another implementation of the present invention, determining at least one target subject bounding box in the target subject bounding box candidate set can be performed using a subject selection algorithm. For example, the subject selection algorithm uses a model integration method to combine the various input element parameters, normalizes the output values of multiple learners, and then determines the subject position in the image according to the weight coefficient fusion method. When there are multiple subjects in the picture, based on the scores obtained by subject calibration, the lens position that can contain the most high-scoring subjects is preferentially selected. In addition, the embodiment of the present invention also supports adjustment based on subject category. For example, it can be set that the face class is not allowed to be cut. After the basic position of the lens is selected, it will be judged whether the left and right boundaries have been cut off by the face class, and fine-tuning will be performed to avoid this situation.

[0051] It should be understood that the above-mentioned various element parameters can be input as different inputs into the subject selection algorithm for determining the target subject bounding box candidate set. In the determination module of at least one target subject bounding box. For example, the determination module can adopt the above-mentioned example, and the above-mentioned multiple inputs can be used as independent inputs. For example, the multiple inputs can also be used as associated inputs. For example, the multiple inputs include but are not limited to the target subject bounding box candidate set, the clarity of the image set to be cropped (including the clarity of the target image), and the subject bounding box parameter set of the image set to be cropped (including the subject bounding box parameters of the target image). For example, determining at least one target subject bounding box in the target subject bounding box candidate set includes: performing saliency detection processing on the target image to obtain multiple saliency probability parameters corresponding to multiple positions in the target image; and determining at least one target subject bounding box in the target subject bounding box candidate set based on the multiple saliency probability parameters and the subject bounding box parameters.

[0052] In another implementation of the present invention, determining at least one target subject bounding box in a target subject bounding box candidate set includes: performing image blur detection on a target image to obtain clarity of the target image; and determining at least one target subject bounding box in the target subject bounding box candidate set based on the clarity of the target image and subject bounding box parameters.

[0053] In another implementation of the present invention, image blur detection may be performed on all images in the cropped image set. For example, the detection may be performed based on the aforementioned first subject category priority order, or based on the aforementioned second subject category priority order.

[0054] Performing image blur detection on the target image to obtain the clarity of the target image includes: performing grayscale processing on the target image to obtain a grayscale image of the target image; and performing grayscale mutation enhancement processing on the grayscale image to determine the clarity of the target image. Figure 3BFIG. 1 is a schematic block diagram of an image blur detection example of an image processing method according to another embodiment of the present invention. Figure 3B As shown, accordingly, the grayscale mutation enhancement processing is performed according to the picture shown in the figure to determine the clarity of the target picture. For example, the grayscale mutation enhancement processing shown may include but is not limited to Laplace transform or Gaussian filtering. It should be understood that Figures 3A and Figure 3B Can also be used as Figure 2B An example of an actual image.

[0055] In another implementation of the present invention, determining at least one target subject bounding box in a target subject bounding box candidate set includes: determining multiple distances from multiple candidate subject bounding boxes in the target subject bounding box candidate set to a center of a target image; and selecting at least one target subject bounding box from the multiple candidate subject bounding boxes by at least comparing the multiple distances.

[0056] In another implementation of the present invention, determining at least one target subject bounding box in a target subject bounding box candidate set includes: determining sizes of multiple candidate subject bounding boxes in the target subject bounding box candidate set; and selecting at least one target subject bounding box from the multiple candidate subject bounding boxes by sorting the bounding box sizes.

[0057] For example, at least one target subject bounding box can be selected from the multiple candidate subject bounding boxes based on both the multiple distances and the sizes of the multiple candidate subject bounding boxes. For example, the distance comparison process for comparing the multiple distances and the bounding box size sorting process can be performed in parallel. For example, the two processes can also be performed serially. For example, the distance comparison process can be performed first, followed by the bounding box size sorting process. Alternatively, the bounding box size sorting process can be performed first, followed by the distance comparison process. For example, the selection process for selecting the at least one target subject bounding box can employ time division multiplexing, i.e., the first selection is performed based on the distance comparison process, the second selection is performed through the bounding box size sorting process, the third selection is performed based on the distance comparison process, and the fourth selection is performed through the bounding box size sorting process. For example, the above-described alternating selection scheme is merely exemplary, and embodiments of the present invention can employ any number of alternating processes. Since the alternating process does not require traversing all processing objects in each process, computational efficiency is improved.

[0058] In another implementation of the present invention, the method further includes image pattern analysis processing. For example, the image pattern analysis processing includes: performing pattern recognition on the original image through edge detection processing to obtain a set of images to be cropped.

[0059] In another implementation of the present invention, edge detection processing is used to identify the layout of the original image to obtain a set of images to be cropped, including: determining the plane coordinate parameters of multiple edge points of each original image in the original image set; determining the polar coordinate parameters corresponding to the plane coordinate parameters through coordinate transformation; and identifying the layout of each original image based on the polar coordinate parameters through straight line detection to obtain the set of images to be cropped.

[0060] For example, when performing the above-mentioned image layout analysis and processing, you can first use a deep learning model for edge detection to obtain a set of edge points for the entire image. For example, the above-mentioned coordinate transformation is the Hough transform. For example, among the clustered points of the projection coordinates of the Hough transform, the first K are selected as the candidate set to enter the verification stage. For example, the verification stage performs a scan and verification on the horizontal and vertical straight line candidates. If the straight line is broken in the image and there are no other straight lines there, it is deleted from the candidate set. For example, for a straight line that runs through the image, check whether there is a straight line at its corresponding position, otherwise delete it. According to the final straight line position matched in the preset layout, the layout of the image can be determined, and then the corresponding valid screen area can be cropped according to the layout.

[0061] For example, using the results of the line detection in the previous step, if there is a straight line that is symmetrical vertically or horizontally, determine whether it is a pure color area (multiple black edges). If so, cut it out from the face.

[0062] It should be understood that the image layout analysis processing of the embodiment of the present invention also includes intelligent size determination processing. For example, based on obtaining the effective picture of the layout and black border cropping, for example, when the user does not specify the cropping picture, the intelligent size determination processing selects from multiple picture sizes. For example, the multiple picture sizes include but are not limited to sizes with aspect ratios (or horizontal and vertical ratios) of 16:9, 4:3, 1:1, 3:4, and 9:16. For example, horizontal pictures in the original picture set are given priority. For example, vertical pictures in the original picture set are given priority. For example, layout selection is performed based on the average size of the significant area. For example, narrow cropping of the significant area size may be included. For example, the preferred cropping size is 9:16->3:4->1:1.

[0063] In another implementation of the present invention, image inpainting can be performed, such as detecting and inpainting patterns like subtitles and logos. For example, image quality enhancement can also be performed. For example, image quality enhancement techniques such as denoising, super-resolution, and HDR processing can be used. This effectively addresses the image quality degradation caused by cropping, thereby improving the cropping effect of smart cropping. The method further includes: determining a cropping region based on at least one target subject bounding box; and performing image quality enhancement on the cropping region to obtain a cropped image.

[0064] In another implementation of the present invention, the cropped area is subjected to image quality enhancement processing to obtain a cropped picture, including: segmenting the candidate patch objects in the target picture to obtain multiple candidate patch object fragments; determining at least one candidate patch object fragment existing in the cropped area; and patching the cropped area by replacing at least one candidate patch object fragment with a background image in the cropped area to obtain a cropped picture.

[0065] It should be understood that image quality enhancement can utilize at least one of denoising, deblurring, high dynamic range imaging, and super-resolution. The solution of this embodiment of the present invention integrates image compression noise removal, super-resolution, and sharpening into a single deep learning model for training, resulting in a high-speed image quality enhancement solution.

[0066] Figure 4 is a schematic block diagram of an image processing method according to another embodiment of the present invention; Figure 4 The image processing method comprises:

[0067] 410: Perform subject detection processing on multiple video frame sets corresponding to multiple videos to obtain multiple subject bounding box candidate sets corresponding to multiple subject categories.

[0068] 420: For a target video frame set in the multiple video frame sets, determine a target subject bounding box candidate set from the multiple subject bounding box candidate sets;

[0069] 430: Determine at least one target subject bounding box in the target subject bounding box candidate set to crop the cover of the target video from the target video frame set.

[0070] Figure 5 FIG. 4 is a schematic block diagram of an image processing method according to another embodiment of the present invention. Figure 5 The image processing method comprises:

[0071] 510: Presenting an operation interface to the user, wherein a plurality of subject categories are presented in the operation interface.

[0072] 520: In response to a selection instruction input by the user in the operation interface, determine a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on the to-be-cropped image set according to the plurality of subject categories.

[0073] 530: Determine at least one target subject bounding box in the target subject bounding box candidate set to crop the target image.

[0074] As an example, the image processing method further includes: obtaining a selection instruction input by a user in an operation interface, where the selection instruction indicates a selection of a target subject. In response to the selection instruction input by the user in the operation interface, determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to a plurality of subject categories includes: determining a subject bounding box candidate set corresponding to the target subject from the plurality of subject bounding box candidate sets as the target subject bounding box candidate set.

[0075] As an example, the image processing method also includes: obtaining a selection instruction input by a user in an operation interface, the selection instruction indicating a selection of a target subject category priority, wherein, in response to the selection instruction input by the user in the operation interface, a target subject bounding box candidate set is determined from multiple subject bounding box candidate sets corresponding to multiple subject categories, including: determining a subject bounding box candidate set that meets the target subject category priority among the multiple subject bounding box candidate sets as the target subject bounding box candidate set.

[0076] As an example, determining at least one target subject bounding box in a target subject bounding box candidate set to crop a target image includes: responding to a user in an operation interface, determining at least one target subject bounding box in a target subject bounding box candidate set to crop the target image.

[0077] As an example, the image processing method further includes: presenting a cropping result of the target image to the user in an operation interface.

[0078] Figure 6 FIG. 4 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention. Figure 6 An image processing device comprising:

[0079] Presentation module 610, presenting an operation interface to the user, wherein the operation interface presents multiple subject categories;

[0080] A first determination module 620, in response to a selection instruction input by a user in an operation interface, determines a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to a plurality of subject categories, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on the to-be-cropped image set according to the plurality of subject categories;

[0081] The second determining module 630 determines at least one target subject bounding box in the target subject bounding box candidate set to crop the target image.

[0082] As an example, the image processing apparatus further includes: a first acquisition module configured to acquire a selection instruction input by a user in an operation interface, the selection instruction indicating selection of a target subject; and a first determination module configured to determine, from among a plurality of subject bounding box candidate sets, a subject bounding box candidate set corresponding to the target subject as the target subject bounding box candidate set.

[0083] In addition, the user interface may present a candidate set of target subject bounding boxes. Option subjects within the candidate set may also be presented for the user to select. An input area for at least one subject bounding box identifier may also be provided so that a target subject bounding box can be determined from among multiple subject bounding boxes based on the identifier input.

[0084] As an example, the image processing apparatus further includes: a first acquisition module configured to acquire a selection instruction input by a user in an operation interface, the selection instruction indicating a selection of a target subject category priority; and a first determination module configured to determine, from among the plurality of subject bounding box candidate sets, a subject bounding box candidate set that meets the target subject category priority as the target subject bounding box candidate set.

[0085] In addition, the user interface may present a candidate set of target subject bounding boxes. Option subjects within the candidate set may also be presented for the user to select. An input area for at least one subject bounding box identifier may also be provided so that a target subject bounding box can be determined from among multiple subject bounding boxes based on the identifier input.

[0086] As an example, the second determining module is specifically configured to: respond to a subject bounding box selection instruction input by a user in the operation interface, determine at least one target subject bounding box in the target subject bounding box candidate set, so as to crop the target image.

[0087] As an example, the image processing apparatus further includes a cropping result presentation module, which presents the cropping result of the target image to the user in the operation interface.

[0088] The method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.), and a PC.

[0089] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0090] Figure 7 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention; Figure 5An image processing device comprising:

[0091] The subject detection processing module 710 performs subject detection processing on a plurality of video frame sets corresponding to a plurality of videos, and obtains a plurality of subject bounding box candidate sets corresponding to a plurality of subject categories;

[0092] A first determining module 720 determines, for a target video frame set in the multiple video frame sets, a target subject bounding box candidate set from the multiple subject bounding box candidate sets;

[0093] The second determining module 730 determines at least one target subject bounding box in the target subject bounding box candidate set to crop the cover of the target video from the target video frame set.

[0094] Since the subject detection process can detect multiple subject categories in the cropped image set, it can then calibrate the subject bounding boxes of different subject categories in the target image, and then determine at least one target subject bounding box from the target subject bounding box candidate set, thereby improving the accuracy of the calibration.

[0095] The method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.), and a PC.

[0096] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0097] Figure 8 is a schematic block diagram of an image processing apparatus according to another embodiment of the present invention; Figure 6 An image processing device comprising:

[0098] The subject detection processing module 810 performs subject detection processing on the image set to be cropped according to multiple subject categories to obtain multiple subject bounding box candidate sets;

[0099] A first determining module 820 determines, for a target picture in the to-be-cropped picture set, a target subject bounding box candidate set from a plurality of subject bounding box candidate sets;

[0100] The second determining module 830 determines at least one target subject bounding box in the target subject bounding box candidate set to crop the target.

[0101] Since the subject detection process can detect multiple subject categories in the cropped image set, it can then calibrate the subject bounding boxes of different subject categories in the target image, and then determine at least one target subject bounding box from the target subject bounding box candidate set, thereby improving the accuracy of the calibration.

[0102] The method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.), and a PC.

[0103] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0104] In another implementation of the present invention, the first determination module is specifically used to: for the target image in the set of images to be cropped, based on the subject category priority order, determine whether the current subject bounding box candidate set corresponding to the current subject category priority is an empty candidate set; if so, based on the subject category priority order, update the current subject category priority using the next subject category priority.

[0105] In another implementation of the present invention, if the current subject bounding box candidate set is a non-empty candidate set, the current subject bounding box candidate set is determined as the target subject bounding box candidate set.

[0106] In another implementation of the present invention, the second determination module is specifically used to: perform saliency detection processing on the target image to obtain multiple saliency probability parameters corresponding to multiple positions in the target image; and determine at least one target subject bounding box in the target subject bounding box candidate set based on the multiple saliency probability parameters and the subject bounding box parameters.

[0107] In another implementation of the present invention, the second determination module is specifically used to: perform image blur detection on the target image to obtain the clarity of the target image; and determine at least one target subject bounding box in the target subject bounding box candidate set based on the clarity of the target image and the subject bounding box parameters.

[0108] In another implementation of the present invention, the second determination module is specifically configured to: perform grayscale processing on the target image to obtain a grayscale image of the target image; and perform grayscale mutation enhancement processing on the grayscale image to determine the clarity of the target image.

[0109] In another implementation of the present invention, the second determination module is specifically used to: determine multiple distances from multiple candidate subject bounding boxes in the target subject bounding box candidate set to the center of the target image; and select at least one target subject bounding box from the multiple candidate subject bounding boxes by at least comparing the multiple distances.

[0110] In another implementation of the present invention, the second determination module is specifically configured to: determine the sizes of multiple candidate subject bounding boxes in the target subject bounding box candidate set; and select at least one target subject bounding box from the multiple candidate subject bounding boxes by sorting the bounding box sizes.

[0111] In another implementation of the present invention, the device method further includes: a pattern recognition module that performs pattern recognition on the original image through edge detection processing to obtain a set of images to be cropped.

[0112] In another implementation of the present invention, the pattern recognition module is specifically used to: determine the plane coordinate parameters of multiple edge points of each original picture in the original picture set; determine the polar coordinate parameters corresponding to the plane coordinate parameters through coordinate transformation; and identify the pattern of each original picture based on the polar coordinate parameters through straight line detection to obtain the picture set to be cropped.

[0113] In another implementation of the present invention, the apparatus further includes a cropping region determination module and an image quality enhancement module. The cropping region determination module is configured to determine a cropping region based on at least one target subject bounding box. The image quality enhancement module is configured to perform image quality enhancement processing on the cropping region to obtain a cropped image.

[0114] In another implementation of the present invention, the image quality enhancement module is specifically used to: segment the candidate repair objects in the target image to obtain multiple candidate repair object fragments; determine at least one candidate repair object fragment existing in the cropped area; and repair the cropped area by replacing at least one candidate repair object fragment with a background image in the cropped area to obtain a cropped image.

[0115] Figure 9 This is a schematic structural diagram of an electronic device in another embodiment of the present invention; the electronic device may include:

[0116] One or more processors 901;

[0117] The computer readable medium 902 may be configured to store one or more programs,

[0118] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in the above embodiments.

[0119] Since the subject detection process can detect multiple subject categories in the cropped image set, it can then calibrate the subject bounding boxes of different subject categories in the target image, and then determine at least one target subject bounding box from the target subject bounding box candidate set, thereby improving the accuracy of the calibration.

[0120] Figure 10 The hardware structure of the electronic device of another embodiment of the present invention is as follows; Figure 10 As shown, the hardware structure of the electronic device may include: a processor 1001, a communication interface 1002, a computer-readable medium 1003 and a communication bus 1004;

[0121] The processor 1001, the communication interface 1002, and the computer-readable medium 1003 communicate with each other via the communication bus 804;

[0122] Optionally, the communication interface 1002 may be an interface of a communication module;

[0123] The processor 1001 may be specifically configured to: perform subject detection processing on the image set to be cropped according to multiple subject categories to obtain multiple subject bounding box candidate sets; determine a target subject bounding box candidate set from the multiple subject bounding box candidate sets for the target image in the image set to be cropped; determine at least one target subject bounding box in the target subject bounding box candidate set to crop the target image, or,

[0124] Performing subject detection processing on a plurality of video frame sets corresponding to a plurality of videos to obtain a plurality of subject bounding box candidate sets corresponding to a plurality of subject categories; determining a target subject bounding box candidate set from the plurality of subject bounding box candidate sets for a target video frame in the plurality of video frames; determining at least one target subject bounding box in the target subject bounding box candidate set to crop the target video frame as a cover of the target video, or,

[0125] An operation interface is presented to a user, wherein a plurality of subject categories are presented in the operation interface; in response to a selection instruction input by the user in the operation interface, a target subject bounding box candidate set is determined from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on a set of images to be cropped according to the plurality of subject categories; and at least one target subject bounding box in the target subject bounding box candidate set is determined to crop the target image.

[0126] The processor 1001 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0127] The computer-readable medium 1003 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0128] Since the subject detection process can detect multiple subject categories in the cropped image set, it can then calibrate the subject bounding boxes of different subject categories in the target image, and then determine at least one target subject bounding box from the target subject bounding box candidate set, thereby improving the accuracy of the calibration.

[0129] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium can be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program configured for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0130] Computer program code configured to perform the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or portion of code, which contains one or more executable instructions configured to implement the specified logical function. The above-mentioned specific embodiments have specific sequential relationships, but these sequential relationships are merely exemplary. During the specific implementation, these steps may be fewer, more, or the execution order may be adjusted. In other words, in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as combinations of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments described in this application may be implemented in software or hardware. The names of these modules do not, in some cases, limit the modules themselves.

[0133] As another aspect, the present application further provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method described in the above embodiment.

[0134] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device: performs subject detection processing on the set of pictures to be cropped according to multiple subject categories, and obtains multiple subject bounding box candidate sets; for the target picture in the set of pictures to be cropped, determines a target subject bounding box candidate set from the multiple subject bounding box candidate sets; determines at least one target subject bounding box in the target subject bounding box candidate set to crop the target picture, or,

[0135] Performing subject detection processing on a plurality of video frame sets corresponding to a plurality of videos to obtain a plurality of subject bounding box candidate sets corresponding to a plurality of subject categories; determining a target subject bounding box candidate set from the plurality of subject bounding box candidate sets for a target video frame in the plurality of video frames; determining at least one target subject bounding box in the target subject bounding box candidate set to crop the target video frame as a cover of the target video, or,

[0136] An operation interface is presented to a user, wherein a plurality of subject categories are presented in the operation interface; in response to a selection instruction input by the user in the operation interface, a target subject bounding box candidate set is determined from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on a set of images to be cropped according to the plurality of subject categories; and at least one target subject bounding box in the target subject bounding box candidate set is determined to crop the target image.

[0137] As used in various embodiments of the present disclosure, the terms "first," "second," "the first," or "the second" may modify various components regardless of order and / or importance, but these terms do not limit the corresponding components. The above terms are merely configured to distinguish one element from another. For example, a first user device and a second user device represent different user devices, even though both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0138] When one element (for example, a first element) is referred to as being “(operably or communicably) coupled” or “(operably or communicably) coupled to” or “connected to” another element (for example, a second element), it should be understood that the one element is directly connected to the other element or that the one element is indirectly connected to the other element via yet another element (for example, a third element). Conversely, it should be understood that when an element (for example, a first element) is referred to as being “directly connected” or “directly coupled” to another element (the second element), there is no element (for example, a third element) interposed therebetween.

[0139] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the aforementioned technical features, but also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features having similar functions disclosed in this application.

Claims

1. An image processing method, comprising: Based on a preset first subject category priority, subject detection processing is performed on the cropped image set according to multiple subject categories to obtain multiple subject bounding box candidate sets; For the target image in the to-be-cropped image set, determining a target subject bounding box candidate set from the multiple subject bounding box candidate sets based on a second subject category priority, wherein the second subject category priority is obtained by adjusting the first subject category priority; Determine at least one target subject bounding box in the target subject bounding box candidate set to crop the target image, wherein a size of the target subject bounding box has a first ratio in a cropping area, and the cropping area has a second ratio in the target image, the second ratio is determined in pattern recognition of the set of images to be cropped, and the first ratio is determined based on the second ratio.

2. The method according to claim 1, wherein The determining, for the target picture in the to-be-cropped picture set, a target subject bounding box candidate set from the plurality of subject bounding box candidate sets based on the second subject category priority, comprises: For the target image in the set of images to be cropped, based on the subject category priority order, determine whether the current subject bounding box candidate set corresponding to the current subject category priority is an empty candidate set. If so, based on the subject category priority order, update the current subject category priority using the next subject category priority.

3. The method according to claim 2, wherein: If the current subject bounding box candidate set is a non-empty candidate set, the current subject bounding box candidate set is determined as the target subject bounding box candidate set.

4. The method according to claim 1, wherein The determining of at least one target subject bounding box in the target subject bounding box candidate set includes: Performing saliency detection processing on the target image to obtain a plurality of saliency probability parameters corresponding to a plurality of positions in the target image; At least one target subject bounding box in the target subject bounding box candidate set is determined based on the multiple saliency probability parameters and subject bounding box parameters.

5. The method according to claim 1, wherein The determining of at least one target subject bounding box in the target subject bounding box candidate set includes: Performing image blur detection on the target image to obtain the clarity of the target image; At least one target subject bounding box in the target subject bounding box candidate set is determined based on the definition of the target image and subject bounding box parameters.

6. The method according to claim 5, wherein: The performing image blur detection on the target image to obtain the clarity of the target image includes: Performing grayscale processing on the target image to obtain a grayscale image of the target image; The grayscale image is subjected to grayscale mutation enhancement processing to determine the clarity of the target image.

7. The method according to claim 1, wherein The determining of at least one target subject bounding box in the target subject bounding box candidate set includes: Determining a plurality of distances from a plurality of candidate subject bounding boxes in the target subject bounding box candidate set to a center of the target image; The at least one target subject bounding box is selected from the plurality of candidate subject bounding boxes at least by comparing the plurality of distances.

8. The method according to claim 1, wherein The determining of at least one target subject bounding box in the target subject bounding box candidate set includes: Determining the sizes of multiple candidate subject bounding boxes in the target subject bounding box candidate set; At least one target subject bounding box is selected from the plurality of candidate subject bounding boxes by sorting the bounding box sizes.

9. The method according to claim 1, wherein The method further includes: performing pattern recognition on the original image through edge detection processing to obtain the set of images to be cropped.

10. The method according to claim 9, wherein: The method of performing pattern recognition on the original image through edge detection to obtain the set of images to be cropped includes: Determining plane coordinate parameters of a plurality of edge points of each original picture in the original picture set; Determine polar coordinate parameters corresponding to the plane coordinate parameters through coordinate transformation; By detecting the straight line and based on the polar coordinate parameters, the layout of each original picture is identified to obtain the set of pictures to be cropped.

11. The method according to claim 1, wherein The method further comprises: determining a clipping region based on the at least one target subject bounding box; Perform image quality enhancement processing on the cropped area to obtain a cropped image.

12. The method according to claim 11, wherein The performing image quality enhancement processing on the cropped area to obtain a cropped image includes: Segmenting the candidate patch objects in the target image to obtain a plurality of candidate patch object segments; determining at least one candidate patch object segment existing in the clipping area; The cropped area is patched by replacing the at least one candidate patch object segment with a background image in the cropped area to obtain the cropped image.

13. An image processing method, comprising: Based on a preset first subject category priority, subject detection processing is performed on a plurality of video frame sets corresponding to the plurality of videos, to obtain a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories; determining, for a target video frame set among the multiple video frame sets, a target subject bounding box candidate set from the multiple subject bounding box candidate sets based on a second subject category priority, wherein the second subject category priority is obtained by adjusting the first subject category priority; Determine at least one target subject bounding box in the target subject bounding box candidate set to crop the cover of the target video from the target video frame set, wherein a size of the target subject bounding box has a first ratio in a cropped area, and the cropped area has a second ratio in the cover of the target video, the second ratio is determined in format recognition of the target video frame set, and the first ratio is determined based on the second ratio.

14. An image processing method, comprising: Presenting an operation interface to a user, wherein the operation interface presents multiple subject categories; In response to a selection instruction input by a user in the operation interface, determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories based on a second subject category priority, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on a to-be-cropped image set according to the plurality of subject categories based on a preset first subject category priority, and the second subject category priority is obtained by adjusting the first subject category priority; Determine at least one target subject bounding box in the target subject bounding box candidate set to crop a target image, wherein a size of the target subject bounding box has a first ratio in a cropping area, and the cropping area has a second ratio in the target image, the second ratio is determined in pattern recognition of the set of images to be cropped, and the first ratio is determined based on the second ratio.

15. The method according to claim 14, wherein The method further includes: obtaining a selection instruction input by a user in the operation interface, wherein the selection instruction indicates selection of a target subject; The step of responding to a selection instruction input by a user in the operation interface and determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories includes: A subject bounding box candidate set corresponding to the target subject is determined from the multiple subject bounding box candidate sets as the target subject bounding box candidate set.

16. The method according to claim 14, wherein The method further includes: obtaining a selection instruction input by a user in the operation interface, the selection instruction indicating a selection of a target subject category priority; The step of responding to a selection instruction input by a user in the operation interface and determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories includes: A subject bounding box candidate set that meets the priority of the target subject category among the multiple subject bounding box candidate sets is determined as the target subject bounding box candidate set.

17. The method according to claim 14, wherein: The determining at least one target subject bounding box in the target subject bounding box candidate set to crop the target image includes: In response to a subject bounding box selection instruction input by a user in the operation interface, at least one target subject bounding box in the target subject bounding box candidate set is determined to crop the target image.

18. The method according to claim 14, wherein The method further comprises: In the operation interface, the cropping result of the target image is presented to the user.

19. An image processing apparatus, comprising: A presentation module presents an operation interface to the user, wherein the operation interface presents multiple subject categories; a first determining module, in response to a selection instruction input by a user in the operation interface, determining a target subject bounding box candidate set from a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories based on a second subject category priority, wherein the plurality of subject bounding box candidate sets are obtained by performing subject detection processing on a to-be-cropped image set according to the plurality of subject categories based on a preset first subject category priority, and the second subject category priority is obtained by adjusting the first subject category priority; The second determination module determines at least one target subject bounding box in the target subject bounding box candidate set to crop the target image, wherein the size of the target subject bounding box has a first ratio in the cropping area, and the cropping area has a second ratio in the target image, the second ratio is determined in the template recognition of the set of images to be cropped, and the first ratio is determined based on the second ratio.

20. An image processing device, comprising: a subject detection processing module, which performs subject detection processing on a plurality of video frame sets corresponding to the plurality of videos based on a preset first subject category priority, and obtains a plurality of subject bounding box candidate sets corresponding to the plurality of subject categories; a first determining module, for a target video frame set among the multiple video frame sets, determining a target subject bounding box candidate set from the multiple subject bounding box candidate sets based on a second subject category priority, wherein the second subject category priority is obtained by adjusting the first subject category priority; The second determination module determines at least one target subject bounding box in the target subject bounding box candidate set to crop the cover of the target video from the target video frame set, wherein the size of the target subject bounding box has a first ratio in the cropping area, and the cropping area has a second ratio in the cover of the target video, the second ratio is determined in the layout recognition of the target video frame set, and the first ratio is determined based on the second ratio.

21. An image processing device, comprising: A subject detection processing module performs subject detection processing on the cropped image set based on a preset first subject category priority and according to multiple subject categories, thereby obtaining multiple subject bounding box candidate sets; a first determining module, for a target image in the to-be-cropped image set, determining a target subject bounding box candidate set from the plurality of subject bounding box candidate sets based on a second subject category priority, wherein the second subject category priority is obtained by adjusting the first subject category priority; The second determination module determines at least one target subject bounding box in the target subject bounding box candidate set to crop the target image, wherein the size of the target subject bounding box has a first ratio in the cropping area, and the cropping area has a second ratio in the target image, the second ratio is determined in the template recognition of the set of images to be cropped, and the first ratio is determined based on the second ratio.

22. An electronic device, comprising: one or more processors; A computer readable medium configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 18.

23. A computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

Images