Image processing methods, apparatus, devices and media

Abstract

This disclosure provides an image processing method, apparatus, device, and medium, relating to the field of artificial intelligence, specifically to computer vision, deep learning, and image processing, and applicable to scenarios such as advertising. The specific implementation of the image processing method is as follows: detecting the original image to obtain the target region where the target object is located in the original image; determining the cropping direction based on the aspect ratio of the original image; determining a pre-cropping region for the original image based on the center position of the target region, the cropping direction, and the target aspect ratio; and processing the original image based on the pre-cropping region to obtain a target image with the target aspect ratio.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to technical fields such as computer vision, deep learning, and image processing, and can be applied to scenarios such as advertising. Background Technology

[0002] With the development of computer and electronic technologies, various scenarios for image display have emerged. The required aspect ratio for image display may differ in different scenarios. To meet the needs of different scenarios, it is necessary to crop or augment the original image. Summary of the Invention

[0003] This disclosure aims to provide an image processing method, apparatus, device, and medium, such that in a target image of a target size ratio obtained through processing, the target object is positioned in the center.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising: detecting an original image to obtain a target region in the original image where a target object is located; determining a cropping direction based on the aspect ratio of the original image; determining a pre-cropping region for the original image based on the center position of the target region, the cropping direction, and the target aspect ratio; and processing the original image based on the pre-cropping region to obtain a target image with the target aspect ratio.

[0005] According to another aspect of this disclosure, an image processing apparatus is provided, comprising: an image detection module for detecting an original image and obtaining a target region in the original image where a target object is located; an orientation determination module for determining a cropping direction based on the size ratio of the original image; a region determination module for determining a pre-cropping region for the original image based on the center position of the target region, the cropping direction, and the target size ratio; and a first image processing module for processing the original image based on the pre-cropping region to obtain a target image with a target size ratio.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image processing method provided in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the image processing method provided in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the image processing method provided in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method and apparatus according to embodiments of the present disclosure;

[0012] Figure 2 This is a schematic flowchart of an image processing method according to an embodiment of the present disclosure;

[0013] Figure 3 This is a schematic flowchart of an image processing method according to another embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram illustrating the principle of determining the target area where the target object is located according to an embodiment of this disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the principle of determining the pre-cut area according to an embodiment of the present disclosure;

[0016] Figure 6 This is a schematic flowchart illustrating the processing of the original image based on a pre-cropped region according to an embodiment of the present disclosure.

[0017] Figure 7 This is a schematic diagram illustrating the principle of completing the original image according to an embodiment of the present disclosure;

[0018] Figure 8 This is a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure;

[0019] Figure 9 This is a block diagram of an electronic device used to implement the image processing method of the embodiments of this disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In advertising and other scenarios, it's often necessary to adjust the aspect ratio of the original image to match the actual scene. This can be done through direct stretching and scaling, one-sided completion, or cropping centered on the original image's center point. However, direct stretching and scaling can lead to image distortion. One-sided completion and cropping may result in the main subject of the original image being located at the edge of the adjusted image, causing important information to be obscured. Cropping may also result in incomplete representation of the main subject in the adjusted image.

[0022] To address this problem, this disclosure provides an image processing method, apparatus, device, and medium. The following will first describe these in conjunction with... Figure 1 The application scenarios of the methods and apparatus provided in this disclosure are described.

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method and apparatus according to embodiments of the present disclosure.

[0024] like Figure 1 As shown, the application scenario 100 of this embodiment may include a terminal device 110, which may be various electronic devices with processing functions, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0025] The terminal device 110 may have various client applications installed, such as image processing applications, instant messaging applications, shopping applications, etc. (for example only).

[0026] For example, the terminal device 110 can process the image 101, specifically by performing completion or cropping to adjust the size and proportions of the image 101, resulting in an adjusted image 102. The image 101 can be, for example, a real-time captured image, an image pre-captured and stored in the terminal device 110, or an image sent from another device. The adjusted image 102 can be used, for example, as an image for advertising, displayed on a webpage or in a client application.

[0027] In one embodiment, the application scenario 100 may further include a server 120, and the terminal device 110 may communicate with the server 120. For example, the server 120 may be a backend management server that supports the operation of client applications installed on the terminal device 110, or it may be a cloud server or a blockchain server, etc., and this disclosure does not limit it in this way.

[0028] In one embodiment, the terminal device 110 may also send the image 101 to the server 120, whereby the server 120 adjusts the size of the image 101 to obtain the adjusted image 102. Alternatively, the server 120 may adjust the size of a locally stored image to obtain the adjusted image. The server 120 may also send the adjusted image to the terminal device for display.

[0029] It should be noted that the image processing method provided in this disclosure can be executed by the terminal device 110 or by the server 120. Accordingly, the image processing apparatus provided in this disclosure can be located in the terminal device 110 or in the server 120.

[0030] It should be understood that Figure 1 The number and type of terminal devices 110 and servers 120 shown are merely illustrative. Depending on implementation needs, there can be any number and type of terminal devices 110 and servers 120.

[0031] The following will combine Figures 2-8 The image processing method provided in this disclosure is described in detail.

[0032] Figure 2 This is a schematic flowchart of an image processing method according to an embodiment of the present disclosure.

[0033] like Figure 2 As shown, the image processing method 200 of this embodiment may include operations S210 to S240.

[0034] In operation S210, the original image is detected to obtain the target region where the target object is located in the original image.

[0035] According to embodiments of this disclosure, a contour extraction algorithm can be used to extract the contours of each object in the original image, and the region enclosed by the largest contour can be taken as the target region where the target object is located. Alternatively, a subject detection algorithm can be used to detect the subject in the original image, and the position of the detected subject in the image can be taken as the target region where the target object is located.

[0036] The contour extraction algorithm may include Canny edge detection algorithm and threshold segmentation algorithm, etc., and this disclosure does not limit it. The subject detection algorithm may include: edge-focused salient object detection algorithm (Boundary-Aware Salient Object Detection, BASNet), pooling-based salient object detection algorithm (PoolNet), and U... 2 -Net, etc., are not limited in this disclosure.

[0037] In operation S220, the cropping direction is determined based on the size ratio of the original image.

[0038] According to embodiments of this disclosure, the direction in the original image that is larger in size relative to the target size ratio can be used as the cropping direction.

[0039] For example, when the original image's aspect ratio is smaller than the target image's aspect ratio, the cropping direction can be determined as a first direction; when the original image's aspect ratio is larger than the target image's aspect ratio, the cropping direction can be determined as a second direction. The aspect ratio of the original image can be the ratio of the size in the second direction to the size in the first direction. For example, if the aspect ratio is a width:height value, then the first direction is the height direction of the original image, and the second direction is the width direction of the original image. This embodiment determines the cropping direction based on the relationship between the original image's aspect ratio and the target image's aspect ratio, allowing the processed target image to retain more information from the original image.

[0040] It is understood that the above principle for determining the cropping direction is merely an example to facilitate understanding of this disclosure, and this disclosure does not limit it. For example, the larger of the height and width directions of the original image can also be used as the cropping direction.

[0041] In operation S230, the pre-cropping area for the original image is determined based on the center position of the target area, the cropping direction, and the target size ratio.

[0042] According to embodiments of this disclosure, the center of the target region can be used as the center of the pre-cropping region, and the region satisfying the target size ratio can be determined as the pre-cropping region. The size of the pre-cropping region in the direction perpendicular to the cropping direction can be equal to the size of the original image in that direction.

[0043] For example, in this embodiment, the center of the target region can be used as the center of the pre-cropping region. The dimensions of the original image in the cropping direction and in the direction perpendicular to the cropping direction can be determined. The dimension of the pre-cropping region in the cropping direction is then determined by taking twice the smaller of two distances from the center to the two edges of the original image in the cropping direction. Subsequently, based on this dimension in the cropping direction and the target size ratio, the dimension of the pre-cropping region in the direction perpendicular to the cropping direction is determined, thus obtaining the pre-cropping region. Alternatively, the dimension of the pre-cropping region in the direction perpendicular to the cropping direction can also be determined by taking twice the smaller of two distances from the center to the two edges of the original image in the direction perpendicular to the cropping direction.

[0044] In operation S240, the original image is processed according to the pre-cropped area to obtain the target image with the target size ratio.

[0045] According to embodiments of this disclosure, the original image can be cropped based on a pre-cropping region to obtain a target image. Alternatively, the original image can be processed based on the positional relationship between the pre-cropping region and the target region. For example, if the overlap between the pre-cropping region and the target region is large, the original image is cropped based on the pre-cropping region. If the overlap between the cropping region and the target region is small, the original image can be completed.

[0046] This embodiment of the disclosure determines the pre-cropping region based on the center of the target area where the detected target object is located and the cropping direction determined according to the size ratio of the original image. This ensures that the target object is located in the central region of the processed target image, while allowing the target image to retain as much information as possible from the original image. Therefore, when the target image is projected into a real-world scene for display, it can express information that better meets the actual needs, thus improving the projection effect.

[0047] Figure 3 This is a schematic flowchart of an image processing method according to another embodiment of the present disclosure.

[0048] According to embodiments of this disclosure, before detecting the original image, the difference between the aspect ratio of the original image and the target image can be compared to determine whether to directly crop the original image or to first detect the original image to determine the pre-cropping region and then process the original image based on the pre-cropping region. This improves image processing efficiency and avoids wasting computational resources. This is because if the aspect ratio of the original image is small compared to the target image, directly cropping the original image ensures that the target object in the original image is fully presented in the cropped image.

[0049] Specifically, such as Figure 3 As shown, the image processing method 300 of this embodiment includes operations S310 to S370. The implementation principles of operations S340 to S370 are similar to those of operations S210 to S240 described above, and will not be repeated here.

[0050] In operation S310, the absolute value of the difference between the size ratio of the original image and the target size ratio is determined.

[0051] For example, if the size ratio of the original image is 9:16 and the size ratio of the target image is 3:4, then the absolute value of the difference determined by operation S310 is |9 / 16-3 / 4|=0.1876.

[0052] In operation S320, it is determined whether the absolute value of the difference is greater than a predetermined difference. The predetermined difference can be set according to actual needs, for example, it can be set to a value close to 0, such as 0.05 or 0.1.

[0053] If the absolute value of the difference is less than or equal to the predetermined difference, then operation S330 is executed. If the absolute value of the difference is greater than the predetermined difference, then operations S340 to S370 are executed.

[0054] In operation S330, the original image is processed according to the target size ratio to obtain the target image.

[0055] For example, the center point of the original image can be used as the center point of the target image for cropping or augmentation. During cropping, only the width or height direction of the original image is cropped. For instance, if the original image size is smaller than the target size, cropping is performed in the first direction; if the original image size is larger than the target size, cropping is performed in the second direction. The original image size is the ratio of its size in the second direction to its size in the first direction. Similarly, during augmentation, augmentation can be performed only in the first or second direction.

[0056] In one embodiment, cropping can be performed symmetrically. Similarly, completion can be performed symmetrically. This ensures that the position of the main object in the target image corresponds to the position of the main object in the original image.

[0057] In operation S340, the original image is detected to obtain the target region where the target object is located in the original image.

[0058] When operating the S350, the cropping direction is determined based on the size ratio of the original image.

[0059] In operation S360, the pre-cropping area for the original image is determined based on the center position of the target area, the cropping direction, and the target size ratio.

[0060] In operation S370, the original image is processed according to the pre-cropped area to obtain the target image with the target size ratio.

[0061] Figure 4 This is a schematic diagram illustrating the principle of determining the target area where the target object is located according to an embodiment of this disclosure.

[0062] In one embodiment, the target object may include, for example, the main subject and text in the original image. When determining the target region, not only the region where the main subject is located but also the region where the text is located should be considered. This allows the target object to express richer and more comprehensive content of the original image, avoiding the loss of important information from the original image in the target image. This is because text in an image is usually descriptive of the main subject, and compared to the image of the main subject, text can usually highlight and express the main features of the main subject.

[0063] For example, such as Figure 4 As shown, in this embodiment 400, when determining the target region, the main object in the original image 410 can be detected first to obtain the first region 401 where the main object is located in the original image. Specifically, the first region 401 where the main object is located can be detected using a main object detection algorithm 420. At the same time, this embodiment detects the text in the original image 410 to obtain the second region 402 where the text is located in the original image. For example, the second region 402 can be detected using a text recognition algorithm 430. The main object detection algorithm 420 can be any of the main object detection algorithms described above. The text recognition algorithm 430 can include, for example, an optical character recognition algorithm (OCR) and a semantic segmentation-based text detection algorithm (e.g., a progressive size expansion network (PSENet)). This disclosure does not limit the specific algorithms used.

[0064] After obtaining the first region 401 and the second region 402, the target region 403 can be determined based on the first region 401 and the second region 402. For example, the circumscribed polygonal region surrounding the first region 401 and the second region 402 can be used as the target region 403. The circumscribed polygonal region can be a circumscribed rectangular region, but this disclosure does not limit it to this.

[0065] For example, if both the first region 401 and the second region 402 are rectangular regions, in an image coordinate system constructed with the top-left vertex of the original image as the origin and two axes parallel to and parallel to the height direction as coordinate axes, the minimum horizontal axis coordinate value among the eight vertices included in the first and second regions can be used as the horizontal axis coordinate value of the top-left vertex of the target region 403, the maximum horizontal axis coordinate value among the eight vertices can be used as the horizontal axis coordinate value of the bottom-right vertex of the target region 403, the minimum vertical axis coordinate value among the eight vertices can be used as the vertical axis coordinate value of the top-left vertex of the target region 403, and the maximum vertical axis coordinate value among the eight vertices can be used as the vertical axis coordinate value of the bottom-right vertex of the target region 403. Based on the horizontal and vertical axis coordinate values ​​of the top-left and bottom-right vertices, a rectangular region can be uniquely determined, which is the target region 403.

[0066] According to embodiments of this disclosure, after obtaining the target region, the coordinates of the center position of the target region in the image coordinate system can be determined based on the coordinates of the top left corner vertex and the bottom right corner vertex in the image coordinate system.

[0067] Once the center position and cutting direction are determined, the pre-cutting area can be identified. For example... Figure 5 This is a schematic diagram illustrating the principle of determining the pre-cut area according to an embodiment of the present disclosure.

[0068] like Figure 5 As shown, in one embodiment 500, when determining the pre-cropping area, the target size b 530 of the pre-cropping area in the direction perpendicular to the cropping direction can be determined first based on the target size ratio 510 and the size a 520 of the original image in the direction perpendicular to the cropping direction. For example, if the cropping direction is the height direction, then the size a 520 is the size in the width direction. Setting the target size ratio as the width:height ratio, the target size b 530 can be calculated using the following formula: Size b = Size a / Size ratio.

[0069] After obtaining the target size b 530, the pre-cutting area 550 can be determined based on the target size b 530 and the center position 540 of the target area.

[0070] For example, if the cropping direction is set to the height direction, this embodiment can use the size of the original image in the width direction as the size of the pre-cropping area 550 in the width direction. Then, starting from the center position 540, the cropping extends upwards and downwards along the height direction until half of the dimension b has been extended both upwards and downwards, thus obtaining the pre-cropping area 550. For example, if, during the upward and downward extension, the upward (or downward) extension reaches the upper (or lower) boundary of the original image before reaching half of dimension b, then the upward extension stops, and the downward (or upward) extension should reach the following size: half of dimension b + (half of dimension b - vertical distance from the center position to the upper (lower) boundary of the original image).

[0071] The pre-cropping region 550 determined by this embodiment contains as much image information as possible from the original image, which helps to improve the overlap rate between the information expressed by the target image processed according to the pre-cropping region 550 and the information expressed by the original image, so that the projection of the target image can achieve the expected effect.

[0072] Figure 6 This is a schematic diagram illustrating the process of processing the original image according to a pre-cropped region according to an embodiment of the present disclosure.

[0073] According to embodiments of this disclosure, after determining the pre-cropping region, for example, the overlap relationship between the pre-cropping region and the target region can be determined first. Subsequently, a processing strategy for the original image is determined based on the overlap relationship.

[0074] For example, such as Figure 6 As shown in this embodiment 600, when implementing operation S220 described above, the inclusion relationship between the pre-cropped region and the target region can be determined first based on the position of the pre-cropped region and the position of the target region. Specifically, operation S641 can be executed to determine whether the pre-cropped region contains the target region. If the pre-cropped region contains the target region, then operation S642 is executed to crop the image of the pre-cropped region in the original image to obtain the target image.

[0075] Alternatively, in this embodiment 600, when implementing operation S220 described above, the intersection-over-union ratio (IoU) between the pre-cropped region and the target region can be determined first based on the positions of the pre-cropped region and the target region, i.e., operation S643 can be executed. Then, operation S644 is executed to determine whether the IoU is greater than or equal to a predetermined IoU threshold. If it is greater than or equal to the predetermined IoU threshold, operation S642 is executed to crop the image of the pre-cropped region in the original image to obtain the target image. The predetermined IoU threshold can be set according to actual needs; for example, it can be set to any value close to 1 and less than 1, such as 0.95 or 0.97. This disclosure does not limit this value.

[0076] Alternatively, in this embodiment 600, operations S641 and S643 to S644 can be executed simultaneously. When operation S641 determines that the pre-cropped region contains the target region or operation S644 determines that the cross-union ratio between the pre-cropped region and the target region is greater than or equal to a predetermined cross-union ratio threshold, operation S642 is executed.

[0077] This embodiment ensures that the cropped target image contains complete information about the main object in the original image by cropping the original image only when the pre-cropped region contains the target region or when the intersection-over-union (IoU) ratio between the pre-cropped region and the target region is greater than or equal to a predetermined IoU threshold. This improves the target image's ability to represent the original image and enhances the projection effect of the target image.

[0078] In this embodiment 600, the original image can be further augmented to obtain the target image when the pre-cropped region does not contain the target region or the intersection-over-union (IoU) ratio between the pre-cropped region and the target region is less than a predetermined IoU threshold. Specifically, when operation S641 determines that the pre-cropped region does not contain the target region and operation S644 determines that the IoU ratio between the pre-cropped region and the target region is less than the predetermined IoU threshold, operation S645 is executed to augment the original image and obtain the target image. This embodiment, through operation S645, can further ensure that the obtained target image contains complete information about the main object in the original image. This improves the expressive power of the target image over the original image and enhances the projection effect of the target image.

[0079] For example, if implementation 600 executes operation S641 first, it can execute operations S643 to S644 when it is determined that the pre-cropped area does not contain the target area. Alternatively, if this embodiment executes operations S643 to S644 first, it can execute operation S641 when it is determined that the cross-union ratio is less than a predetermined cross-union ratio threshold.

[0080] Figure 7 This is a schematic diagram illustrating the principle of completing the original image according to an embodiment of the present disclosure.

[0081] According to embodiments of this disclosure, when it is necessary to perform completion processing on the original image, for example, the target direction to be completed can be determined first, and then completion processing can be performed in the target direction. The principle of determining the target direction is similar to the principle of determining the cropping direction in operation S330 described above.

[0082] For example, the target direction for padding in the original image can be determined based on the relationship between the original image's aspect ratio and the target image's aspect ratio. Specifically, when the original image's aspect ratio is larger than the target image's aspect ratio, the target direction for padding is determined as the first direction. When the original image's aspect ratio is smaller than the target image's aspect ratio, the target direction for padding is determined as the second direction. The aspect ratio of the original image is determined by the ratio between the original image's size in the second direction and its size in the first direction. For example, the first direction could be the height direction, and the second direction could be the width direction. In this way, padding can be performed on the directions in the original image that are smaller in proportion to the target image's height and width.

[0083] For example, when performing completion processing, the original image can be completed based on predetermined pixel values ​​or image blurring algorithms.

[0084] In one embodiment, after determining the target direction, the original image can be completed using symmetrical completion. This ensures that the original image portion is located in the central region of the completed target image, allowing the target image to prominently display the target object in the original image. Specifically, when the original image is determined to be smaller in proportion to the target size in a first direction, the target direction can be defined as two opposite directions parallel to the first direction. For example, if the first direction is the width direction, the target direction includes the left and right directions. When the original image is determined to be smaller in proportion to the target size in a second direction, the target direction can be defined as two opposite directions parallel to the second direction. For example, if the second direction is the height direction, the target direction includes the up and down directions.

[0085] like Figure 7 As shown, in one embodiment 700, the original image can be completed by operating S741 to S747. In this embodiment, the size ratio of the original image is set to the ratio of width to height.

[0086] In operation S741, it is determined whether the size ratio of the original image, which is not equal to the target size ratio, is greater than the target size ratio. If it is less, operation S742 is executed; if it is greater, operation S743 is executed.

[0087] In operation S742, the target direction is determined to include two opposite directions in the width direction.

[0088] When operating S743, the target direction is determined, including two opposite directions in the altitude direction.

[0089] After determining the target direction, operation S744 can be executed to determine the pixel values ​​of the edge region of the original image in the target direction with a predetermined size. The predetermined size can be set according to actual needs. For example, if the target direction is two opposite directions in the width direction, the edge region can be the region containing a predetermined number of pixel columns arranged along the height direction in the two opposite directions. If the target direction is two opposite directions in the height direction, the edge region can be the region containing a predetermined number of pixel rows arranged along the width direction in the two opposite directions. The predetermined number can be, for example, a natural number greater than 1 such as 3 or 5, and this disclosure does not limit it.

[0090] After determining the pixel values ​​of the edge regions, operations S745 to S746 can be performed to complete the original image based on the dominant color when a dominant color exists in the edge regions. This makes the resulting target image more closely resemble the original image, resulting in a more aesthetically pleasing and realistic completed target image.

[0091] In operation S745, it is determined whether the proportion of pixels with the same pixel value in the edge region is greater than or equal to a predetermined proportion. If so, operation S746 is executed.

[0092] In operation S746, the original image is completed based on the same pixels in the target direction. It can be understood that when the target direction includes two opposing directions, operations S744 to S746 can be performed for each direction.

[0093] According to embodiments of this disclosure, when the proportion of pixels with the same pixel value in the determined edge region is less than a predetermined proportion, the original image can be completed, for example, based on the predetermined pixel value. Alternatively, an image blurring algorithm can be used to complete the original image, i.e., operation S747 is performed. The image blurring algorithm may include, for example, Gaussian blurring, mean blurring, etc., and an appropriate image blurring algorithm can be selected according to actual needs; this disclosure does not limit this. By using an image blurring algorithm to complete the original image, the completed target image can be made more aesthetically pleasing, reducing any sense of incongruity.

[0094] Based on the image processing method provided in this disclosure, this disclosure also provides an image processing apparatus, which will be described below in conjunction with... Figure 8 The device is described in detail.

[0095] Figure 8 This is a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0096] like Figure 8As shown, the image processing apparatus 800 of this embodiment may include an image detection module 810, a direction determination module 820, a region determination module 830, and a first image processing module 840.

[0097] The image detection module 810 is used to detect the original image and obtain the target region where the target object is located in the original image. In one embodiment, the image detection module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0098] The orientation determination module 820 is used to determine the cropping direction based on the aspect ratio of the original image. In one embodiment, the orientation determination module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0099] The region determination module 830 is used to determine a pre-cropping region for the original image based on the center position of the target region, the cropping direction, and the target size ratio. In one embodiment, the region determination module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0100] The first image processing module 840 is used to process the original image according to the pre-cropped area to obtain a target image with a target size ratio. In one embodiment, the first image processing module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0101] According to embodiments of this disclosure, the first image processing module 840 may include an intersection-over-union (IoU) determination submodule and a cropping submodule. The IoU determination submodule is used to determine the IoU between the pre-cropped region and the target region. The cropping submodule is used to crop the pre-cropped region of the original image in response to an IoU greater than or equal to a predetermined IoU threshold, thereby obtaining the target image.

[0102] According to embodiments of this disclosure, the first image processing module 840 may include a relationship determination submodule and a cropping submodule. The relationship determination submodule is used to determine the inclusion relationship between the pre-cropped region and the target region based on the positions of the pre-cropped region and the target region. The cropping submodule is used to crop the image of the pre-cropped region in the original image in response to the pre-cropped region including the target region, thereby obtaining the target image.

[0103] According to an embodiment of this disclosure, the first image processing module 840 may include a completion submodule, configured to perform completion processing on the original image to obtain the target image in response to the intersection-union ratio between the pre-cropped region and the target region being less than a predetermined intersection-union ratio threshold and the pre-cropped region not containing the target region.

[0104] According to embodiments of this disclosure, the completion submodule may include a target direction determination unit and a completion processing unit. The target direction determination unit determines the target direction in the original image that needs to be completed based on the size relationship between the original image's aspect ratio and the target image's aspect ratio. The completion processing unit performs completion processing on the original image based on the target direction.

[0105] According to embodiments of this disclosure, the target direction determination unit may include a first determination subunit and a second determination subunit. The first determination subunit is configured to determine, in response to a size ratio of the original image being greater than a target size ratio, that the target direction includes two opposite directions parallel to a first direction. The second determination subunit is configured to determine, in response to a size ratio of the original image being less than the target size ratio, that the target direction includes two opposite directions parallel to a second direction; the second direction is perpendicular to the first direction. The size ratio of the original image is determined based on the ratio between the size of the original image in the second direction and the size of the original image in the first direction.

[0106] According to embodiments of this disclosure, the completion processing unit may include a pixel value determination subunit and a first completion subunit. The pixel value determination subunit is used to determine the pixel values ​​of an edge region of a predetermined size in the target direction of the original image. The first completion subunit is used to perform completion processing on the original image in the target direction based on the same pixel values, in response to determining that the proportion of pixels with the same pixel value in the edge region is greater than or equal to a predetermined proportion.

[0107] According to an embodiment of this disclosure, the above-mentioned completion processing unit further includes a second completion subunit, which is used to complete the original image in the target direction by employing an image blurring algorithm in response to determining that the proportion of pixels with the same pixel value in the edge region is less than a predetermined proportion based on the pixel value.

[0108] According to embodiments of this disclosure, the region determination module 830 may include a target size determination submodule and a region determination submodule. The target size determination submodule is used to determine the target size of the pre-cropping region in the direction perpendicular to the cropping direction based on the target size ratio and the size of the original image in the direction perpendicular to the cropping direction. The region determination submodule is used to determine the pre-cropping region based on the center position and the target size.

[0109] According to embodiments of this disclosure, the orientation determination module 820 may include a first determination submodule and a second determination submodule. The first determination submodule is configured to determine a cropping direction including a first direction in response to a situation where the aspect ratio of the original image is smaller than the target aspect ratio. The second determination submodule is configured to determine a cropping direction including a second direction, perpendicular to the first direction, in response to a situation where the aspect ratio of the original image is larger than the target aspect ratio. The aspect ratio of the original image is determined based on the ratio between the size of the original image in the second direction and the size of the original image in the first direction.

[0110] According to embodiments of this disclosure, the image detection module 810 may include a subject detection submodule, a text detection submodule, and a region determination submodule. The subject detection submodule detects the main object in the original image and obtains a first region where the main object is located in the original image. The text detection submodule detects text in the original image and obtains a second region where the text is located in the original image. The region determination submodule determines a target region based on the first and second regions.

[0111] According to an embodiment of this disclosure, the image detection module 810 is specifically used to: detect the original image in response to the absolute value of the difference between the size ratio of the original image and the target size ratio being greater than a predetermined difference, and obtain the target region where the target object is located in the original image.

[0112] According to embodiments of this disclosure, the image processing apparatus 800 may further include a second image processing module, configured to process the original image according to the target size ratio in response to the absolute value of the difference between the size ratio of the original image and the target size ratio being less than or equal to a predetermined difference, thereby obtaining a target image.

[0113] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information in this disclosed technical solution comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. In this disclosed technical solution, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0114] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0115] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement the image processing methods of embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0116] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0117] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0118] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0123] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0124] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0125] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: In response to the absolute value of the difference between the size ratio of the original image and the target size ratio being greater than a predetermined difference, the main object and text region in the original image are detected to obtain the target region where the target object is located in the original image, wherein the original image is an image of an advertising scenario and the target size ratio is a size ratio adapted to the advertising scenario; The cropping direction is determined based on the aspect ratio of the original image; Starting from the center of the target area, extend to both sides along the cropping direction to determine the area that meets the target size ratio as the pre-cropping area. If the extension to one side touches the boundary of the original image before reaching half of the target size, then stop extending to that side and compensate for the corresponding difference in the extension to the other side. The original image is processed according to the pre-cropped region to obtain a target image with the target size ratio, which is used as an image for advertising. This includes: Determine the intersection-union ratio (IUU) between the pre-trimmed region and the target region; The inclusion relationship between the pre-cropped region and the target region is determined based on the position of the pre-cropped region and the position of the target region; In response to the intersection-over-union ratio being greater than or equal to a predetermined intersection-over-union threshold, or the pre-cropped region containing the target region, the image of the pre-cropped region in the original image is cropped to obtain the target image; In response to the fact that the intersection-union ratio between the pre-cropped region and the target region is less than a predetermined intersection-union ratio threshold, and the pre-cropped region does not contain the target region, the original image is completed to obtain the target image.

2. The method according to claim 1, wherein, The original image is augmented to obtain the target image, which includes: Based on the size relationship between the original image and the target image, determine the target direction that needs to be filled in the original image; and The original image is completed according to the target direction.

3. The method according to claim 2, wherein, The step of determining the target direction that needs to be filled in the original image based on the size relationship between the original image size ratio and the target size ratio includes: In response to the original image's aspect ratio being greater than the target image's aspect ratio, the target direction is determined to include two opposite directions parallel to the first direction; and In response to the original image's aspect ratio being smaller than the target aspect ratio, the target direction is determined to include two opposite directions parallel to the second direction; the second direction is perpendicular to the first direction. The aspect ratio of the original image is determined based on the ratio between the size of the original image in the second direction and the size of the original image in the first direction.

4. The method according to claim 2, wherein, The step of completing the original image according to the target direction includes: Determine the pixel values ​​of the edge region of the original image of a predetermined size in the target direction; and In response to determining that the proportion of pixels with the same pixel value in the edge region is greater than or equal to a predetermined proportion based on the pixel value, the original image is completed in the target direction based on the same pixel value.

5. The method according to claim 4, wherein, The step of completing the original image according to the target direction further includes: In response to determining that the proportion of pixels with the same pixel value in the edge region is less than a predetermined proportion based on the pixel value, an image blurring algorithm is used to complete the original image in the target direction.

6. The method according to claim 1, wherein, The step of determining the region that meets the target size ratio as the pre-cutting region includes: Based on the target size ratio and the size of the original image in the direction perpendicular to the cropping direction, determine the target size of the pre-cropping region in the direction perpendicular to the cropping direction; and The pre-cutting area is determined based on the center position and the target size.

7. The method according to claim 1, wherein, Determining the cropping direction based on the aspect ratio of the original image includes: In response to the original image's aspect ratio being smaller than the target aspect ratio, the cropping direction is determined to include a first direction; and In response to the original image's aspect ratio being larger than the target image's aspect ratio, the cropping direction is determined to include a second direction, the second direction being perpendicular to the first direction. The aspect ratio of the original image is determined based on the ratio between the size of the original image in the second direction and the size of the original image in the first direction.

8. The method according to claim 1, wherein, The process of detecting the main object and text regions in the original image to obtain the target region where the target object is located in the original image includes: Detect the main object in the original image to obtain a first region where the main object is located in the original image; Detecting text in the original image to obtain a second region containing the text in the original image; and The target area is determined based on the first area and the second area.

9. The method according to claim 1, further comprising: In response to the absolute value of the difference between the size ratio of the original image and the target size ratio being less than or equal to the predetermined difference, the original image is processed according to the target size ratio to obtain the target image.

10. An image processing apparatus, comprising: The image detection module is used to detect the main object and text region in the original image in response to the absolute value of the difference between the size ratio of the original image and the target size ratio being greater than a predetermined difference, and to obtain the target region where the target object is located in the original image, wherein the original image is an image of an advertising scenario, and the target size ratio is a size ratio adapted to the advertising scenario; An orientation determination module is used to determine the cropping direction based on the size ratio of the original image; The region determination module is used to determine regions that meet the target size ratio as pre-cropping regions, starting from the center position of the target region and extending to both sides along the cropping direction. If, while extending to one side, the module touches the boundary of the original image before reaching half the target size, then the extension to that side is stopped, and the corresponding difference is compensated by extending to the other side. The first image processing module is used to process the original image according to the pre-cropped area to obtain a target image with the target size ratio, which is used as an image for advertising. The first image processing module includes: The intersection-union ratio (IUGR) determination submodule is used to determine the IUGR between the pre-cropped region and the target region. The relationship determination submodule is used to determine the inclusion relationship between the pre-cropped region and the target region based on the position of the pre-cropped region and the position of the target region; A cropping submodule is configured to crop the image of the pre-cropped region in the original image to obtain the target image in response to the intersection-to-union ratio being greater than or equal to a predetermined intersection-to-union ratio threshold, or the pre-cropped region containing the target region; The completion submodule is used to complete the original image in response to the fact that the intersection-union ratio between the pre-cropped region and the target region is less than a predetermined intersection-union ratio threshold, and the pre-cropped region does not contain the target region, to obtain the target image.

11. The apparatus according to claim 10, wherein, The completion submodule includes: A target orientation determination unit is used to determine the target orientation that needs to be filled in the original image based on the size relationship between the original image's aspect ratio and the target image's aspect ratio; and The completion processing unit is used to complete the original image according to the target direction.

12. The apparatus according to claim 11, wherein, The target direction determination unit includes: A first determining subunit is configured to, in response to the original image's aspect ratio being greater than the target aspect ratio, determine that the target direction includes two opposite directions parallel to the first direction; and The second determining subunit is configured to, in response to the original image's aspect ratio being smaller than the target aspect ratio, determine that the target direction includes two opposite directions parallel to the second direction; the second direction is perpendicular to the first direction. The aspect ratio of the original image is determined based on the ratio between the size of the original image in the second direction and the size of the original image in the first direction.

13. The apparatus according to claim 11, wherein, The completion processing unit includes: A pixel value determination subunit is configured to determine the pixel values ​​of an edge region of a predetermined size in the target direction of the original image; and The first completion subunit is configured to, in response to determining that the proportion of pixels with the same pixel value in the edge region is greater than or equal to a predetermined proportion based on the pixel value, perform completion processing on the original image in the target direction based on the same pixel value.

14. The apparatus according to claim 13, wherein, The completion processing unit further includes: The second completion subunit is used to complete the original image in the target direction by employing an image blurring algorithm in response to determining that the proportion of pixels with the same pixel value in the edge region is less than the predetermined proportion based on the pixel value.

15. The apparatus according to claim 10, wherein, The region determination module includes: The target size determination submodule is used to determine the target size of the pre-cropping region in the direction perpendicular to the cropping direction based on the target size ratio and the size of the original image in the direction perpendicular to the cropping direction; and The region determination submodule is used to determine the pre-cutting region based on the center position and the target size.

16. The apparatus according to claim 10, wherein, The direction determination module includes: A first determining submodule is configured to, in response to the original image's size ratio being smaller than the target size ratio, determine that the cropping direction includes a first direction; and The second determining submodule is configured to, in response to the original image's aspect ratio being greater than the target image's aspect ratio, determine that the cropping direction includes a second direction, wherein the second direction is perpendicular to the first direction. The aspect ratio of the original image is determined based on the ratio between the size of the original image in the second direction and the size of the original image in the first direction.

17. The apparatus according to claim 10, wherein, The image detection module includes: The subject detection submodule is used to detect the subject object in the original image and obtain the first region where the subject object is located in the original image; A text detection submodule is used to detect text in the original image and obtain a second region in the original image where the text is located; and The region determination submodule is used to determine the target region based on the first region and the second region.

18. The apparatus of claim 10, further comprising: The second image processing module is configured to process the original image according to the target size ratio in response to the absolute value of the difference between the size ratio of the original image and the target size ratio being less than or equal to the predetermined difference, thereby obtaining the target image.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images