Image display area extraction method and related device

Abstract

The embodiment of the present application provides a method for extracting the display area of ​​an image and a related device, the method comprising: generating a target salient area assignment map including salient areas and non-salient areas based on the original image, wherein the pixel assignment values ​​of the pixels in the salient area are higher than the pixel assignment values ​​of the pixels in the non-salient area; constructing a convolution kernel according to the display ratio of the display area; using the convolution kernel to move multiple times on the target salient area assignment map, and after each movement, determining the convolution result of the convolution kernel and the area where the convolution kernel currently stays on the target salient area assignment map as the recommendation index of the area; determining the target area in the target salient area assignment map based on the recommendation index of each area, and extracting the display area from the original image based on the position of the target area in the target salient area assignment map. The embodiment of the present application improves the display effect. The embodiment of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for extracting a display area of ​​an image and a related device. Background Art

[0002] Applications that provide information streams need to display thumbnails. In this scenario, the dimensions of the thumbnails may differ from those of the actual images. Displaying the actual images directly without cropping them will result in image distortion, resulting in poor display quality.

[0003] Currently, there are some methods for cropping images. However, the cropping areas determined by these methods are often not accurate enough, which can also lead to poor display effects. Summary of the Invention

[0004] The embodiments of the present application provide a method and related device for extracting the display area of ​​an image, which can effectively avoid problems such as poor display effects caused by image deformation, poor image cropping areas, etc., at least to a certain extent.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a method for extracting a display area of ​​an image is provided, the method comprising: generating a target salient area assignment map including a salient area and a non-salient area based on an original image, the pixel assignment values ​​of pixels in the salient area being higher than the pixel assignment values ​​of pixels in the non-salient area; constructing a convolution kernel based on a display ratio of the display area, wherein the convolution kernel comprises multiple layers of areas distributed from inside to outside, and the weights of elements in each layer of areas are smaller than the weights of elements in the inner layer of each layer of areas; using the convolution kernel to move multiple times on the target salient area assignment map, and after each movement, determining a convolution result of the convolution kernel and an area where the convolution kernel currently stays on the target salient area assignment map as a recommendation index of the area; determining a target area in the target salient area assignment map based on the recommendation index of each area, and extracting a display area from the original image based on the position of the target area in the target salient area assignment map.

[0007] According to one aspect of an embodiment of the present application, a display area extraction device for an image is provided, the device comprising: a generation unit for generating a target salient area assignment map including a salient area and a non-salient area based on an original image, wherein the pixel assignment value of a pixel point in the salient area is higher than the pixel assignment value of a pixel point in the non-salient area; a construction unit for constructing a convolution kernel according to a display ratio of the display area, wherein the convolution kernel comprises multiple layers of areas distributed from the inside to the outside, and the weights of elements in each layer of areas are smaller than the weights of elements in the inner layer of each layer of areas; a convolution unit for using the convolution kernel to move multiple times on the target salient area assignment map, and after each movement, determining a convolution result of the convolution kernel and the area where the convolution kernel currently stays on the target salient area assignment map as a recommendation index of the area; a determination and extraction unit for determining a target area in the target salient area assignment map according to the recommendation index of each area, and extracting a display area from the original image according to the position of the target area in the target salient area assignment map.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the convolution kernel includes a central area, a middle area located outside the central area, and an edge area located outside the middle area. The weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the middle area are greater than the weights of the elements in the edge area.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the generation unit is configured to: input the original image into the face detection model to obtain the original marked image with the face area marked output by the face detection model; and obtain the target salient area assignment map by assigning values ​​to the pixel points in the face area and other areas in the original marked image respectively.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the generation unit is configured to: assign values ​​to the pixel points in the face area and other areas in the original marked image respectively to obtain a first salient area assignment map; process the original image according to a predetermined salient area detection algorithm to obtain a second salient area assignment map; and perform a fusion operation on the first salient area assignment map and the second salient area assignment map to obtain a target salient area assignment map.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the generation unit is configured to: perform weighted processing on the pixel values ​​of the pixel points in the face area of ​​the first salient area assignment map by a weighting coefficient to obtain a weighted first salient area assignment map; superimpose the weighted first salient area assignment map with the pixel values ​​of the corresponding pixel points in the second salient area assignment map to obtain a target salient area assignment map.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the determination and extraction unit is configured to: determine the maximum recommendation index among the recommendation indexes of each region; and use the region corresponding to the maximum recommendation index as the target region in the target salient region assignment map.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the determination and extraction unit is configured to: determine multiple candidate regions according to the recommendation index of each region; and determine the target region in the target salient region assignment map from the multiple candidate regions.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the determination and extraction unit is configured to: repeatedly perform the candidate area selection step until the number of selected candidate areas reaches a predetermined number; wherein the candidate area selection step includes: determining the maximum recommendation index among the recommendation indexes of the unselected areas, and selecting the area corresponding to the maximum recommendation index as the candidate area; setting the recommendation index of the neighboring area corresponding to the candidate area to zero, and the distance between the center point of the neighboring area and the center point of the candidate area is less than the predetermined distance.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the middle area includes four middle areas and four corner areas, the elements in the middle area are in the same row or column as the elements in the central area, and the elements in the corner area are in different rows and columns from the elements in the central area; the weights of the elements in the edge area are negative, the weights of the elements in the central area and the middle area are positive, the weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the central area are greater than the weights of the elements in the corner area.

[0016] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for extracting the display area of ​​a picture as described in the above embodiment is implemented.

[0017] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for extracting the display area of ​​a picture as described in the above embodiment.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the display area extraction method of the picture as described in the above embodiment.

[0019] In the technical solutions provided in some embodiments of the present application, a target salient region assignment map including salient regions and non-salient regions is first generated based on the original image, and the pixel assignment values ​​of the pixels in the salient region are made higher than the pixel assignment values ​​of the pixels in the non-salient region, and then a convolution kernel is constructed according to the display ratio of the display area; then, a sliding window convolution operation is performed on the target salient region assignment map using the convolution kernel to obtain a recommendation index for each region where the convolution kernel stops in the target salient region assignment map; finally, the target region can be determined in the target salient region assignment map based on the recommendation index of each region, and the corresponding display region in the original image can be extracted accordingly. Therefore, in the solution of the embodiment of the present application, on the one hand, since the convolution kernel is constructed according to the display ratio of the display area, the extracted display area will not be deformed; on the other hand, since the weights of each element of the convolution kernel increase from the outside to the inside, the target area obtained by the convolution operation based on the convolution kernel will contain more salient regions and will not separate the salient regions, thereby being able to more accurately extract the display area, improving the display effect and thus improving the user experience.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0022] Figure 1 A schematic diagram showing the principle of image cropping in related art is shown;

[0023] Figure 2 A schematic diagram showing an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied;

[0024] Figure 3A schematic diagram showing an application scenario of a method for extracting a display area of ​​a picture according to an embodiment of the present application is shown;

[0025] Figure 4 A flowchart of a method for extracting a display area of ​​a picture according to an embodiment of the present application is shown;

[0026] Figure 5 An embodiment according to the present application is shown Figure 4 A flowchart of the details of step 410 in an embodiment;

[0027] Figure 6 A flowchart of obtaining a target salient area value map by assigning values ​​to pixels in an original marked image according to an embodiment of the present application is shown;

[0028] Figure 7 A schematic diagram showing the principle of fusing a first salient region value assignment map and a second salient region value assignment map according to an embodiment of the present application is shown;

[0029] Figure 8 A schematic diagram of a convolution kernel according to an embodiment of the present application is shown;

[0030] Figure 9 A schematic diagram illustrating the principle of performing a convolution operation on a target salient region assignment map using a convolution kernel according to an embodiment of the present application is shown;

[0031] Figure 10 An embodiment according to the present application is shown Figure 4 A flowchart of the details of step 440 in the embodiment;

[0032] Figure 11 Another embodiment according to the present application is shown Figure 4 A flowchart of the details of step 440 in the embodiment;

[0033] Figure 12 shows a schematic diagram of the overall process according to an embodiment of the present application;

[0034] Figure 13 A block diagram of a device for extracting a display area of ​​a picture according to an embodiment of the present application is shown;

[0035] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0040] Image cropping is a common method in image processing. The goal of image cropping is to extract important or eye-catching areas from an image for display. For example, for an image containing a person, the goal of cropping the image can be to extract the area containing the person's face.

[0041] In related technologies, there are mainly two ways to crop images:

[0042] Method 1: crop the center area of ​​the image.

[0043] In this way, the central area of ​​the image will be directly extracted as the display area of ​​the image.

[0044] Method 2: cropping the center of the significant area.

[0045] This method takes into account the salient areas of the image. Salient areas are areas of interest or importance identified through an algorithm. Specifically, this method first determines the salient areas of the image, then uses a predefined dilation algorithm to calculate the center position of all salient areas. The image is then cropped based on this center position. Method 2 delivers superior results to Method 1.

[0046] Figure 1 Schematic diagram showing the principle of image cropping in related technology. Figure 1 As shown, the two images below are the images to be cropped. The white areas in the upper left image are the two salient regions obtained by performing face detection on the image below. Each salient region corresponds to the area where a face is located in the image below. Therefore, the salient regions here can also be called original face feature maps. The center point of each original face feature map corresponds to the corresponding position in the image. For example, the center point of the original face feature map on the left corresponds to the eye position of the person on the left in the image. This position can be called the original corresponding center.

[0047] After obtaining the two original facial feature maps, all original facial feature maps are expanded using a predetermined expansion algorithm to obtain expanded facial feature maps. The white area in the upper right corner of the image is the expanded facial feature map, and the black dot within the white area is the center point of the expanded facial feature map. The corresponding position of the center point of the expanded facial feature map in the image is shown in the lower right corner of the image. As can be seen, this position is located between the two faces and can be referred to as the new corresponding center. When cropping the image, the new corresponding center can be used as the center point and cropped according to the desired size of the cropped area.

[0048] However, both of the above two methods have certain defects.

[0049] The drawback of this method of cropping the center area of ​​the image is that it directly and violently uses the center area of ​​the image. Different images often differ in composition and layout, so this method ignores the differences in the image itself and often does not achieve the best cropping effect.

[0050] The drawback of the salient region center cropping method is that when faced with images with scattered salient regions, the expanded regions may not be connected. In this case, this method will extract the largest salient region, while other salient regions will be ignored. Therefore, this method is difficult to achieve good results in many cases.

[0051] The inventors discovered that in order to extract the best display area of ​​an image, the following points need to be achieved:

[0052] 1. Selecting the best display area depends on the salient areas of the image, and the display area of ​​the image needs to include as many salient areas as possible.

[0053] 2. Because the center of the image is more likely to attract people's attention than the edges, large prominent areas need to appear in the center of the display area as much as possible.

[0054] 3. The display area will not split large prominent areas. For example, the edge of the display area should not divide the same face into two halves.

[0055] To this end, this application first provides a method for extracting the display area of ​​an image. This method actually involves cropping the image to obtain the display area. The method for extracting the display area of ​​an image provided in the embodiments of this application can overcome the above-mentioned drawbacks. It can not only include more salient areas, but also prevent the same salient area from being split by the extracted display area, thereby achieving a better display effect for the extracted display area.

[0056] Figure 2 A schematic diagram shows an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

[0057] like Figure 2 As shown, the system architecture 200 includes a user access terminal, a server 220 and a content upload terminal 230. The user access terminal includes a first user access terminal 211, a second user access terminal 212 and a third user access terminal 213. Each user access terminal and the content upload terminal 230 establishes a communication connection with the server 220 through a network. Each user access terminal and the content upload terminal 230 can send information to the server 220 and can also receive information from the server 220. An access client is deployed on each user access terminal. The access client can be an APP in various information flow forms such as a news APP. A content upload client is deployed on the content upload terminal 230, and a server that provides services for the access client and the content upload client is deployed on the server 220. In the embodiment of the present application, the server 220 is the implementation terminal. When the display area extraction method of the picture provided in the present application is applied to Figure 2In the system architecture shown, a specific process may be as follows: First, the user of the content upload terminal 230 uploads the information content and the original cover image of the information information to the server 220 by accessing the client. The information content may include the title, text, etc. Then, after receiving the original cover image, the server 220 performs the following processing operations in sequence: generating a target salient area assignment map including salient areas and non-salient areas based on the original cover image; constructing a convolution kernel based on the display ratio of the display area, and making the convolution kernel include multiple layers of areas enclosed from the inside to the outside, and the weights of the elements in the inner layer area are greater than the weights of the elements in the inner layer area; using the convolution kernel to perform a sliding window convolution operation in the target salient area assignment map, and using the convolution result of the convolution kernel and each area where it stays as the recommendation index of the area; using the recommendation index of the area to determine the target area, and extracting the display area in the cover image according to the position of the target area. Finally, the server 220 uses the extracted display area as the cover image of the information entrance, and integrates the cover image and the information content into information and pushes it to the access client on one or more user access terminals. Users of the user access terminals can see the information entrance through the access client, and can also further open the information for browsing according to the title and cover image displayed in the information entrance.

[0058] In one embodiment of the present application, the convolution kernel includes a central area, a middle area located outside the central area, and an edge area located outside the middle area. The weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the middle area are greater than the weights of the elements in the edge area.

[0059] In one embodiment of the present application, the target salient region assignment map is obtained by fusing a plurality of salient region assignment maps obtained by performing different processing operations on the original image.

[0060] It should be understood that Figure 2 The number of user access terminals, servers, and content upload terminals is merely illustrative. Any number of user access terminals, servers, and content upload terminals may be provided as needed. For example, server 220 may be a server cluster consisting of multiple servers, and multiple content upload terminals may be provided simultaneously.

[0061] It should be noted that Figure 2 The embodiment shown is only one embodiment of the present application. Figure 2In the embodiment, the user access terminal is a smart phone and the content upload terminal is a desktop computer. However, in other embodiments of the present application, the user access terminal and the content upload terminal can be various types of terminal devices, such as but not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.; although in Figure 2 In the embodiment, the user access terminal and the content upload terminal are different types of terminals, but in other embodiments of the present application, the user access terminal and the content upload terminal can be the same type of terminal; although in Figure 2 The solution in the embodiment is applied to the scene of extracting the display area of ​​the picture, but it is easy to understand that the solution in the embodiment of the present application can be applied to various scenes where pictures need to be cropped; Figure 2 In the embodiment, the content upload terminal 230 uploads the original cover image to the server 220, and the server 220 extracts the corresponding display area. However, in other embodiments of the present application, the content upload terminal 230 can also directly extract the display area from the original cover image and then upload the display area to the server 220. The embodiment of the present application does not impose any limitation on this, and the scope of protection of the present application should not be limited thereby.

[0062] Furthermore, it is easy to understand that the image display area extraction method provided in the embodiments of the present application is generally executed by a user terminal, and accordingly, the image display area extraction device is generally provided in the user terminal. However, in other embodiments of the present application, the server may also have similar functions as the user terminal to implement the image display area extraction solution provided in the embodiments of the present application.

[0063] As mentioned above, the solution of the embodiment of the present application can be implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0064] The image display area extraction method provided in the embodiments of the present application can be applied to various software that requires image cropping, and the cropped images can be displayed in these software. For example, the solution of the embodiments of the present application can be specifically applied to apps in the form of information streams. The entrances to each piece of information displayed in these apps can be provided with the image display area extracted by the image display area extraction method provided in the embodiments of the present application. Figure 3 A schematic diagram showing an application scenario of a method for extracting a display area of ​​a picture according to an embodiment of the present application is shown in FIG. Figure 3 As shown in FIG, which shows a schematic diagram of the interface of a game community app, the game community app provides game-related information to users in the form of an information flow. The entrance to each information contains an image, which is located to the right of the information entrance. The image here can be the display area extracted based on the method of the embodiment of the present application. Users can further browse the detailed content of the information by clicking on the information entrance.

[0065] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0066] Figure 4 The flowchart of the method for extracting the display area of ​​an image according to an embodiment of the present application is shown. The method for extracting the display area of ​​an image can be executed by various devices capable of computing and processing, such as user terminals or servers. User terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The embodiment of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 4 As shown, the method for extracting the display area of ​​the image includes at least the following steps:

[0067] In step 410, a target salient region value map including salient regions and non-salient regions is generated according to the original image, wherein the pixel values ​​of the pixels in the salient regions are higher than the pixel values ​​of the pixels in the non-salient regions.

[0068] The original image can be in various formats, such as .jpg, .bmp, or .png. It can be a photograph, a computer-processed image, or even a frame from a video file. The original image can contain a variety of objects, including people, scenery, and objects.

[0069] The target salient area assignment map includes multiple pixels, each of which has a pixel value. The target salient area assignment map is equivalent to a matrix, the size of which can be equal to the size of the original image; the pixel values ​​of the pixels are elements in the matrix, and the number of elements in the matrix is ​​equal to the number of pixels in the original image. Therefore, the pixel value of the pixel can be the pixel value of the pixel (such as the grayscale value) or a value outside the pixel value range. The pixel values ​​of each pixel in the salient area can be the same or different; correspondingly, the pixel values ​​of each pixel in the non-salient area can be the same or different. It only needs to meet the following condition: the pixel value of each pixel in the non-salient area is lower than the pixel value of any pixel in the salient area.

[0070] Specifically, when the pixel values ​​of each pixel in the salient area of ​​the target salient area assignment map are equal and the pixel values ​​of each pixel in the non-salient area of ​​the target salient area assignment map are equal, the pixel values ​​of each pixel in the salient area can all be 255 grayscale values, and the pixel values ​​of the pixels in the non-salient area can all be 0 grayscale values, that is, in the target salient area assignment map, the salient area is white and the non-salient area is black.

[0071] Figure 5 An embodiment according to the present application is shown Figure 4 Flowchart of details of step 410 in an embodiment. Figure 5 The embodiment shown is proposed for scenes where the original picture contains human faces. Figure 5 As shown, the following steps may be included:

[0072] In step 411, the original image is input into the face detection model to obtain the original marked image with the face area marked, which is output by the face detection model.

[0073] The face detection model can use various algorithms that can identify the face area in the image, such as deep learning or machine learning algorithms. Specifically, the face detection model can use the YoloV3 deep learning model.

[0074] After the original image is input into the face detection model, the face detection model can detect the face area in the original image and then output the original labeled image with the face area marked. Therefore, the original labeled image is obtained by marking the face area in the original image. The face area in the original labeled image can be marked by a rectangular box.

[0075] In other embodiments of the present application, the corresponding original labeled image can be obtained according to the type of entity in the original image through a detection model that can identify the corresponding type of entity. For example, the original labeled image with the vehicle area marked can be output through a vehicle detection model.

[0076] In step 412, by assigning values ​​to the pixels in the face area and other areas in the original labeled image, a target salient area assignment map is obtained.

[0077] The face area is the salient area, and the rest of the area is the non-salient area. The pixels in the face area of ​​the original labeled image can be assigned a first pixel value, and the pixels in the rest of the area outside the face area of ​​the original labeled image can be assigned a second pixel value, where the second pixel value is greater than the first pixel value. Specifically, the first pixel value can be a grayscale value of 255, and the second pixel value can be a grayscale value of 0. Therefore, the face area in the target salient area assignment image is white, and the rest of the area is black.

[0078] Figure 6 The flowchart of obtaining the target salient area assignment map by assigning values ​​to the pixels in the original marked image according to one embodiment of the present application is shown. Figure 6 As shown, the following steps may be included:

[0079] In step 610, values ​​are assigned to the pixels in the face area and other areas in the original marked image to obtain a first salient area value map.

[0080] By setting different pixel values ​​for the pixels in the face area and other areas in the original marked image, the pixel values ​​of the pixels in the face area in the generated first salient area value map are made higher than the pixel values ​​of the pixels in other areas.

[0081] In step 620, the original image is processed according to a predetermined salient region detection algorithm to obtain a second salient region value map.

[0082] The predetermined salient region detection algorithm may adopt various salient detection algorithms such as FT (Frequency-tuned salient region detection) algorithm, DSS algorithm, and RBD algorithm.

[0083] Therefore, in other embodiments of the present application, the target salient region assignment map can also be directly obtained by processing the original image using a predetermined salient region detection algorithm, without the need to identify it through a face detection model.

[0084] The second salient region value assignment map may also include salient regions and non-salient regions, and the pixel values ​​assigned to the pixels in the salient regions are higher than the pixel values ​​assigned to the pixels in the non-salient regions.

[0085] In one embodiment of the present application, the second salient region assignment map can be obtained by processing the original image in the following manner:

[0086] 1. Use a 5*5 window to perform Gaussian smoothing on the original image to obtain a smoothed image;

[0087] 2. Convert the smoothed image from RGB color space to CIELAB color space to obtain a converted image;

[0088] 3. Calculate the average value of l, a, and b for the entire converted image;

[0089] 4. Calculate l for each pixel in the converted image according to the following formula. a The Euclidean distance between the l, a, and b values ​​and the average value of l, a, and b of the transformed image is used as the saliency value corresponding to each pixel to obtain the saliency map:

[0090] S(x, y)=||I μ -I whc (x, y)||,

[0091] Among them, S(x, y) is the saliency value of the pixel at the coordinate (x, y) in the transformed image, and I μ is the arithmetic mean of the pixel values ​​of each pixel in the converted image in the Lab color space, I whc (x, y) is the pixel value of the pixel at coordinate (x, y) in the converted image in Lab color space, and ||·|| is the Euclidean distance;

[0092] 5. Normalization: Divide the saliency value of each pixel in the saliency map by the maximum saliency value in the saliency map, and use the calculated result as the final saliency value of the corresponding pixel to obtain the second saliency region assignment map.

[0093] In step 630 , a fusion operation is performed on the first salient region value assignment map and the second salient region value assignment map to obtain a target salient region value assignment map.

[0094] The fusion operation can be performed on the first salient area value map and the second salient area value map in various ways. For example, corresponding pixels of the first salient area value map and the second salient area value map can be directly superimposed.

[0095] exist Figure 6In an embodiment, the target salient area assignment map is obtained by performing a fusion operation on the first salient area assignment map and the second salient area assignment map, so that the positions of the salient areas and the non-salient areas in the target salient area assignment map can be made more accurate, thereby ultimately improving the extraction effect of the display area.

[0096] In one embodiment of the present application, a fusion operation is performed on the first salient area assignment map and the second salient area assignment map to obtain a target salient area assignment map, including: weighting the pixel assignments of the pixels in the face area in the first salient area assignment map by a weighting coefficient to obtain a weighted first salient area assignment map; and superimposing the weighted first salient area assignment map with the pixel assignments of the corresponding pixels in the second salient area assignment map to obtain a target salient area assignment map.

[0097] The weighting coefficient may be a value greater than 1, for example, 2. Figure 7 FIG2 shows a schematic diagram of the principle of fusing the first salient region value map and the second salient region value map according to an embodiment of the present application. Figure 7As shown in FIG, three images of the same size are shown, from left to right: the first salient area assignment map, the second salient area assignment map, and the target salient area assignment map obtained by fusing the two. In the first salient area assignment map on the far left, pixels in the face area (salient area) are assigned a grayscale value of 255 and displayed as white, while pixels in other areas are assigned a grayscale value of 0 and displayed as black. In the second salient area assignment map in the middle, pixels in the salient area are assigned a grayscale value of 255 and displayed as white, while pixels in the non-salient area are assigned a grayscale value of 0 and displayed as black. When fusing the first salient area assignment map with the second salient area assignment map, the pixel values ​​of the pixels in the face area in the first salient area assignment map can be multiplied by a weighting factor of 2. Thus, the pixel values ​​of the pixels in the face area in the first salient area assignment map become 510 after weighting processing. Then, since there are pixels with pixel values ​​of 255 and 0 in the area corresponding to the face area in the second salient area assignment map, the pixel values ​​of the pixels in the area corresponding to the face area in the finally generated target salient area assignment map are 765 and 510, respectively. Since the pixel values ​​of the pixels in other areas in the first salient area assignment map are 0, the pixel values ​​of the pixels in other areas in the target salient area assignment map are consistent with the pixel values ​​of the corresponding pixels in the second salient area assignment map, that is, the pixel values ​​of the pixels in other areas in the target salient area assignment map are 255 and 0, respectively. The pixel points with a pixel value of 765 in the target salient area assignment map can be set to orange-red, the pixel points with a pixel value of 510 can be set to yellow, the pixel points with a pixel value of 255 can be set to blue, and the pixel points with a pixel value of 0 can be kept black, so that the area corresponding to the face area can be highlighted and the pixels in the area can be further distinguished according to the pixel assignment.

[0098] In an embodiment of the present application, in order to obtain a target salient area assignment map, the pixel values ​​of the pixels in the face area of ​​the first salient area assignment map are first weighted to obtain a weighted first salient area assignment map, and then the target salient area assignment map is obtained by superimposing the weighted first salient area assignment map with the pixel values ​​of the corresponding pixels in the second salient area assignment map. This can further highlight the pixels in the target salient area assignment map that correspond to the face area of ​​the first salient area assignment map, thereby improving the accuracy of the target salient area assignment map and thus improving the effect of display area extraction.

[0099] In other embodiments of the present application, the pixel values ​​of the pixels in the face area of ​​the first salient area value assignment map may be directly set to 510.

[0100] Please continue reading Figure 4In step 420, a convolution kernel is constructed according to the display ratio of the display area, wherein the convolution kernel includes multiple layers of regions distributed from the inside to the outside, and the weight of the elements in each layer of region is less than the weight of the elements in the inner layer of each layer of region.

[0101] In the convolution process, given an input image, the pixel values ​​of a small area in the input image are weighted averaged to obtain the corresponding pixel value of each pixel in the output image, where the weights used are defined by a function called the convolution kernel.

[0102] For example, if the display ratio of the display area is 200×400, then the display ratio of the display area is 1:2. Therefore, the display ratio of the constructed convolution kernel is also 1:2. The actual size of the convolution kernel can be 200×400, 100×200, or 400×800. When the size of the convolution kernel is different from the size of the display area, the final extracted display area needs to be scaled to the same size as the display area.

[0103] The convolution kernel moves from the inside to the outside, and the weights of the elements in each layer decrease successively.

[0104] In one embodiment of the present application, the convolution kernel includes a central area, a middle area located outside the central area, and an edge area located outside the middle area. The weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the middle area are greater than the weights of the elements in the edge area.

[0105] The weights of the elements in the middle area can be the same or different; the weights of the elements in the central area can be the same or different; the weights of the elements in the edge area can be the same or different.

[0106] In one embodiment of the present application, the middle area includes four middle areas and four corner areas, the elements in the middle area are in the same row or column as the elements in the central area, and the elements in the corner area are in different rows and columns from the elements in the central area; the weights of the elements in the edge area are negative, the weights of the elements in the central area and the middle area are positive, the weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the central area are greater than the weights of the elements in the corner area.

[0107] Regardless of whether they are elements in a central region or elements between central regions, the weights of the elements can be the same or different; regardless of whether they are elements in a corner region or elements between corner regions, the weights of the elements can be the same or different.

[0108] In an embodiment of the present application, by setting the weights of the elements in the edge area to negative numbers, when the edge area of ​​the convolution kernel passes through the salient area of ​​the target salient area assignment map, the weights of the elements in the edge area will bring a greater penalty to the convolution result, so that the final extracted display area will not separate the salient area.

[0109] Figure 8 A schematic diagram of a convolution kernel according to an embodiment of the present application is shown. If the size of the display area is 8×8, the size of the convolution kernel can also be 8×8. Figure 8 As shown in the figure, a convolution kernel of size 8×8 is shown. The weights of the elements in the edge region of the convolution kernel are all -r, where -r is a negative number. The weights of the elements in the edge region are the edge damping coefficient, which means that the edge of the convolution kernel will have a certain resistance when passing through the salient area. The larger the r, the higher the resistance. The weights of the elements in the central region of the convolution kernel are all a, the weights of the elements in the four middle regions are all b, and the weights of the elements in the four corner regions are all c. The central region is the most attractive, so the weights of the elements in the central region are the highest among the non-edge regions. The corner regions are the most easily overlooked, so the weights of the elements in the corner regions are the lowest among the non-edge regions. The weights of the elements in the central region are between the weights of the elements in the corner regions and the weights of the elements in the central region, that is, a>b>c.

[0110] In step 430, the convolution kernel is used to move multiple times on the target salient region assignment map, and after each movement, the convolution result of the convolution kernel and the area where the convolution kernel currently stays on the target salient region assignment map is determined as the recommendation index of the area.

[0111] The process of moving the convolution kernel on the target salient area assignment map and determining the convolution result is called sliding window convolution. The sliding window convolution method can be the same as the way of convolution on images in deep learning.

[0112] Figure 9 FIG2 shows a schematic diagram of the principle of performing a convolution operation on a target salient region assignment map using a convolution kernel according to an embodiment of the present application. Figure 9 As shown in the figure, the lower layer is the target salient region assignment map, and the upper layer is the convolution result. Each convolution result corresponds to a 4×4 region in the target salient region assignment map. Since this region is where the convolution kernel resides, the convolution kernel size is also 4×4. You can choose an appropriate step size based on the target salient region assignment map or the size of the original image. For example, you can choose a step size of one twentieth of the image width. During the sliding window convolution process, each movement and convolution generates a value that serves as the recommendation index for the corresponding region.

[0113] In step 440 , a target region in the target salient region assignment map is determined according to the recommendation index of each region, and a display region is extracted from the original image according to the position of the target region in the target salient region assignment map.

[0114] The target area in the target salient area assignment map is located at the same position as the display area extracted from the original image.

[0115] The display area is the image created by cropping the original image. After extracting the display area, you can push it to the app in the information flow format. The display area is usually rectangular. When pushing the display area, you can further crop it into a rounded rectangle to improve the display effect.

[0116] Figure 10 An embodiment according to the present application is shown Figure 4 Flowchart of details of step 440 in the embodiment. Figure 10 As shown, step 440 may specifically include the following steps:

[0117] In step 441 , the maximum recommendation index is determined among the recommendation indexes of the respective regions.

[0118] Each time a region is convolved, a corresponding recommendation index can be obtained; by comparing the recommendation indexes of each region, the maximum recommendation index is obtained.

[0119] In step 442, the region corresponding to the maximum recommendation index is used as the target region in the target salient region assignment map.

[0120] In the embodiment of the present application, by using the area corresponding to the maximum recommendation index as the target area, the finally extracted display area can have a better display effect.

[0121] Figure 11 Another embodiment according to the present application is shown Figure 4 Flowchart of details of step 440 in the embodiment. Figure 11 As shown, step 440 may further include the following steps:

[0122] In step 441 ′, a plurality of candidate regions are determined based on the recommendation index of each region.

[0123] The regions can be sorted from largest to smallest according to their recommendation index, and a predetermined number of regions with the highest recommendation index are selected from all regions as candidate regions. The recommendation index of the candidate regions is usually higher than that of other regions.

[0124] In one embodiment of the present application, multiple candidate regions are determined based on the recommendation index of each region, including:

[0125] Repeat the candidate region selection step until the number of selected candidate regions reaches a predetermined number;

[0126] Among them, the candidate area selection step includes: determining the maximum recommendation index among the recommendation indexes of the unselected areas, and selecting the area corresponding to the maximum recommendation index as the candidate area; setting the recommendation index of the adjacent area corresponding to the candidate area to zero, and the distance between the center point of the adjacent area and the center point of the candidate area is less than the predetermined distance.

[0127] Specifically, after the convolution is completed, an array storing the recommendation index of each region will be obtained; if the length of the convolution kernel is the same as the length of the target salient region assignment map, then after the convolution is completed, a 1×n array will be obtained; if the length of the convolution kernel is different from the length of the target salient region assignment map, then after the convolution is completed, an m×n array will be obtained.

[0128] This embodiment will perform the following steps based on the array:

[0129] 1) Take out the maximum recommendation index in the current array and select the area corresponding to the maximum recommendation index as the candidate area;

[0130] 2) Taking the center point of the area corresponding to the last maximum recommendation index as the center, set the center point of each area and the area within a length r from the center as the zero area, and set the recommendation index corresponding to the zero area in the current array to 0;

[0131] 3) Return to step 1) until the number of selected candidate regions reaches the requirement.

[0132] By setting the recommendation index of the zero-set area around the candidate area to 0, it is possible to prevent the obtained candidate areas from being too close.

[0133] In step 442 ′, a target region in the target salient region assignment map is determined from the plurality of candidate regions.

[0134] The target area can be determined from multiple candidate areas in a variety of ways. For example, a candidate area can be randomly selected from multiple candidate areas as the target area; for another example, multiple candidate areas can be output to a human-computer interaction interface, and a candidate area can be manually selected from the multiple candidate areas through the human-computer interaction interface as the target area; for another example, multiple candidate areas can be input into an aesthetic evaluation model to obtain the aesthetic evaluation score corresponding to each candidate area output by the aesthetic evaluation model, and finally the candidate area with the highest aesthetic evaluation score can be used as the target area.

[0135] Since the selection of the target area is subjective, even if the area corresponding to the maximum recommendation index is selected as the target area, it may not necessarily achieve the best display effect. In the embodiment of the present application, by first determining multiple candidate areas and then determining the target area from the multiple candidate areas, the flexibility of selecting the target area is improved, and the quality of the final extracted display area is guaranteed to a certain extent.

[0136] Figure 12 Schematic diagram of the overall process according to an embodiment of the present application is shown. Figure 12 As shown, the following steps may be included:

[0137] In step 1210 , a picture containing a salient area is obtained.

[0138] Generate an image containing salient regions and other regions based on the original image.

[0139] In step 1220, a 2D convolution kernel is constructed according to the cropping ratio.

[0140] Construct a two-dimensional convolution kernel according to the ratio of the image to be cropped.

[0141] In step 1230, a sliding window convolution is performed on the image using a 2D convolution kernel.

[0142] Use a two-dimensional convolution kernel to perform sliding window convolution on the image containing salient areas to obtain the recommendation index corresponding to each area in the image.

[0143] In step 1240 , the region with a higher recommendation index after convolution is selected as the final cropping region.

[0144] After the final cropping area is obtained, the original image can be cropped according to the position of the final cropping area.

[0145] In summary, according to the display area extraction method of the picture provided in the embodiment of the present application, a display area extraction scheme with strong robustness and robustness is provided, which can extract the optimal display area; for pictures with relatively scattered distribution of significant areas, a relatively ideal display area can also be extracted, and the final extracted display area contains as many significant areas as possible without cutting the significant areas from the middle; in addition, the embodiment of the present application also provides a quantitative recommendation index for whether each area is suitable for display, which provides certain data support for subsequent processing.

[0146] The following describes an embodiment of the device of the present application, which can be used to execute the method for extracting the display area of ​​an image in the above-mentioned embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method for extracting the display area of ​​an image in the above-mentioned embodiment of the present application.

[0147] Figure 13 A block diagram of a device for extracting a display area of ​​a picture according to an embodiment of the present application is shown.

[0148] Reference Figure 13 As shown, a device 1300 for extracting a display area of ​​a picture according to an embodiment of the present application includes: a generation unit 1310, a construction unit 1320, a convolution unit 1330, and a determination and extraction unit 1340. Among them, the generation unit 1310 is used to generate a target significant area assignment map including a significant area and a non-significant area according to the original image, and the pixel assignment value of the pixel point in the significant area is higher than the pixel assignment value of the pixel point in the non-significant area; the construction unit 1320 is used to construct a convolution kernel according to the display ratio of the display area, wherein the convolution kernel includes multiple layers of areas distributed from the inside to the outside, and the weights of the elements in each layer area are less than the weights of the elements in the inner layer area of ​​each layer area; the convolution unit 1330 is used to use the convolution kernel to move multiple times on the target significant area assignment map, and after each movement, determine the convolution result of the convolution kernel and the area where the convolution kernel currently stays on the target significant area assignment map as the recommendation index of the area; the determination and extraction unit 1340 is used to determine the target area in the target significant area assignment map according to the recommendation index of each area, and extract the display area from the original image according to the position of the target area in the target significant area assignment map.

[0149] In some embodiments of the present application, based on the aforementioned scheme, the convolution kernel includes a central area, a middle area located outside the central area, and an edge area located outside the middle area. The weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the middle area are greater than the weights of the elements in the edge area.

[0150] In some embodiments of the present application, based on the aforementioned scheme, the generation unit 1310 is configured to: input the original image into the face detection model to obtain the original marked image with the face area marked output by the face detection model; and obtain the target salient area assignment map by assigning values ​​to the pixel points in the face area and other areas in the original marked image.

[0151] In some embodiments of the present application, based on the aforementioned scheme, the generation unit 1310 is configured to: assign values ​​to the pixel points in the face area and other areas in the original marked image respectively to obtain a first significant area assignment map; process the original image according to a predetermined significant area detection algorithm to obtain a second significant area assignment map; and perform a fusion operation on the first significant area assignment map and the second significant area assignment map to obtain a target significant area assignment map.

[0152] In some embodiments of the present application, based on the aforementioned scheme, the generation unit 1310 is configured to: perform weighted processing on the pixel values ​​of the pixel points in the face area of ​​the first salient area assignment map by a weighting coefficient to obtain a weighted first salient area assignment map; and superimpose the weighted first salient area assignment map with the pixel values ​​of the corresponding pixel points in the second salient area assignment map to obtain a target salient area assignment map.

[0153] In some embodiments of the present application, based on the aforementioned scheme, the determination and extraction unit 1340 is configured to: determine the maximum recommendation index among the recommendation indexes of each region; and use the region corresponding to the maximum recommendation index as the target region in the target salient region assignment map.

[0154] In some embodiments of the present application, based on the aforementioned scheme, the determination and extraction unit 1340 is configured to: determine multiple candidate regions according to the recommendation index of each region; and determine the target region in the target salient region assignment map from the multiple candidate regions.

[0155] In some embodiments of the present application, based on the aforementioned scheme, the determination and extraction unit 1340 is configured to: repeatedly perform the candidate area selection step until the number of selected candidate areas reaches a predetermined number; wherein the candidate area selection step includes: determining the maximum recommendation index among the recommendation indexes of the unselected areas, and selecting the area corresponding to the maximum recommendation index as the candidate area; setting the recommendation index of the neighboring area corresponding to the candidate area to zero, and the distance between the center point of the neighboring area and the center point of the candidate area is less than the predetermined distance.

[0156] In some embodiments of the present application, based on the aforementioned scheme, the middle area includes four middle areas and four corner areas, the elements in the middle area are in the same row or column as the elements in the central area, and the elements in the corner area are in different rows and columns from the elements in the central area; the weights of the elements in the edge area are negative, the weights of the elements in the central area and the middle area are positive, the weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the central area are greater than the weights of the elements in the corner area.

[0157] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0158] It should be noted that Figure 14 The computer system 1400 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0159] like Figure 14 As shown, computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1402 or programs loaded from storage unit 1408 into random access memory (RAM) 1403, such as executing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1403. CPU 1401, ROM 1402, and RAM 1403 are connected to each other via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0160] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, and the like; an output section 1407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1408 including a hard disk; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. Removable media 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1410 as needed, so that computer programs read from the removable media can be installed in the storage section 1408 as needed.

[0161] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1409, and / or installed from a removable medium 1411. When the computer program is executed by the central processing unit (CPU) 1401, the various functions defined in the system of the present application are executed.

[0162] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0165] As one aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0166] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0167] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0168] It is understandable that in the specific implementation of this application, data related to image processing is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0169] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0170] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for extracting a display area of ​​a picture, characterized in that: The method comprises: generating a target salient area value map including salient areas and non-salient areas according to the original image, wherein the pixel values ​​assigned to the pixels in the salient areas are higher than the pixel values ​​assigned to the pixels in the non-salient areas; Constructing a convolution kernel according to the display ratio of the display area, wherein the convolution kernel includes multiple layers of regions distributed from the inside to the outside, and the weight of the elements in each layer of regions is smaller than the weight of the elements in the inner layer of each layer of regions; Using the convolution kernel to move multiple times on the target salient area assignment map, and after each movement, determining a convolution result of the convolution kernel and the area where the convolution kernel currently stays on the target salient area assignment map as a recommendation index for the area; A target region in the target salient region assignment map is determined according to the recommendation index of each region, and a display region is extracted from the original image according to the position of the target region in the target salient region assignment map.

2. The method for extracting the display area of ​​a picture according to claim 1, wherein: The convolution kernel includes a central area, a middle area located outside the central area, and an edge area located outside the middle area. The weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the middle area are greater than the weights of the elements in the edge area.

3. The method for extracting the display area of ​​a picture according to claim 1, wherein: Generating a target salient area assignment map including salient areas and non-salient areas according to the original image includes: Inputting the original image into the face detection model to obtain the original marked image with the face area marked, which is output by the face detection model; By assigning values ​​to the pixels in the face area and other areas in the original marked image respectively, a target salient area assignment map is obtained.

4. The method for extracting the display area of ​​a picture according to claim 3, wherein: The target salient area assignment map is obtained by assigning values ​​to the pixels in the face area and other areas in the original marked image, including: Assigning values ​​to the pixels in the face area and other areas of the original marked image respectively to obtain a first salient area assignment map; Processing the original image according to a predetermined salient region detection algorithm to obtain a second salient region value assignment map; A fusion operation is performed on the first salient area value assignment map and the second salient area value assignment map to obtain a target salient area value assignment map.

5. The method for extracting the display area of ​​a picture according to claim 4, wherein: The fusing operation of the first salient area value assignment map and the second salient area value assignment map to obtain a target salient area value assignment map includes: Performing weighted processing on pixel values ​​of pixels in the face region of the first salient region value map using a weighting coefficient to obtain a weighted first salient region value map; The pixel values ​​of corresponding pixels in the weighted first salient area value assignment map and the second salient area value assignment map are superimposed to obtain a target salient area value assignment map.

6. The method for extracting the display area of ​​a picture according to any one of claims 1 to 5, characterized in that: Determining the target region in the target salient region assignment map according to the recommendation index of each region includes: Determine the maximum recommendation index among the recommendation indexes of each region; The area corresponding to the maximum recommendation index is used as the target area in the target salient area assignment map.

7. The method for extracting the display area of ​​a picture according to any one of claims 1 to 5, characterized in that: Determining the target region in the target salient region assignment map according to the recommendation index of each region includes: Determine multiple candidate areas based on the recommendation index of each area; Determine a target region in the target salient region assignment map from the multiple candidate regions.

8. The method for extracting the display area of ​​a picture according to claim 7, wherein: The multiple candidate regions are determined based on the recommendation index of each region, including: Repeat the candidate region selection step until the number of selected candidate regions reaches a predetermined number; The candidate region selection step includes: Determining a maximum recommendation index among the recommendation indexes of the unselected regions, and selecting the region corresponding to the maximum recommendation index as a candidate region; The recommendation index of the neighboring area corresponding to the candidate area is set to zero, and the distance between the center point of the neighboring area and the center point of the candidate area is less than a predetermined distance.

9. The method for extracting a display area of ​​a picture according to claim 2, wherein: The middle area includes four middle areas and four corner areas. The elements in the middle area are in the same row or column as the elements in the central area, and the elements in the corner area are in different rows and columns from the elements in the central area. The weights of the elements in the edge area are negative, and the weights of the elements in the central area and the middle area are positive. The weights of the elements in the central area are greater than the weights of the elements in the middle area, and the weights of the elements in the central area are greater than the weights of the elements in the corner area.

10. A device for extracting a display area of ​​a picture, characterized in that: The device comprises: A generating unit, configured to generate a target salient area value map including salient areas and non-salient areas according to an original image, wherein the pixel values ​​assigned to pixels in the salient areas are higher than the pixel values ​​assigned to pixels in the non-salient areas; a construction unit, configured to construct a convolution kernel according to a display ratio of the display area, wherein the convolution kernel includes multiple layers of regions distributed from the inside to the outside, and the weight of the elements in each layer of regions is smaller than the weight of the elements in the inner layer of each layer of regions; a convolution unit, configured to use the convolution kernel to move multiple times on the target salient area assignment map, and after each movement, determine a convolution result of the convolution kernel with the area where the convolution kernel currently resides on the target salient area assignment map as a recommendation index for the area; The determination and extraction unit is used to determine the target area in the target salient area assignment map according to the recommendation index of each area, and extract the display area from the original image according to the position of the target area in the target salient area assignment map.

Images