Image processing method

Abstract

The invention provides an image processing method, which is applied to the technical field of artificial intelligence. The image processing method comprises the following steps: displaying a first target image; in response to a target interaction operation for the first target image, determining a target object indicated by the target interaction operation; cutting the first target image based on the target object to obtain a second target image; wherein the second target image comprises a target object meeting the target composition condition.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as image processing, and in particular to an image processing method. Background Technology

[0002] When non-professional users want to crop and edit parts of an image, the results are often less than ideal due to their lack of knowledge in photography and aesthetics. Summary of the Invention

[0003] In view of this, the present disclosure provides an image processing method.

[0004] According to a first aspect of this disclosure, an image processing method is provided, comprising: displaying a first target image; determining a target object indicated by a target interaction operation in response to a target interaction operation on the first target image; and cropping the first target image based on the target object to obtain a second target image; wherein the second target image contains a target object that satisfies target composition conditions.

[0005] According to embodiments of this disclosure, cropping a first target image based on a target object to obtain a second target image includes: determining a plurality of candidate image regions containing the target object in the first target image; determining a target image region containing the target object that satisfies the target composition conditions among the plurality of candidate image regions; and cropping the target image region from the first target image to obtain the second target image.

[0006] According to an embodiment of this disclosure, cropping a first target image based on a target object to obtain a second target image includes: determining a reference image region in the first target image that satisfies target conditions; inputting image data of the reference image region into a target model to obtain a target image region containing a target object that satisfies target composition conditions; cropping the target image region from the first target image to obtain the second target image; wherein, the target conditions characterize that the salience of the target object in the reference image region is higher than the target salience.

[0007] According to an embodiment of this disclosure, determining a reference image region that satisfies the target conditions in a first target image includes: determining an initial image region containing a target object in the first target image; scaling the region boundary of the initial image region; and determining a reference image region that satisfies the target conditions in the first target image.

[0008] According to embodiments of this disclosure, the scaling process has a target parameter, which characterizes a scalable boundary value for scaling the process, and the scalable boundary value is related to the size of the initial image region.

[0009] According to embodiments of this disclosure, the reference image region is an image region obtained by scaling an initial image region with a target scaling value, where the target scaling value is related to the salience of the target object in the reference image region.

[0010] According to embodiments of this disclosure, if the salience of the target object in the intermediate image region is higher than the target salience, the intermediate image region with the largest size is used as the reference image region; the intermediate image region is the image region formed by scaling the initial image region with different target scaling values.

[0011] According to an embodiment of this disclosure, scaling the region boundary of an initial image region to determine a reference image region that meets the target conditions includes: scaling the initial image region with a first scaling value to obtain a first intermediate image region; if the first intermediate image region meets the target conditions, scaling the first intermediate image region with a second scaling value to obtain a second intermediate image region; if the second intermediate image region does not meet the target conditions, determining the first intermediate image region as the reference image region, wherein the second scaling value is greater than the first scaling value.

[0012] According to embodiments of this disclosure, the method further includes: if the boundary of the reference image region exceeds the boundary of the first target image, taking the boundary where the first target image and the reference image region overlap as the boundary of the reference image region.

[0013] According to embodiments of this disclosure, in response to a target interaction operation on a first target image, determining the target object indicated by the target interaction operation includes: dividing the first target image into multiple image regions with semantic labels, each of the multiple image regions having a different target object, and the semantic labels representing the target objects in the image regions; and in response to clicking at least one target location in the first target image, determining the target object of the image region corresponding to each target location.

[0014] A second aspect of this disclosure provides an image processing apparatus, comprising: a display module for displaying a first target image; a determination module for determining a target object indicated by a target interaction operation in response to a target interaction operation on the first target image; and a cropping module for cropping the first target image based on the target object to obtain a second target image; wherein the second target image contains a target object that satisfies target composition conditions.

[0015] A third aspect of this disclosure provides an electronic device, comprising: a display screen for displaying an image; a processor for displaying a first target image on the display screen; determining a target object indicated by a target interaction operation in response to a target interaction operation on the first target image; and cropping the first target image based on the target object to obtain a second target image; wherein the second target image contains a target object that satisfies target composition conditions.

[0016] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the image processing method described above.

[0017] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image processing method.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of an image in the related art is shown;

[0021] Figure 2 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 3 One of the schematic diagrams of an image processing method according to an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram of an image processing method according to an embodiment of the present disclosure is shown in the second illustration;

[0024] Figure 5 A schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0025] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0026] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] This disclosure provides an image processing method. Before introducing the technical solution provided by this disclosure, the related technologies involved in this disclosure will be described first.

[0032] When non-professional users want to crop and edit parts of an image, the results are often less than ideal due to their lack of knowledge in photography and aesthetics.

[0033] For example, such as Figure 1The image shown illustrates a user's desire to crop out the kitten from the image for retouching. One approach is manual image editing, which requires a certain level of compositional skills and aesthetic knowledge. Lacking professional retouching expertise, users may struggle to refine the kitten image effectively (e.g., cropping, composition, color correction), leading to a poor user experience. Another approach is to use automated image editing software, but this may result in the software retaining the wrong part of the image during cropping. For example, the user might want to keep the kitten, but the software might retain the little girl portion, resulting in an unsatisfactory retouching effect and a diminished user experience.

[0034] Embodiments of this disclosure provide an image processing method, including: displaying a first target image; determining a target object indicated by a target interaction operation in response to a target interaction operation on the first target image; cropping the first target image based on the target object to obtain a second target image; wherein the second target image contains a target object that satisfies target composition conditions.

[0035] The image processing method of this disclosure can be applied to electronic devices with a display screen. This electronic device can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, AR / VR device, etc. This disclosure does not impose any special limitations on the specific form of the device. The operating system of the electronic device may include, but is not limited to, Flyme, Android, iOS, Symbian, BlackBerry, Windows Phone, etc.

[0036] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0037] The following will be through Figures 2-4 The image processing method according to the embodiments of this disclosure will be described in detail.

[0038] Figure 2 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown schematically.

[0039] like Figure 3 As shown, the image processing method of this embodiment includes operations S210 to S230.

[0040] In operation S210, the first target image is displayed.

[0041] In operation S220, in response to a target interaction operation on the first target image, the target object indicated by the target interaction operation is determined.

[0042] In operation S230, the first target image is cropped based on the target object to obtain the second target image.

[0043] The second target image contains a target object that meets the target composition conditions.

[0044] For example, the first target image may be an image to be edited obtained by an electronic device. The electronic device needs to perform editing operations on the first target image, such as cropping, reframing, color correction, etc.

[0045] The first target image can be a preview image captured by the image acquisition device of an electronic device at the current moment. The first target image can also be an image captured by the electronic device at the current moment. Alternatively, the first target image can be an image downloaded by the electronic device at the current moment. Or, the first image can be an image selected by the electronic device from an image database (such as a photo album) at the current moment. This disclosure does not specifically limit the method of obtaining the first target image.

[0046] After acquiring the first target image, the electronic device displays the first target image on the display area of ​​the electronic device. For example, if the electronic device is a mobile phone, the user opens an image (the first target image) from the photo album, and the target image is displayed on the phone's screen.

[0047] The target object can be a person, animal, or object in the image.

[0048] The target interaction operation can be based on the interaction between the user and the electronic device, specifying at least one target object from a first target image. That is, the target object determined based on the target interaction operation is compatible with the user's intention. The target interaction operation can be used to instruct the selection of one or more target objects in the first target image.

[0049] Targeted interactive operations include, but are not limited to: click operations, selection operations, long press operations, voice commands, and gesture operations. For example, a user briefly touches the location of the kitten on the screen displaying the first target image using a finger, stylus, or mouse, thus confirming that the user wants to select the kitten in the first target image. Or, a user inputs a voice command through a microphone: "Select the little girl and the kitten in the image," thus confirming that the user wants to select the little girl and the kitten in the first target image.

[0050] Compositional conditions can refer to how the target object is positioned within the second target image. Different compositional methods result in variations in the target object's relative position, size, and alignment within the cropping area. Compositional methods can include: rule of thirds, central composition, diagonal composition, horizontal composition, symmetrical composition, curved composition, vertical composition, triangular composition, and repetitive pattern composition. Different compositional methods for the target object result in different aesthetic evaluation values ​​for the image including the target object.

[0051] The target composition condition can be the way the target object is composed in the second target image so that the aesthetic evaluation value of the second target image is the highest.

[0052] After selecting the target object, the first target image is cropped. The cropped area surrounds the selected target object, and the composition of the target object in the image corresponding to the cropped area (the second target image) satisfies the target composition conditions.

[0053] When the target objects are different, the composition of the second target image regarding the target object can be the same or different. For example, when the target object is a kitten, using a central composition will result in the highest aesthetic evaluation score for the second target image. When the target object is a little girl, using a rule-of-thirds composition will result in the highest aesthetic evaluation score for the second target image.

[0054] Different compositions of the target object result in different cropping methods around it. These different cropping methods can vary in the area or shape of the cropped object. The aesthetic effect of the different compositions of the target object in the first target image is evaluated, and the boundary of the image corresponding to the composition with the highest aesthetic evaluation value is used as the cropping line.

[0055] The second target image can be a portion of the first target image relating to the target object, where the composition of the target object within that portion satisfies the target composition conditions. The boundary of this portion is the edge line used to crop the first target image.

[0056] Understandably, based on the interaction between the user and the image, a target object that matches the user's intent is determined, and the image is automatically cropped and composed around the target object to generate an image that conforms to aesthetic standards.

[0057] As described above, in operation S230, the first target image is cropped based on the target object to obtain the second target image. In one possible implementation, this operation may further include: determining a plurality of candidate image regions containing the target object in the first target image; determining a target image region containing the target object that satisfies the target composition conditions among the plurality of candidate image regions; and cropping the target image region from the first target image to obtain the second target image.

[0058] For example, the candidate image region can be a region formed in the first target image by using one or more composition methods for the target object, and the target object is located within the candidate image region.

[0059] The target image region can be a candidate image region that meets the target composition conditions. For example, the candidate image region with the highest aesthetic evaluation value can be used as the target image region. If multiple candidate image regions have different composition methods, then the aesthetic evaluation values ​​of the corresponding image portions in the first target image will be different. Therefore, the candidate image region with the highest aesthetic evaluation value will be used as the target image region.

[0060] A composition method for the target object can be preset, and one or more composition methods can be used. If one composition method is preset, a candidate image region is generated for the target object, and this candidate image region is regarded as the target image region that meets the target composition conditions. If multiple composition methods are preset, candidate image regions corresponding to each of the multiple composition methods can be generated for the target object, and the candidate image region that meets the target composition conditions among the multiple candidate image regions is taken as the target image region. The embodiments of this disclosure do not impose a specific limit on the number of preset composition methods, and the number of composition methods can be adjusted according to actual needs.

[0061] In one example, if only one pre-defined composition method is used, and that method is the rule of thirds, then based on the user's interaction with the first target image, after identifying the target object, a candidate image region is obtained by composing a picture of the target object within the first target image according to the rule of the rule of thirds. Since there is only one composition method, it is assumed that the candidate image region corresponding to this composition method meets the target composition conditions, and this candidate image region is used as the target image region. The first target image is then cropped according to the boundaries of the target image region to obtain the second target image.

[0062] In another example, when multiple pre-defined composition methods are available, such as rule of thirds, central composition, and symmetrical composition, after determining the target object based on the user's interaction with the first target image, the first candidate image region is obtained by composing a composition for the target object in the first target image according to the rule of rule of rule; the second candidate image region is obtained by composing a composition for the target object in the first target image according to the rule of central composition; and the third candidate image region is obtained by composing a composition for the target object in the first target image according to the rule of symmetrical composition. Aesthetic evaluation is then performed on the image regions corresponding to the first, second, and third candidate image regions to obtain a first aesthetic evaluation value corresponding to the first candidate image region, a second aesthetic evaluation value corresponding to the second candidate image region, and a third aesthetic evaluation value corresponding to the third candidate image region. If the third aesthetic evaluation value > the first aesthetic evaluation value > the second aesthetic evaluation value, then the third candidate image region is selected as the target image region that satisfies the target composition conditions, and the first target image is cropped according to the boundary of the target image region to obtain the second target image.

[0063] In some embodiments, when there are N candidate image regions, i.e. there are N ways to compose the target object, the composition confidence of the N candidate image regions is determined; aesthetic evaluation is performed on the n candidate image regions whose composition confidence satisfies the first condition to obtain aesthetic evaluation values ​​for the n candidate image regions; when the first aesthetic evaluation value satisfies the second condition, the candidate image region corresponding to the first aesthetic evaluation value is taken as the target image region.

[0064] Composition confidence can be defined as the degree of match between the actual position of the target object in the candidate image region and its corresponding composition. For example, if the target object is exactly on the golden section point of the candidate image region, the composition confidence might be 100%; if the target object deviates slightly from the golden section point of the candidate image region, the composition confidence might be 90%.

[0065] The first criterion could be selecting the top three candidate image regions from multiple composition confidence rankings, from highest to lowest. The second criterion could be selecting the region with the highest aesthetic evaluation score.

[0066] Candidate image regions corresponding to various composition methods of the target object are generated. The composition confidence between the candidate image regions and their corresponding composition methods is determined, and they are sorted from high to low composition confidence. The candidate image regions corresponding to the top three composition confidence are aesthetically evaluated to obtain aesthetic evaluation values ​​for the three candidate image regions. The candidate image region with the highest aesthetic evaluation value is selected as the target image region.

[0067] As described above, in operation S230, the first target image is cropped based on the target object to obtain the second target image. In another possible implementation, this operation may further include: determining a reference image region in the first target image that satisfies the target conditions; inputting the image data of the reference image region into the target model to obtain a target image region containing the target object that satisfies the target composition conditions; cropping the target image region from the first target image to obtain the second target image; wherein, the target conditions characterize that the salience of the target object in the reference image region is higher than the target salience.

[0068] For example, saliency can be the degree to which a target object stands out in a reference image region. For instance, saliency can be determined based on the size of the target object's area in the image region; for example, the larger the area of ​​the target object in the image region, the higher the saliency. Saliency can also be determined based on the target object's position in the image region; for example, the closer the target object is to the center of the image region, the higher the saliency.

[0069] Target saliency can be used to indicate the expected saliency of a target object in a reference image region. For example, if there is one target object in the reference image region, the target saliency could be 50%. If there are multiple objects (including target and non-target objects) in the reference image region, the target saliency can be determined based on the saliency of the non-target objects in the image region. If the saliency of the non-target objects in the reference image region is 60%, then the target saliency of the target object needs to be set to a value greater than 60% so that the target model can accurately distinguish the target object in the reference image region. It is necessary to ensure that the saliency of the target object in the reference image region is greater than the saliency of the non-target objects in the reference image region.

[0070] Target saliency can also be the degree to which a target object stands out in a reference image region relative to non-target objects in the same region. For example, in a reference image region containing multiple objects (including target and non-target objects), target saliency can be the difference between the saliency of the target object and the saliency of the non-target objects, such as 10%.

[0071] The significance of the target can be set according to the actual situation, and this disclosure does not limit it.

[0072] The reference image region can be an image region containing the target object, where the salience of the target object is higher than that of the target. The reference image region can serve as the basis for subsequent different composition methods for the target object; that is, different composition methods can be applied to the target object based on the reference image region.

[0073] The target model can adjust the composition of the input image in different ways, predict the aesthetic evaluation value of the image corresponding to each composition method, and finally output the composition box corresponding to the composition method with the highest aesthetic evaluation value. The composition box corresponds to the boundary of the target image region.

[0074] In one example, after determining the target object based on the user's interaction with the first target image, a reference image region is defined within the first target image. This reference image region contains the target object, and the salience of the target object within the reference image region is greater than the target salience (e.g., 50%). The image data corresponding to the reference image region is then input into the target model. The target model predicts various composition methods and their aesthetic evaluation values, outputting the composition box corresponding to the composition method with the highest aesthetic evaluation value. This composition box is used as the boundary of the target image region. Finally, a second target image of the target object is obtained by cropping from the first target image based on the boundary of the target image region.

[0075] Understandably, by processing the baseline image region containing the target object using a target model, an image with the optimal composition for the target object can be obtained. Users can obtain images of the target object that meet their expectations without needing to learn professional aesthetics, thus improving the user experience.

[0076] Figure 3 One of the schematic diagrams of an image processing method according to an embodiment of the present disclosure is shown.

[0077] As described above, the operation involves determining a reference image region in the first target image that satisfies the target conditions. In one possible implementation, this operation may further include determining an initial image region containing the target object in the first target image; scaling the region boundary of the initial image region; and determining a reference image region in the first target image that satisfies the target conditions.

[0078] For example, the initial image region can be either the minimum or maximum bounding rectangle of the target object. If the initial image region is the minimum bounding rectangle of the target object, the base image region is an image region obtained by enlarging the initial image region. If the initial image region is the maximum bounding rectangle of the target object, the base image region is an image region obtained by shrinking the initial image region. The criterion for scaling the initial image region is that the salience of the target object in the scaled image region is higher than the target salience.

[0079] In one example, refer to Figure 3After determining the target object to be a kitten based on the user's interaction with the first target image, a minimum bounding rectangle corresponding to the kitten is generated in the first target image as the initial image region. The initial image region is then enlarged to obtain a reference image region, which contains the target object, and the salience of the target object in the reference image region is greater than the target salience (e.g., 50%). The image data corresponding to the reference image region is then input into the target model. The target model predicts various composition methods and their aesthetic evaluation values, outputting the composition box corresponding to the composition method with the highest aesthetic evaluation value. This composition box is used as the boundary of the target image region. A second target image of the target object is then obtained by cropping the first target image according to the boundary of the target image region. Different composition boxes correspond to different composition image regions.

[0080] To ensure the target model can predict an image region containing the selected target object, the initial image region is scaled to obtain a baseline image region. The target object is positioned as centrally as possible within this baseline image region, with sufficient space around it to accommodate various compositional needs. Using the baseline image region containing the target object in the first target image for composition prediction ensures that the composition targeting the target object is less affected by other significant objects.

[0081] In some embodiments, the scaling process has a target parameter that characterizes a scalable boundary value for scaling, which is related to the size of the initial image region.

[0082] For example, the parameter can be a range for scaling the initial image region, with a maximum scaling value and a minimum scaling value, the maximum scaling value corresponding to the largest scalable boundary value and the minimum scaling value corresponding to the smallest scalable boundary value. The target parameter is a scaling value between the largest and smallest scalable boundary values.

[0083] The scalable boundary value can be the distance that the boundary of the initial image region expands outwards in all directions.

[0084] In one example, the minimum bounding rectangle of the target object, a kitten, has a length of W and a width of H. Therefore, the size of the minimum bounding rectangle is W*H. Assume the expansion factor is r, with a value ranging from [0.5, 1]. Then, the maximum scalable boundary value dmax = MAX(W, H)*r. Establish a coordinate system with the lower left corner of the first target image as the origin. The lower left corner of the minimum bounding rectangle is (x1, y1), and the upper right corner is (x2, y2). Expand the minimum bounding rectangle horizontally and vertically by d to obtain the reference image region. The minimum scalable boundary value dmin is a value greater than 0.

[0085] In some embodiments, the reference image region is an image region obtained by scaling an initial image region with a target scaling value, where the target scaling value is related to the salience of the target object in the reference image region.

[0086] The target scaling value is located within the scalable boundary value corresponding to the parameter, and the target scaling value is a certain scalable boundary value between the largest and smallest scalable boundary values.

[0087] For example, the target scaling value can be the scaling value corresponding to when the salience of the target object in the reference image region is higher than the target salience.

[0088] The initial image region is scaled to different degrees (i.e., different scaling values ​​are used). When the salience of the target object in the image region of different scaling degrees is higher than the target salience, the scaling value is taken as the target scaling value, and the image region corresponding to the scaling value is taken as the reference image region.

[0089] In some embodiments, if the salience of the target object in the intermediate image region is higher than the target salience, the intermediate image region with the largest size is used as the reference image region; the intermediate image region is the image region formed by scaling the initial image region with different target scaling values.

[0090] In one example, during the scaling of the initial image region, different scaling values ​​are adjusted to obtain intermediate image regions of different sizes. When the salience of the target object in the intermediate image region is greater than that of non-target objects, and the salience of the target object in the intermediate image region is at its maximum salience, the scaling value corresponding to this intermediate image region is used as the target scaling value, and this intermediate image region is used as the reference image region.

[0091] Figure 4 A second schematic diagram of an image processing method according to an embodiment of the present disclosure is shown.

[0092] As described above, the operation involves scaling the region boundaries of an initial image region to determine a reference image region that meets the target conditions. In one possible implementation, this operation may further include: scaling the initial image region by a first scaling value to obtain a first intermediate image region; if the first intermediate image region meets the target conditions, scaling the first intermediate image region by a second scaling value to obtain a second intermediate image region; if the second intermediate image region does not meet the target conditions, determining the first intermediate image region as the reference image region, where the second scaling value is greater than the first scaling value.

[0093] In one example, refer to Figure 4Given that the initial image region is the minimum bounding rectangle of the target object (kitten), the size of the minimum bounding rectangle is 5*10. Assuming the expansion coefficient r is 0.8 and the maximum scalable boundary value dmax is 8, the scaling value d ranges from (0, 8). Within this range, different scaling values ​​are used to scale the initial image region. When the initial image region is expanded with the first scaling value d1=3, the coordinates of the lower left corner of the minimum bounding rectangle are (x1, y1), and the coordinates of the upper right corner are (x2, y2). Expanding the minimum bounding rectangle left and right by d1 and up and down by d1 respectively yields the first intermediate image region. Since the little girl (non-target object) is not in the first intermediate image region, and the puppy (non-target object) is partially in the first intermediate region, the salience of the target object (kitten) in the first intermediate image region is greater than the salience of the non-target objects (puppy and little girl), thus satisfying the target condition. Therefore, the target scaling value is increased to the second scaling value. When the initial image region is expanded with a value of d2=4, the coordinates of the lower left corner of the minimum bounding rectangle are (x1, y1), and the coordinates of the upper right corner are (x2, y2). The minimum bounding rectangle is expanded left and right by d2, and up and down by d2, to obtain the second intermediate image region. At this point, since the little girl (not the target object) is not in the second intermediate image region, and the puppy (not the target object) is within it, and the area of ​​the puppy in the second intermediate region is greater than the area of ​​the kitten, the salience of the target object (kitten) in the second intermediate image region is less than the salience of the non-target object (puppy). Therefore, the target condition is not met, and the first scaling value is used as the target scaling value, and the first intermediate image region corresponding to the first scaling value is used as the reference image region.

[0094] As described above, the image processing method of this embodiment may further include the operation: if the boundary of the reference image region exceeds the boundary of the first target image, the boundary where the first target image and the reference image region overlap is taken as the boundary of the reference image region.

[0095] In one example, continue to refer to Figure 4After scaling the initial image region with a first scaling value, if the target object in the first intermediate image region meets the target condition, and scaling continues with a second scaling value, if the first edge (bottom edge) of the boundary of the second intermediate image region exceeds the bottom boundary of the first target image, then the overlapping portion of the bottom boundary of the first target image and the first edge (bottom edge) of the second intermediate image region is taken as the bottom boundary of the second intermediate image region. Similarly, if the second edge (top edge) of the boundary of the second intermediate image region exceeds the top boundary of the first target image, then the overlapping portion of the top boundary of the first target image and the second edge (top edge) of the second intermediate image region is taken as the top boundary of the second intermediate image region. Finally, if the third edge (left edge) of the boundary of the second intermediate image region exceeds the left boundary of the first target image, then the overlapping portion of the left boundary of the first target image and the third edge (left edge) of the second intermediate image region is taken as the left boundary of the second intermediate image region.

[0096] As described above, in operation S220, in response to a target interaction operation on the first target image, the target object indicated by the target interaction operation is determined. In one possible implementation, the operation may further include: dividing the first target image into multiple image regions with semantic labels, each image region having a different target object, the semantic labels representing the target objects in the image regions; and in response to clicking at least one target location in the first target image, determining the target object of the image region corresponding to each target location.

[0097] For example, semantic labels can be identifiers used to distinguish different target objects in an image. For instance, when the types of target objects differ in different regions of a first target image—for example, if the first region contains a kitten, the semantic label could be "kitten"—and the second region contains a little girl, the semantic label could be "little girl." When the types of target objects are the same in different regions of the first target image—for example, if the first region contains a blue kitten and the second region contains a yellow kitten—the first semantic label could be "blue kitten," and the second semantic label could be "yellow kitten." That is, semantic labels for different target objects can be used to distinguish multiple target objects.

[0098] In one example, the first target image is divided into several distinct regions. Each region contains a target object, and each region is identified by a "semantic label." The semantic label for the first region is "person." The semantic label for the second region is "car." The semantic label for the third region is "tree." When a user briefly touches the location of the tree on the screen displaying the first target image (the target location), since the coordinates of the target location fall within the coordinate range of the object "tree," the target object selected by the user in the first target image is determined to be "tree."

[0099] Understandably, semantic segmentation technology divides an image into different regions, each labeled with a different object. The system can then respond to user clicks and determine the object corresponding to the clicked location as the target object. Through user interaction with the image, the system quickly identifies the target object in the image that matches the user's intent.

[0100] Based on the above image processing method, this disclosure also provides an image processing apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0101] Figure 5 A schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0102] like Figure 5 As shown, the image processing apparatus 300 of this embodiment includes a display module 310, a determination module 320, and a cropping module 330.

[0103] The display module 310 is used to display the first target image. In one embodiment, the display module 310 can be used to perform the operation S210 described above, which will not be repeated here.

[0104] The determining module 320 is configured to determine the target object indicated by the target interaction operation in response to a target interaction operation on the first target image. In one embodiment, the determining module 320 may be configured to perform the operation S220 described above, which will not be repeated here.

[0105] The cropping module 330 is used to crop the first target image based on the target object to obtain a second target image. In one embodiment, the cropping module 330 can be used to perform the operation S230 described above, which will not be repeated here.

[0106] The second target image contains a target object that meets the target composition conditions.

[0107] According to embodiments of this disclosure, any plurality of modules among the display module 310, the determining module 320, and the trimming module 330 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the display module 310, the determining module 320, and the trimming module 330 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the display module 310, the determining module 320, and the trimming module 330 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0108] Based on the above image processing method, this disclosure also provides an electronic device. The following will be combined with... Figure 6 Provide a detailed description of the electronic device.

[0109] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0110] like Figure 6 As shown, the electronic device 400 of this embodiment includes: a display screen 410 for displaying an image; a processor 420 for displaying a first target image on the display screen 410; determining a target object indicated by a target interaction operation in response to a target interaction operation on the first target image; cropping the first target image based on the target object to obtain a second target image; wherein the second target image contains a target object that satisfies the target composition conditions.

[0111] For the first target image, the second target image, etc., the above can be referred to. Figure 2 The description in the text will not be repeated here.

[0112] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure.

[0113] like Figure 7As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0114] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0115] According to embodiments of this disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0116] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0117] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0118] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the image processing methods provided in the embodiments of this disclosure.

[0119] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0120] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0121] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0122] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0124] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0125] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image processing method, comprising: Display the first target image; In response to a target interaction operation on the first target image, determine the target object indicated by the target interaction operation; The first target image is cropped based on the target object to obtain the second target image; The second target image contains the target object that satisfies the target composition conditions.

2. The method according to claim 1, wherein cropping the first target image based on the target object to obtain the second target image comprises: In the first target image, determine a plurality of candidate image regions containing the target object; A target image region containing the target object that satisfies the target composition conditions is determined from the candidate image regions. The target image region is cropped from the first target image to obtain the second target image.

3. The method according to claim 1, wherein cropping the first target image based on the target object to obtain the second target image comprises: In the first target image, a reference image region that meets the target conditions is determined; The image data of the reference image region is input into the target model to obtain a target image region containing the target object that satisfies the target composition conditions; The target image region is cropped from the first target image to obtain the second target image; The target condition indicates that the salience of the target object in the reference image region is higher than the target salience.

4. The method according to claim 3, wherein determining the reference image region satisfying the target condition in the first target image comprises: Determine an initial image region containing the target object in the first target image; The region boundaries of the initial image region are scaled to determine a reference image region in the first target image that meets the target conditions.

5. The method according to claim 4, wherein the scaling process has a target parameter, the parameter representing a scalable boundary value for performing the scaling process, the scalable boundary value being related to the size of the initial image region.

6. The method according to claim 4 or 5, wherein the reference image region is an image region obtained by scaling the initial image region with a target scaling value, and the target scaling value is related to the salience of the target object in the reference image region.

7. The method according to claim 6, wherein if the salience of the target object in the intermediate image region is higher than the target salience, the intermediate image region with the largest size is used as the reference image region; the intermediate image region is an image region formed by scaling the initial image region with different target scaling values.

8. The method according to claim 6, wherein scaling the region boundary of the initial image region to determine a reference image region that satisfies the target conditions includes: The initial image region is scaled using a first scaling value to obtain a first intermediate image region; If the first intermediate image region meets the target condition, the first intermediate image region is scaled with a second scaling value to obtain a second intermediate image region; If the second intermediate image region does not meet the target condition, the first intermediate image region is determined as the reference image region, and the second scaling value is greater than the first scaling value.

9. The method according to claim 2, further comprising: If the boundary of the reference image region exceeds the boundary of the first target image, the boundary where the first target image and the reference image region overlap is taken as the boundary of the reference image region.

10. The method of claim 1, wherein in response to a target interaction operation on the first target image, determining the target object indicated by the target interaction operation comprises: The first target image is divided into multiple image regions with semantic labels, each of which has a different target object, and the semantic labels represent the target objects in the image regions. In response to clicking at least one target location in the first target image, a target object in the image region corresponding to each target location is determined.

Images