Image collage model training method and device and image collage method and device

Abstract

The present disclosure provides a training method and device for an image collage model and an image collage method and device. The training method comprises: obtaining a training sample including a plurality of sub-images, and collage the plurality of sub-images into one image to obtain an initial collage image to be adjusted; obtaining the aesthetic features of the current collage image to be adjusted; inputting the obtained aesthetic features into the image collage model to obtain a predicted collage adjustment action sequence to be executed, executing the predicted collage adjustment action sequence on the current collage image to be adjusted, and returning to execute the step of obtaining the aesthetic features of the current collage image to be adjusted; when it is determined not to continue adjusting the collage image, adjusting the parameters of the image collage model according to the reward function of executing each predicted collage adjustment action sequence to train the image collage model.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing technology, and more specifically, to a training method and device for an image collage model, and an image collage method and device. Background Art

[0002] Image collage technology refers to the technology of collaging multiple images or multiple key frames in a video into one image. Image collage is not only beautiful, but also contains a lot of information. It can attract the user's attention in a short time and convey more information. It can be applied to a variety of business scenarios.

[0003] Most existing image collage technologies have high production costs, and some beautiful collage effects even require highly skilled designers to manually process them. Automatic image collage algorithms can significantly reduce the production costs of image collages. Users only need to input videos or multiple images to complete the image collage with one click. However, there are few studies on automatic image collages, and the quality of image collages is poor. Summary of the invention

[0004] Exemplary embodiments of the present disclosure provide a method and device for training an image collage model and an image collage method and device, so as to at least solve the problems in the above-mentioned related technologies, or may not solve any of the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for an image collage model is provided, the training method comprising: obtaining a training sample including a plurality of sub-images, and collaging the plurality of sub-images into one image to obtain an initial collage image to be adjusted; obtaining aesthetic features of the current collage image to be adjusted; inputting the obtained aesthetic features into the image collage model to obtain a predicted collage adjustment action sequence to be executed, executing the predicted collage adjustment action sequence on the current collage image to be adjusted, and returning to execute the step of obtaining the aesthetic features of the current collage image to be adjusted; when it is determined not to continue adjusting the collage image, adjusting the parameters of the image collage model according to a reward function for executing each predicted collage adjustment action sequence to train the image collage model.

[0006] Optionally, the acquired aesthetic features are input into the image collage model to obtain a predicted collage adjustment action sequence that needs to be executed, the predicted collage adjustment action sequence is executed on the current collage image to be adjusted, and the step of acquiring the aesthetic features of the current collage image to be adjusted is returned, including: inputting the acquired aesthetic features into the image collage model to obtain the predicted collage adjustment action sequence that needs to be executed; executing the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the current collage adjustment; cropping the collage image after the current collage adjustment according to a specified aspect ratio; determining whether to continue adjusting the collage image, wherein when it is determined to continue adjusting the collage image, the collage image obtained by this cropping is used as the current collage image to be adjusted, and the step of acquiring the aesthetic features of the current collage image to be adjusted is returned.

[0007] Optionally, the step of obtaining the aesthetic features of the current collage image to be adjusted includes: inputting the current collage image to be adjusted into an aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; fusing the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0008] Optionally, the step of fusing the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted includes: fusing the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the current collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the current collage image to be adjusted.

[0009] Optionally, the step of fusing the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted includes: fusing the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics to obtain the aesthetic features of the current collage image to be adjusted.

[0010] Optionally, the types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; wherein, the types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; wherein, the types of local adjustment actions include at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0011] Optionally, the reward function for executing each predicted collage adjustment action sequence is calculated in the following manner: based on the difference between the aesthetic score of the collage image obtained after executing the currently predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the previously predicted collage adjustment action sequence, the reward function for executing the currently predicted collage adjustment action sequence is calculated.

[0012] Optionally, the reward function for executing the t-th predicted collage adjustment action sequence is calculated based on: t, the difference between the aesthetic score of the collage image obtained after executing the t-th predicted collage adjustment action sequence, and the aesthetic score of the collage image obtained after executing the t-1-th predicted collage adjustment action sequence.

[0013] Optionally, the aesthetic score of the collage image is calculated based on the number of aesthetic boxes of the collage image and / or the blank loss area of ​​the collage image.

[0014] Optionally, the step of cropping the collage image adjusted by this collage according to a specified aspect ratio includes: cropping a plurality of candidate views that meet the specified aspect ratio from the collage image adjusted by this collage; inputting the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; and using the candidate view with the highest aesthetic score among the plurality of candidate views as the collage image obtained by this cropping.

[0015] Optionally, the step of fusing the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted includes: weighting the aesthetic features of the multiple image blocks based on the Euclidean distance of the image block from the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted to obtain the aesthetic features of the collage image to be adjusted.

[0016] According to a second aspect of an embodiment of the present disclosure, there is provided an image collage method, the method comprising: obtaining a plurality of sub-images to be collaged, and collaging the plurality of sub-images to be collaged into one image to obtain an initial collage image to be adjusted; obtaining aesthetic features of the current collage image to be adjusted; inputting the obtained aesthetic features into an image collage model to obtain a predicted collage adjustment action sequence to be executed, executing the predicted collage adjustment action sequence on the current collage image to be adjusted, and returning to execute the step of obtaining the aesthetic features of the current collage image to be adjusted; when it is determined not to continue adjusting the collage image, outputting the adjusted collage image, wherein the image collage model is trained in the following manner: adjusting parameters of the image collage model based on a reward function for executing each predicted collage adjustment action sequence of the image collage model on a training sample.

[0017] Optionally, the acquired aesthetic features are input into the image collage model to obtain a predicted collage adjustment action sequence to be executed, the predicted collage adjustment action sequence is executed on the current collage image to be adjusted, and the step of acquiring the aesthetic features of the current collage image to be adjusted is returned, including: inputting the acquired aesthetic features into the image collage model to obtain the predicted collage adjustment action sequence to be executed; executing the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the current collage adjustment; cropping the collage image after the current collage adjustment according to the aspect ratio specified by the user; determining whether to continue adjusting the collage image, wherein when it is determined to continue adjusting the collage image, the collage image obtained by this cropping is used as the current collage image to be adjusted, and the step of acquiring the aesthetic features of the current collage image to be adjusted is returned; wherein, when it is determined not to continue adjusting the collage image, the step of outputting the adjusted collage image includes: when it is determined not to continue adjusting the collage image, outputting the collage image obtained by this cropping.

[0018] Optionally, the step of obtaining the aesthetic features of the current collage image to be adjusted includes: inputting the current collage image to be adjusted into an aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; fusing the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0019] Optionally, the step of fusing the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted includes: fusing the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the current collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the current collage image to be adjusted.

[0020] Optionally, the step of fusing the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted includes: fusing the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics to obtain the aesthetic features of the current collage image to be adjusted.

[0021] Optionally, the types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; wherein, the types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; wherein, the types of local adjustment actions include at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0022] Optionally, the step of cropping the collage image adjusted by this collage according to the aspect ratio specified by the user includes: cropping a plurality of candidate views that meet the aspect ratio specified by the user from the collage image adjusted by this collage; inputting the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; and using the candidate view with the highest aesthetic score among the plurality of candidate views as the collage image obtained by this cropping.

[0023] Optionally, the step of fusing the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted includes: weighting the aesthetic features of the multiple image blocks based on the Euclidean distance of the image block from the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted to obtain the aesthetic features of the collage image to be adjusted.

[0024] Optionally, the image collage model is trained using the training method described above.

[0025] According to a third aspect of an embodiment of the present disclosure, a training device for an image collage model is provided, the training device comprising: an initial collage unit, configured to obtain a training sample including a plurality of sub-images, and collage the plurality of sub-images into one image to obtain an initial collage image to be adjusted; an aesthetic feature acquisition unit, configured to obtain aesthetic features of a current collage image to be adjusted; a prediction adjustment unit, configured to input the acquired aesthetic features into the image collage model, obtain a predicted collage adjustment action sequence to be executed, execute the predicted collage adjustment action sequence on the current collage image to be adjusted, and return the aesthetic features of the current collage image to be adjusted acquired by the aesthetic feature acquisition unit; and a training unit, configured to, when determining not to continue adjusting the collage image, adjust the parameters of the image collage model according to a reward function for executing each predicted collage adjustment action sequence, so as to train the image collage model.

[0026] Optionally, the prediction adjustment unit includes: a prediction unit, configured to input the acquired aesthetic features into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction; a collage adjustment unit, configured to execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after this collage adjustment; a cropping unit, configured to crop the collage image after this collage adjustment according to a specified aspect ratio; and an adjustment end determination unit, configured to determine whether to continue adjusting the collage image, wherein, when it is determined to continue adjusting the collage image, the aesthetic feature acquisition unit uses the collage image obtained by this cropping as the current collage image to be adjusted, and acquires the aesthetic features of the current collage image to be adjusted.

[0027] Optionally, the aesthetic feature acquisition unit is configured to input the current collage image to be adjusted into an aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and fuse the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0028] Optionally, the aesthetic feature acquisition unit is configured to fuse the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

[0029] Optionally, the aesthetic feature acquisition unit is configured to perform a fusion process on the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics, so as to obtain the aesthetic features of the current collage image to be adjusted.

[0030] Optionally, the types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; wherein, the types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; wherein, the types of local adjustment actions include at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0031] Optionally, the reward function for executing each predicted collage adjustment action sequence is calculated in the following manner: based on the difference between the aesthetic score of the collage image obtained after executing the currently predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the previously predicted collage adjustment action sequence, the reward function for executing the currently predicted collage adjustment action sequence is calculated.

[0032] Optionally, the reward function for executing the t-th predicted collage adjustment action sequence is calculated based on: t, the difference between the aesthetic score of the collage image obtained after executing the t-th predicted collage adjustment action sequence, and the aesthetic score of the collage image obtained after executing the t-1-th predicted collage adjustment action sequence.

[0033] Optionally, the aesthetic score of the collage image is calculated based on the number of aesthetic boxes of the collage image and / or the blank loss area of ​​the collage image.

[0034] Optionally, the cropping unit is configured to crop a plurality of candidate views that meet the specified aspect ratio from the collage image after this collage adjustment; input the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; and use the candidate view with the highest aesthetic score among the plurality of candidate views as the collage image obtained by this cropping.

[0035] Optionally, the aesthetic feature acquisition unit is configured to perform weighted processing on the aesthetic features of the multiple image blocks based on the Euclidean distance between the image block and the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted, so as to obtain the aesthetic features of the collage image to be adjusted.

[0036] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image collage device, comprising: an initial collage unit, configured to obtain a plurality of sub-images to be collaged, and collage the plurality of sub-images to be collaged into one image to obtain an initial collage image to be adjusted; an aesthetic feature acquisition unit, configured to obtain aesthetic features of a current collage image to be adjusted; a prediction adjustment unit, configured to input the acquired aesthetic features into an image collage model, obtain a predicted collage adjustment action sequence to be executed, execute the predicted collage adjustment action sequence on the current collage image to be adjusted, and return the aesthetic features of the current collage image to be adjusted acquired by the aesthetic feature acquisition unit; an output unit, configured to output an adjusted collage image when it is determined that the collage image is not to be adjusted further, wherein the image collage model is trained in the following manner: adjusting parameters of the image collage model based on a reward function for executing the collage adjustment action sequence predicted by the image collage model each time for a training sample.

[0037] Optionally, the prediction and adjustment unit includes: a prediction unit, configured to input the acquired aesthetic features into the image collage model to obtain a collage adjustment action sequence that needs to be executed for this prediction; a collage adjustment unit, configured to execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after this collage adjustment; a cropping unit, configured to crop the collage image after this collage adjustment according to an aspect ratio specified by a user; an adjustment end determination unit, configured to determine whether to continue adjusting the collage image, wherein, when it is determined to continue adjusting the collage image, the aesthetic feature acquisition unit uses the collage image obtained by this cropping as the current collage image to be adjusted, and obtains the aesthetic features of the current collage image to be adjusted; wherein the output unit is configured to output the collage image obtained by this cropping when it is determined not to continue adjusting the collage image.

[0038] Optionally, the aesthetic feature acquisition unit is configured to input the current collage image to be adjusted into an aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and fuse the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0039] Optionally, the aesthetic feature acquisition unit is configured to fuse the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

[0040] Optionally, the aesthetic feature acquisition unit is configured to perform a fusion process on the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics, so as to obtain the aesthetic features of the current collage image to be adjusted.

[0041] Optionally, the types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; wherein, the types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; wherein, the types of local adjustment actions include at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0042] Optionally, the cropping unit is configured to: crop out a plurality of candidate views that meet a user-specified aspect ratio from the collage image after this collage adjustment; input the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; and use the candidate view with the highest aesthetic score among the plurality of candidate views as the collage image obtained by this cropping.

[0043] Optionally, the aesthetic feature acquisition unit is configured to perform weighted processing on the aesthetic features of the multiple image blocks based on the Euclidean distance between the image block and the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted, so as to obtain the aesthetic features of the collage image to be adjusted.

[0044] Optionally, the image collage model is trained using the training device as described above.

[0045] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method of the image collage model as described above and / or the image collage method as described above.

[0046] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the image collage model as described above and / or the image collage method as described above.

[0047] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the image collage model training method and / or the image collage method as described above.

[0048] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: it can automatically produce a high-quality collage image with a specific aspect ratio.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0051] Figure 1 A flowchart showing a method for training an image collage model according to an exemplary embodiment of the present disclosure;

[0052] Figure 2 A flowchart showing a method for training an image collage model according to another exemplary embodiment of the present disclosure;

[0053] Figure 3 An example of a collage adjustment action according to an exemplary embodiment of the present disclosure is shown;

[0054] Figure 4 A flow chart showing the method of obtaining the aesthetic features of the collage image to be adjusted according to an exemplary embodiment of the present disclosure is shown;

[0055] Figure 5 A flowchart showing an image collage method according to an exemplary embodiment of the present disclosure is shown;

[0056] Figure 6 A flowchart showing an image collage method according to another exemplary embodiment of the present disclosure;

[0057] Figure 7 A structural block diagram showing a training device for an image collage model according to an exemplary embodiment of the present disclosure;

[0058] Figure 8 A structural block diagram showing an image collage device according to an exemplary embodiment of the present disclosure is shown;

[0059] Fig. 9 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0060] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0063] Figure 1 A flowchart of a method for training an image collage model according to an exemplary embodiment of the present disclosure is shown.

[0064] Reference Figure 1 In step S101, a training sample including a plurality of sub-images is obtained, and the plurality of sub-images are collaged into one image to obtain an initial collage image to be adjusted. In other words, the one image obtained by collage is the initial collage image to be adjusted.

[0065] As an example, the multiple sub-images may be multiple separate images, or multiple key frames in the same video. For example, the training sample may include a video key frame sequence.

[0066] As an example, the multiple sub-images may be collaged into one image according to the order of the multiple sub-images (for example, the order of the multiple sub-images in a sequence).

[0067] As an example, a preset appropriate template may be used to collage the multiple sub-images into one image. For example, the template may be a grid-shaped template.

[0068] In step S102, the aesthetic features of the collage image to be adjusted are obtained.

[0069] It should be understood that when step S102 is performed for the first time for a certain training sample, the current collage image to be adjusted is the initial collage image to be adjusted. That is, the initial value of the current collage image to be adjusted is: the initial collage image to be adjusted.

[0070] It should be understood that various appropriate methods can be used to obtain the aesthetic features of the current collage image to be adjusted. Figure 4 An exemplary embodiment of step S102 is described below.

[0071] In step S103, the acquired aesthetic features are input into the image collage model to obtain a predicted collage adjustment action sequence to be executed, the predicted collage adjustment action sequence is executed on the current collage image to be adjusted, and the process returns to step S102. Specifically, when it is determined to continue adjusting the collage image, the process returns to step S102.

[0072] In step S104, when it is determined not to continue adjusting the collage image, the parameters of the image collage model are adjusted according to the reward function of executing each predicted collage adjustment action sequence to train the image collage model.

[0073] In other words, before determining not to continue adjusting the collage image, step S102 and step S103 are iteratively executed, that is, after each execution of step S103, the execution returns to step S102 based on the collage image adjusted this time.

[0074] The following will be combined Figure 2 An exemplary embodiment of step S103 is described below.

[0075] Reference Figure 2 In step S1031, the acquired aesthetic features are input into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction.

[0076] As an example, the image collage model may be constructed based on a reinforcement learning algorithm.

[0077] As an example, the aesthetic features obtained by each execution of step S102 before executing step S1031 can be input into the image collage model to obtain the collage adjustment action sequence that needs to be executed this time. t ={o0,o1,…,o t} Input the image collage model, so that the image collage model (ie, the agent) determines which collage adjustment action or actions need to be performed next according to the current state. Here, o trepresents the observation value at the current time t, that is, the aesthetic feature of the collage image obtained at the current time t, that is, the aesthetic feature of the current collage image to be adjusted obtained by executing step S102 this time. Accordingly, o t-1 It is the aesthetic feature of the collage image obtained by performing step S102 last time.

[0078] As an example, the action space may be composed of a series of predefined actions, and the overall layout of the collage and / or the local position details may be adjusted according to the actual scenario.

[0079] As an example, the types of collage adjustment actions may include: layout adjustment actions and / or local adjustment actions. For example, a collage position adjustment action for a plurality of images may be considered a layout adjustment action. For example, a collage position adjustment action for a single image may be considered a local adjustment action.

[0080] As an example, the types of layout adjustment actions may include: exchanging positions of two sub-images and terminating the layout adjustment action.

[0081] Specifically, exchanging the positions of the two sub-images means exchanging the positions of the two sub-images in the collage image. Terminating the layout adjustment action means not performing the overall layout adjustment any more.

[0082] As an example, the type of local adjustment action includes at least one of the following items: adjusting the sub-image placement position, not adjusting the sub-image placement position, rotating the sub-image placement angle, not adjusting the sub-image placement angle, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0083] Figure 3 An example of a collage adjustment action according to an exemplary embodiment of the present disclosure is shown. Figure 3 As shown, adjusting the placement of the sub-image may include: moving the sub-image horizontally / vertically over a short distance.

[0084] As an example, a layout adjustment action sequence that needs to be executed may be predicted first, and when the predicted tiling adjustment action sequence that needs to be executed includes a termination layout adjustment action, a local adjustment action sequence that needs to be executed is predicted.

[0085] As an example, the action space for layout adjustment may include: swap operation and termination action. When the agent performs the swap operation, it swaps the positions of two sub-images to generate a better composition faster. The termination layout operation is the trigger of the agent. After selecting this operation, the agent will stop the global optimization process and no longer adjust the layout. For detailed local adjustments, the agent will operate on different sub-images, and the attributes include layers, short-distance relative horizontal / vertical movement, and rotation angle.

[0086] In step S1032, the predicted collage adjustment action sequence is executed on the current collage image to be adjusted to obtain the collage image after the current collage adjustment.

[0087] In step S1033, the collage image after the current collage adjustment is cropped according to the specified aspect ratio.

[0088] As an example, a plurality of candidate views conforming to the specified aspect ratio may be cropped from the collage image after the current collage adjustment; the plurality of candidate views are input into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; and the candidate view with the highest aesthetic score among the plurality of candidate views is used as the collage image obtained by this cropping.

[0089] As an example, the aesthetic scoring network may be a lightweight model View Proposal Net (VPN).

[0090] In step S1034, it is determined whether to continue adjusting the collage image. If it is determined to continue adjusting the collage image, the collage image obtained by this cropping is used as the current collage image to be adjusted, and the process returns to step S102 to enter the next collage adjustment iteration.

[0091] As an example, the purpose of cropping the collage image after this collage adjustment is to make the irregularly shaped collage image fit the specified aspect ratio canvas. The aspect ratio information is the environmental information, and feedback is provided to the agent after each collage image adjustment. Specifically, after the agent adjusts the collage image each time (that is, each step), multiple candidate views are automatically cropped from the adjusted collage image to adapt to the canvas aspect ratio information. After the candidate views pass through the evaluation network (e.g., the aesthetic scoring network), the one with the highest score will transition to the next iteration. Therefore, step S1033 can encourage the agent to choose to expand the salient information and suppress the blank space of the canvas every time it interacts with the environment.

[0092] It should be understood that the present disclosure does not limit the execution order of step S1034 , for example, it can be executed after step S1032 .

[0093] As an example, whether to continue adjusting the collage image may be determined based on the number of times the predicted collage adjustment action sequence is executed (i.e., the number of times step S1032 is executed), wherein when the number of times the predicted collage adjustment action sequence is executed reaches a preset threshold, it is determined not to continue adjusting the collage image. For example, the preset threshold may be 6.

[0094] It should be understood that other appropriate methods may be used to determine whether to continue adjusting the collage image (ie, determine whether to end the training round for the training sample), and the present disclosure is not limited thereto.

[0095] When it is determined in step S1034 that the collage image is not to be adjusted further, step S104 is executed to adjust the parameters of the image collage model according to the reward function of executing each predicted collage adjustment action sequence to train the image collage model.

[0096] In other words, the image collage model is trained by adjusting the parameters of the image collage model according to the reward function of each previously executed predicted collage adjustment action sequence.

[0097] As an example, a reward function for executing each predicted tile adjustment action sequence is calculated, that is, the reward for executing each predicted tile adjustment action sequence is obtained by calculating the reward function.

[0098] As an example, the reward function for executing each predicted collage adjustment action sequence can be calculated in the following way: based on the difference between the aesthetic score of the collage image obtained after executing the currently predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the previously predicted collage adjustment action sequence, the reward function for executing the currently predicted collage adjustment action sequence is calculated.

[0099] As an example, the reward function for executing the tth predicted collage adjustment action sequence can be calculated based on the following items: t, the difference between the aesthetic score of the collage image obtained after executing the tth predicted collage adjustment action sequence, and the aesthetic score of the collage image obtained after executing the t-1th predicted collage adjustment action sequence.

[0100] As an example, the reward function r for performing the t-th predicted tile adjustment action sequence is t (C t ) can be calculated by the following formula:

[0101] r t (C t ) = r′ t (C t )-0.01*(t+1),

[0102] Among them, r′ t (C t ) represents: the difference between the aesthetic score of the collage image obtained after executing the t-th predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the t-1-th predicted collage adjustment action sequence.

[0103] As an example, the aesthetic score of the collage image may be calculated based on the number of aesthetic boxes of the collage image and / or the white space loss area of ​​the collage image.

[0104] The image collage model aims to automatically adjust the original collage image to generate the collage image that best meets the public's aesthetic taste. Therefore, the reward function should enable the agent to obtain reward information every time it makes an adjustment, calculate the reward function to give the agent sufficient feedback information, and finally accumulate the reward score to produce a more aesthetically pleasing adjustment method.

[0105] The aesthetic scoring network disclosed in the present invention can use the lightweight model View Proposal Net (VPN) to perform aesthetic scoring on collage images. In order to accelerate model convergence and ensure learning stability, the present invention can use collage C t The number of aesthetic boxes a (C t ) as a supervisory signal and add collage C t Blank Loss b (C t ) to increase the compactness of the picture content. During the training process, the recommended number of aesthetic boxes (the number of aesthetic boxes is the intermediate output of the aesthetic scoring network, and the aesthetic score is the final output of the aesthetic scoring network) can be used as a simplified version of the agent supervision to ensure stability:

[0106] s t (C t )=λ a s a (C t )-λ b s b (C t )

[0107] Each time a predicted sequence of tile adjustment actions is executed, the reward for this action can be calculated using the difference between the aesthetic score of the newly adjusted tile image and the tile image obtained by the previous execution of the predicted sequence of tile adjustment actions:

[0108] r′ t (C t )=s t+1 (C t+1 )-s t (C t )

[0109] Finally, since the reward scheme implicitly considers the number of steps as a cost, the agent should follow a greedy strategy to avoid unnecessary actions, thereby reducing the accumulated utility and speeding up the tiling optimization process:

[0110] r t (C t ) = r′ t (C t )-0.01*(t+1)

[0111] Where t represents the number of steps the agent has taken since the beginning to help speed up tile generation. This constraint will reduce the overall reward as the number of steps the agent takes increases.

[0112] As an example, after each execution of step S1032, the reward function for that time may be calculated.

[0113] As an example, the network structure of the image collage model may include: a strategy network θ p and value network θ v Policy Network θ p and value network θ v The training of can follow the advantage actor-critic (A2C) algorithm. Both networks use the current state s at time t t As input, the value network outputs the value representing the state s t The expected total reward after that is used to measure how good the current state is. Therefore, the total reward R at a certain time t t Expressed as Where V(s t +k ) represents the estimated value of the total reward of the value network at time t+k. The decay factor is represented by γ, and the closer to time t, the greater the value of R t The greater the contribution of . The gradient of the value network is The value network estimates the reward V(s) at time t (t) ) to be equal to the actual subsequent reward sum R at time t t Try to get as close as possible, use the difference between the two as the loss, and update along the loss gradient to make the valuation network more accurate.

[0114] The output of the policy network is the collage image s at the current time t. t The probability distribution of taking corresponding adjustment actions in the state, the best action is the action with the largest probability distribution, which can be t The best target operation to achieve a collage result with a high aesthetic quality score is to obtain the maximum subsequent reward sum estimate in the state. The following formula can be followed when updating the policy network gradient:

[0115]

[0116] Among them, A(a (t) ,s (t) )=R (t) -V(s (t) ) represents the actual subsequent reward sum R at time t t and the estimated reward value V(s) of the value network at time t (t) ), π(a (t) |s(t) ) represents the action probability distribution output by the policy network.

[0117] Furthermore, it should be understood that the image collage model may be trained using multiple samples.

[0118] Figure 4 A flow chart of obtaining the aesthetic features of a collage image to be adjusted according to an exemplary embodiment of the present disclosure is shown.

[0119] like Figure 4 As shown, in step S1021, the current collage image to be adjusted is input into the aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted.

[0120] As an example, in order to enable the aesthetic feature generation network to capture both global and local detail information, the entire collage image and multiple randomly cut image patches in the collage image can be input into the aesthetic feature generation network.

[0121] It should be understood that an image block is different from a sub-image, and an image block may include part or all of at least one sub-image.

[0122] As an example, the aesthetic feature generation network may be a deep neural network.

[0123] The present disclosure considers that since a collage image is composed of multiple sub-images, directly extracting aesthetic features from the entire collage image may cause a large amount of information loss and lead to a large amount of local details being lost. Therefore, the present disclosure uses information from multiple tiles to represent the aesthetic information of the collage image.

[0124] As an example, the aesthetic feature generation network can be pre-trained on the CPC (Compatible Photo Composition) aesthetic dataset, and then used to extract aesthetic features from the collage encoding generated in a single step. In general, the aesthetic feature generation network uses multiple patches to represent the collage and extracts general aesthetic features from the disorderly sampled image blocks, aiming to explore the comprehensive aesthetic properties between images.

[0125] In step S1022, the aesthetic features of the obtained multiple image blocks are fused to obtain the aesthetic features of the current collage image to be adjusted.

[0126] Here, the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0127] As an example, the image element aesthetic feature may represent at least one of the following aesthetic contents of the image: harmonious colors, good light, balanced elements, good content. It should be understood that other image element aesthetic aspects may also be represented, and the present disclosure is not limited thereto.

[0128] As an example, the composition aesthetic feature may represent at least one of the following aesthetic contents of the image: composition matching between adjacent sub-images (for example, it may include color matching between adjacent sub-images, etc.), highlighting important sub-images. It should be understood that other aspects of composition aesthetics may also be represented, and the present disclosure is not limited thereto.

[0129] As an example, the aesthetic features of the multiple image blocks may be fused based on the distance between the image block and the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

[0130] As an example, the aesthetic features of the obtained multiple image blocks may be fused through an attention mechanism to obtain the aesthetic features of the current collage image to be adjusted.

[0131] As an example, the aesthetic features of the obtained multiple image blocks may be fused through an attention mechanism for composition aesthetics to obtain the aesthetic features of the current collage image to be adjusted.

[0132] As an example, the aesthetic features of the multiple image blocks may be weighted based on the Euclidean distance of the image block from the center of the current collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the current collage image to be adjusted to obtain the aesthetic features of the current collage image to be adjusted.

[0133] As an example, the aesthetic feature F(C) of the current collage image to be adjusted can be expressed as: F(C)=α(C)·f(C),

[0134] Among them, α(C)=[α1, α2,...,α n ],

[0135] Among them, when hour, is 1, otherwise is 0,

[0136] Where C represents the current collage image to be adjusted, α(C) represents the weight matrix, f(C) represents the image block aesthetic feature matrix combined with composition aesthetics, and l i represents the Euclidean distance between the i-th image block and the center of the current collage image to be adjusted, P i represents the i-th image block among the multiple image blocks, s(Pi ) represents the area of ​​the i-th image block, s(C) represents the area of ​​the current collage image to be adjusted, η represents the area ratio coefficient, (y i , x i ) represents the center coordinate of the i-th image block, (h c , w c ) represents the size of the current collage image to be adjusted, h c Indicates the height of the current collage image to be adjusted, w c Indicates the width of the current collage image to be adjusted, (y c , x c ) represents the center coordinates of the current collage image to be adjusted, Represents the aesthetic features of the i-th image block. That is, the current collage image to be adjusted can be finally encoded as F(C) and input into the image collage model.

[0137] In order to effectively fuse the disordered blocks in the collage image, the present disclosure proposes an attention-based feature fusion method so as to simultaneously learn the general image aesthetic features and the typical composition aesthetic features in the collage. Since the most concerned part of the collage image is the composition and color matching of the sub-images, the present disclosure shifts more attention to the composition between adjacent sub-images rather than the local part of a single sub-image. As described above, the selection criteria for image block features with an area ratio greater than η are set to find better composition quality and more harmonious content positions.

[0138] The attention mechanism assigns dynamic weights to the selected image block features in order to effectively learn the complex structural features of the collage image. The present disclosure uses the central law in photography to pay special attention to the image blocks close to the center of the collage image in order to highlight the important images and further improve the aesthetic quality. The present disclosure uses the Euclidean distance and area factor to comprehensively weight the features. The attention mechanism is introduced in the process of multi-image block feature fusion to shift more focus to the center. The expression of the attention mechanism is as follows:

[0139]

[0140] The present invention considers the following technical problems and proposes the technical solution of the present invention:

[0141] (1) Since high-quality image collage requires certain professional experience, it is difficult to collect a large amount of labeled data;

[0142] (2) The effect of image collage is difficult to measure. It is difficult to automatically generate collage images that meet the public's visual aesthetics. The results obtained by using traditional layout optimization methods do not take into account whether the matching between images is coordinated and whether the overall composition of the collage is beautiful. It only meets the simple optimization of indicators and does not meet the design standards that meet the public's aesthetics.

[0143] (3) Due to the reliance on manually defined features, it is unable to provide sufficient feature representation for complex collage images, which greatly limits the quality of image collage;

[0144] (4) Collage involves many functional modules, and how to adjust and integrate them to optimize the effect is also a challenge.

[0145] According to the technical solution disclosed in the present invention, the automatic image collage is divided into three steps for generation. Through reinforcement learning modeling, the collage is decomposed into a series of explainable operation steps. The generation of the collage is simulated as a sequential decision-making process of adjusting the global layout and placement, rotation angle, and layer order. A deep collage aesthetic network is designed to extract and evaluate the features of the collage. Finally, a collage image with a specific aspect ratio can be generated, thereby improving the quality of the image collage results.

[0146] According to the technical solution disclosed in the present invention, collages with central attention-based or extensible aesthetic rules can be generated; a deep aesthetic network and attention mechanism fusion are designed to extract general aesthetic features and composition aesthetic features of the collage, which can capture richer high-level aesthetic features than manual definition; the use of reinforcement learning methods to generate collage images is more interpretable, and the collage is modeled as a reinforcement learning process of serialized decision-making, and the collage generation is split into interpretable operation steps through layout generation and detail adjustment.

[0147] Figure 5 A flowchart of an image collage method according to an exemplary embodiment of the present disclosure is shown.

[0148] Reference Figure 5 In step S201, a plurality of sub-images to be collaged are obtained, and the plurality of sub-images to be collaged are collaged into one image to obtain an initial collage image to be adjusted.

[0149] As an example, the multiple sub-images to be collaged may be multiple separate images, or may be multiple key frame sequences in the same video.

[0150] In step S202, the aesthetic features of the collage image to be adjusted are obtained.

[0151] In step S203, the acquired aesthetic features are input into the image collage model to obtain a predicted collage adjustment action sequence to be executed, the predicted collage adjustment action sequence is executed on the current collage image to be adjusted, and the process returns to step S202. Specifically, when it is determined to continue adjusting the collage image, the process returns to step S202.

[0152] The image collage model is trained in the following manner: adjusting parameters of the image collage model based on a reward function for executing collage adjustment action sequences predicted by the image collage model each time for training samples.

[0153] In step S204 , when it is determined not to continue adjusting the collage image, the adjusted collage image is output.

[0154] In other words, before determining not to continue adjusting the collage image, step S202 and step S203 are iteratively executed, that is, after each execution of step S203, the execution returns to step S202 based on the collage image adjusted this time.

[0155] The following will be combined Figure 6 An exemplary embodiment of step S203 is described below.

[0156] Reference Figure 6 In step S2031, the acquired aesthetic features are input into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction.

[0157] In step S2032, the predicted collage adjustment action sequence is executed on the current collage image to be adjusted to obtain the collage image after the current collage adjustment.

[0158] In step S2033, the collage image after the current collage adjustment is cropped according to the aspect ratio specified by the user.

[0159] In step S2034, it is determined whether to continue adjusting the collage image. If it is determined to continue adjusting the collage image, the collage image obtained by this cropping is used as the current collage image to be adjusted, and the process returns to step S202.

[0160] When it is determined in step S2034 that the collage image is not to be adjusted further, step S204 is executed, wherein step S204 may include: step S2041 , outputting the collage image obtained by this cropping.

[0161] As an example, the current collage image to be adjusted can be input into an aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and the aesthetic features of the obtained multiple image blocks are fused to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0162] As an example, the aesthetic features of the multiple image blocks may be fused based on the distance between the image block and the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

[0163] As an example, the aesthetic features of the obtained multiple image blocks may be fused through an attention mechanism for composition aesthetics to obtain the aesthetic features of the current collage image to be adjusted.

[0164] As an example, the aesthetic features of the multiple image blocks may be weighted based on the Euclidean distance of the image block from the center of the current collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the current collage image to be adjusted to obtain the aesthetic features of the current collage image to be adjusted.

[0165] As an example, the aesthetic feature F(C) of the current collage image to be adjusted can be expressed as: F(C)=α(C)·f(C),

[0166] Among them, α(C)=[α1, α2,..., α n ],

[0167] Among them, when hour, is 1, otherwise is 0,

[0168] Where C represents the current collage image to be adjusted, α(C) represents the weight matrix, f(C) represents the image block aesthetic feature matrix combined with composition aesthetics, and l i represents the Euclidean distance between the i-th image block and the center of the current collage image to be adjusted, P i represents the i-th image block among the multiple image blocks, s(P i ) represents the area of ​​the i-th image block, s(C) represents the area of ​​the current collage image to be adjusted, η represents the area ratio coefficient, (y i , x i ) represents the center coordinate of the i-th image block, (h c , w c ) represents the size of the current collage image to be adjusted, hc Indicates the height of the current collage image to be adjusted, w c Indicates the width of the current collage image to be adjusted, (y c , x c ) represents the center coordinates of the current collage image to be adjusted, represents the aesthetic features of the i-th image block.

[0169] As an example, the types of tile adjustment actions may include: layout adjustment actions and / or local adjustment actions.

[0170] As an example, the types of layout adjustment actions may include: exchanging positions of two sub-images and terminating the layout adjustment action.

[0171] As an example, the type of local adjustment action may include at least one of the following items: adjusting the sub-image placement position, not adjusting the sub-image placement position, rotating the sub-image placement angle, not adjusting the sub-image placement angle, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0172] As an example, multiple candidate views that meet the aspect ratio specified by the user can be cropped from the collage image after this collage adjustment; the multiple candidate views are input into the aesthetic scoring network to obtain estimated aesthetic scores of the multiple candidate views; and the candidate view with the highest aesthetic score among the multiple candidate views is used as the collage image obtained this time.

[0173] As an example, the image collage model may be trained using the training method described in the above exemplary embodiment.

[0174] The specific processing in the image collage method according to the exemplary embodiment of the present disclosure has been described in detail in the embodiment of the training method of the above-mentioned related image collage model, and will not be elaborated here.

[0175] Figure 7 A structural block diagram of a training device for an image collage model according to an exemplary embodiment of the present disclosure is shown.

[0176] Reference Figure 7 According to an exemplary embodiment of the present disclosure, the image collage model training device 10 includes: an initial collage unit 101, an aesthetic feature acquisition unit 102, a prediction adjustment unit 103, and a training unit 104.

[0177] Specifically, the initial collage unit 101 is configured to obtain a training sample including a plurality of sub-images, and to collage the plurality of sub-images into one image to obtain an initial collage image to be adjusted.

[0178] The aesthetic feature acquisition unit 102 is configured to acquire the aesthetic feature of the collage image to be adjusted.

[0179] The prediction adjustment unit 103 is configured to input the acquired aesthetic features into the image collage model, obtain a predicted collage adjustment action sequence that needs to be executed, execute the predicted collage adjustment action sequence on the current collage image to be adjusted, and return the aesthetic features of the current collage image to be adjusted acquired by the aesthetic feature acquisition unit 102.

[0180] The training unit 104 is configured to adjust the parameters of the image collage model according to the reward function of executing each predicted collage adjustment action sequence to train the image collage model when it is determined not to continue adjusting the collage image.

[0181] As an example, the prediction adjustment unit 103 may include: a prediction unit (not shown), a tiling adjustment unit (not shown), a cropping unit (not shown), and an adjustment end determination unit (not shown).

[0182] Specifically, the prediction unit is configured to input the acquired aesthetic features into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction.

[0183] The collage adjustment unit is configured to execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the current collage adjustment.

[0184] The cropping unit is configured to crop the collage image after the collage adjustment according to a specified aspect ratio.

[0185] The adjustment end determination unit is configured to determine whether to continue adjusting the collage image, wherein, when it is determined to continue adjusting the collage image, the aesthetic feature acquisition unit 102 uses the collage image obtained by this cropping as the current collage image to be adjusted, and acquires the aesthetic features of the current collage image to be adjusted.

[0186] As an example, the aesthetic feature acquisition unit 102 may be configured to input the current collage image to be adjusted into the aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and perform fusion processing on the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0187] As an example, the aesthetic feature acquisition unit 102 may be configured to fuse the aesthetic features of the multiple image blocks based on the distance between the image block and the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

[0188] As an example, the aesthetic feature acquisition unit 102 may be configured to perform a fusion process on the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics, so as to obtain the aesthetic features of the current collage image to be adjusted.

[0189] As an example, the types of collage adjustment actions may include: layout adjustment actions and / or local adjustment actions; wherein, the types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; wherein, the types of local adjustment actions may include at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0190] As an example, the reward function for executing each predicted collage adjustment action sequence can be calculated in the following way: based on the difference between the aesthetic score of the collage image obtained after executing the currently predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the previously predicted collage adjustment action sequence, the reward function for executing the currently predicted collage adjustment action sequence is calculated.

[0191] As an example, the reward function for executing the tth predicted collage adjustment action sequence can be calculated based on the following items: t, the difference between the aesthetic score of the collage image obtained after executing the tth predicted collage adjustment action sequence, and the aesthetic score of the collage image obtained after executing the t-1th predicted collage adjustment action sequence.

[0192] As an example, the aesthetic score of the collage image may be calculated based on the number of aesthetic boxes of the collage image and / or the white space loss area of ​​the collage image.

[0193] As an example, the cropping unit may be configured to crop a plurality of candidate views conforming to the specified aspect ratio from the collage image after the current collage adjustment; input the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; and use the candidate view with the highest aesthetic score among the plurality of candidate views as the collage image obtained by this cropping.

[0194] As an example, the aesthetic feature acquisition unit 102 may be configured to perform weighted processing on the aesthetic features of the multiple image blocks based on the Euclidean distance between the image block and the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted, so as to obtain the aesthetic features of the collage image to be adjusted.

[0195] Figure 8 A structural block diagram of an image collage device according to an exemplary embodiment of the present disclosure is shown.

[0196] Reference Figure 8 According to an exemplary embodiment of the present disclosure, the image collage device 20 includes: an initial collage unit 201 , an aesthetic feature acquisition unit 202 , a prediction adjustment unit 203 , and an output unit 204 .

[0197] Specifically, the initial collage unit 201 is configured to obtain a plurality of sub-images to be collaged, and collage the plurality of sub-images to be collaged into one image to obtain an initial collage image to be adjusted.

[0198] The aesthetic feature acquisition unit 202 is configured to acquire the aesthetic feature of the collage image to be adjusted.

[0199] The prediction adjustment unit 203 is configured to input the acquired aesthetic features into the image collage model, obtain a predicted collage adjustment action sequence that needs to be executed, execute the predicted collage adjustment action sequence on the current collage image to be adjusted, and return the aesthetic features of the current collage image to be adjusted acquired by the aesthetic feature acquisition unit 202.

[0200] The image collage model is trained in the following manner: adjusting parameters of the image collage model based on a reward function for executing collage adjustment action sequences predicted by the image collage model each time for training samples.

[0201] The output unit 204 is configured to output the adjusted collage image when it is determined not to continue adjusting the collage image.

[0202] As an example, the prediction adjustment unit 203 may include: a prediction unit (not shown), a tiling adjustment unit (not shown), a cropping unit (not shown), and an adjustment end determination unit (not shown).

[0203] Specifically, the prediction unit is configured to input the acquired aesthetic features into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction.

[0204] The collage adjustment unit is configured to execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the current collage adjustment.

[0205] The cropping unit is configured to crop the collage image after the collage adjustment according to the aspect ratio specified by the user.

[0206] The adjustment end determination unit is configured to determine whether to continue adjusting the collage image, wherein, when it is determined to continue adjusting the collage image, the aesthetic feature acquisition unit 202 uses the collage image obtained by this cropping as the current collage image to be adjusted, and acquires the aesthetic features of the current collage image to be adjusted.

[0207] The output unit 204 may be configured to output the collage image obtained by this cropping when it is determined that the collage image is not to be adjusted further.

[0208] As an example, the aesthetic feature acquisition unit 202 may be configured to input the current collage image to be adjusted into the aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and perform fusion processing on the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

[0209] As an example, the aesthetic feature acquisition unit 202 may be configured to fuse the aesthetic features of the multiple image blocks based on the distance of the image block from the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

[0210] As an example, the aesthetic feature acquisition unit 202 may be configured to perform a fusion process on the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics, so as to obtain the aesthetic features of the current collage image to be adjusted.

[0211] As an example, the types of collage adjustment actions may include: layout adjustment actions and / or local adjustment actions; wherein, the types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; wherein, the types of local adjustment actions may include at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

[0212] As an example, the cropping unit can be configured to: crop out multiple candidate views that meet the aspect ratio specified by the user from the collage image after this collage adjustment; input the multiple candidate views into the aesthetic scoring network to obtain estimated aesthetic scores of the multiple candidate views; and use the candidate view with the highest aesthetic score among the multiple candidate views as the collage image obtained by this cropping.

[0213] As an example, the aesthetic feature acquisition unit 202 may be configured to perform weighted processing on the aesthetic features of the multiple image blocks based on the Euclidean distance between the image block and the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted, so as to obtain the aesthetic features of the collage image to be adjusted.

[0214] As an example, the image collage model may be trained using the training device 10 as described above.

[0215] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0216] In addition, it should be understood that the various units in the image collage model training device 10 and the image collage device 20 according to the exemplary embodiment of the present disclosure can be implemented as hardware components and / or software components. Those skilled in the art can implement various units, for example, using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) according to the processing performed by the defined various units.

[0217] Fig. 9 FIG. 1 is a block diagram showing a structure of an electronic device according to an exemplary embodiment of the present disclosure. Fig. 9 The electronic device 30 includes: at least one memory 301 and at least one processor 302, wherein the at least one memory 301 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 302, the image collage model training method and / or the image collage method as described in the above exemplary embodiments are executed.

[0218] As an example, the electronic device 30 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 30 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 30 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0219] In the electronic device 30, the processor 302 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 302 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0220] The processor 302 may execute instructions or codes stored in the memory 301, wherein the memory 301 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0221] The memory 301 may be integrated with the processor 302, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 301 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 301 and the processor 302 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 302 can read files stored in the memory.

[0222] In addition, the electronic device 30 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 30 may be connected to each other via a bus and / or a network.

[0223] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the training method of the image collage model and / or the image collage method as described in the above exemplary embodiments. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0224] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to complete the image collage model training method and / or the image collage method as described in the above exemplary embodiments.

[0225] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0226] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image collage model, characterized in that: The training method comprises: Acquire a training sample including a plurality of sub-images, and collage the plurality of sub-images into one image to obtain an initial collage image to be adjusted; Obtaining aesthetic features of the current collage image to be adjusted; Inputting the acquired aesthetic features into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction; Execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the collage adjustment; Crop the collage image adjusted by this collage according to the specified aspect ratio; Determine whether to continue adjusting the collage image, wherein, when it is determined to continue adjusting the collage image, the collage image obtained by this cropping is used as the current collage image to be adjusted, and return to execute the step of obtaining the aesthetic features of the current collage image to be adjusted; When it is determined not to continue adjusting the collage image, the parameters of the image collage model are adjusted according to the reward function of executing each predicted collage adjustment action sequence to train the image collage model.

2. The training method according to claim 1, characterized in that: The steps of obtaining the aesthetic features of the current collage image to be adjusted include: Inputting the current collage image to be adjusted into an aesthetic feature generation network to obtain aesthetic features of a plurality of image blocks randomly sampled from the current collage image to be adjusted; The obtained aesthetic features of the multiple image blocks are fused to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

3. The training method according to claim 2, characterized in that: The step of fusing the obtained aesthetic features of the plurality of image blocks to obtain the aesthetic features of the current collage image to be adjusted comprises: Based on the distance between the image block and the center of the current collage image to be adjusted and / or the area of ​​the image block, the aesthetic features of the multiple image blocks are fused to obtain the aesthetic features of the current collage image to be adjusted.

4. The training method according to claim 2, characterized in that: The step of fusing the obtained aesthetic features of the plurality of image blocks to obtain the aesthetic features of the current collage image to be adjusted comprises: The aesthetic features of the obtained multiple image blocks are fused through an attention mechanism for composition aesthetics to obtain the aesthetic features of the current collage image to be adjusted.

5. The training method according to claim 1, characterized in that: Types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; The types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; Among them, the type of local adjustment action includes at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

6. The training method according to claim 1, characterized in that: The reward function for executing each predicted tile-adjusted action sequence is calculated as follows: Based on the difference between the aesthetic score of the collage image obtained after executing the currently predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the previously predicted collage adjustment action sequence, the reward function for executing the currently predicted collage adjustment action sequence is calculated.

7. The training method according to claim 6, characterized in that: The reward function for executing the t-th predicted tile adjustment action sequence is calculated based on the following terms: The value of t; the difference between the aesthetic score of the collage image obtained after executing the t-th predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the t-1-th predicted collage adjustment action sequence.

8. The training method according to claim 6, characterized in that: The aesthetic score of the collage image is calculated based on the number of aesthetic boxes of the collage image and / or the blank loss area of ​​the collage image.

9. The training method according to claim 1, characterized in that: The steps of cropping the collage image after this collage adjustment according to the specified aspect ratio include: Crop a plurality of candidate views conforming to the specified aspect ratio from the collage image after the collage adjustment; Inputting the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; The candidate view with the highest aesthetic score among the multiple candidate views is used as the collage image obtained by this cropping.

10. The training method according to claim 3, characterized in that: The step of fusing the aesthetic features of the plurality of image blocks based on the distance between the image block and the center of the current collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the current collage image to be adjusted comprises: Based on the Euclidean distance between the image block and the center of the current collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the current collage image to be adjusted, the aesthetic features of the multiple image blocks are weighted to obtain the aesthetic features of the current collage image to be adjusted.

11. An image collage method, characterized in that: The image collage method comprises: Acquire a plurality of sub-images to be collaged, and collage the plurality of sub-images to be collaged into one image to obtain an initial collage image to be adjusted; Obtaining aesthetic features of the current collage image to be adjusted; The acquired aesthetic features are input into the image collage model to obtain the collage adjustment action sequence that needs to be executed for this prediction; Execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the collage adjustment; Crop the collage image adjusted by this collage according to the aspect ratio specified by the user; Determine whether to continue adjusting the collage image, wherein, when it is determined to continue adjusting the collage image, the collage image obtained by this cropping is used as the current collage image to be adjusted, and return to execute the step of obtaining the aesthetic features of the current collage image to be adjusted; When it is determined not to continue adjusting the collage image, the collage image obtained by this cropping is output. The image collage model is trained in the following manner: adjusting the parameters of the image collage model based on a reward function for executing each predicted collage adjustment action sequence of the image collage model on training samples.

12. The image collage method according to claim 11, characterized in that: The steps of obtaining the aesthetic features of the current collage image to be adjusted include: Inputting the current collage image to be adjusted into an aesthetic feature generation network to obtain aesthetic features of a plurality of image blocks randomly sampled from the current collage image to be adjusted; The obtained aesthetic features of the multiple image blocks are fused to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

13. The image collage method according to claim 12, characterized in that: The step of fusing the obtained aesthetic features of the plurality of image blocks to obtain the aesthetic features of the current collage image to be adjusted comprises: Based on the distance between the image block and the center of the current collage image to be adjusted and / or the area of ​​the image block, the aesthetic features of the multiple image blocks are fused to obtain the aesthetic features of the current collage image to be adjusted.

14. The image collage method according to claim 12, characterized in that: The step of fusing the obtained aesthetic features of the plurality of image blocks to obtain the aesthetic features of the current collage image to be adjusted comprises: The aesthetic features of the obtained multiple image blocks are fused through an attention mechanism for composition aesthetics to obtain the aesthetic features of the current collage image to be adjusted.

15. The image collage method according to claim 11, characterized in that: Types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; The types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; Among them, the type of local adjustment action includes at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

16. The image collage method according to claim 11, characterized in that: The steps of cropping the collage image after the collage adjustment according to the aspect ratio specified by the user include: From the collage image adjusted by this collage, multiple candidate views meeting the aspect ratio specified by the user are cropped; Inputting the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; The candidate view with the highest aesthetic score among the multiple candidate views is used as the collage image obtained by this cropping.

17. The image collage method according to claim 13, characterized in that: The step of fusing the aesthetic features of the plurality of image blocks based on the distance between the image block and the center of the current collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the current collage image to be adjusted comprises: Based on the Euclidean distance between the image block and the center of the current collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the current collage image to be adjusted, the aesthetic features of the multiple image blocks are weighted to obtain the aesthetic features of the current collage image to be adjusted.

18. The image collage method according to claim 11, characterized in that: The image collage model is trained using the training method described in any one of claims 1 to 10.

19. A training device for an image collage model, characterized in that: The training equipment comprises: An initial collage unit is configured to obtain a training sample including a plurality of sub-images, and to collage the plurality of sub-images into one image to obtain an initial collage image to be adjusted; an aesthetic feature acquisition unit, configured to acquire the aesthetic feature of the current collage image to be adjusted; A prediction unit, configured to input the acquired aesthetic features into the image collage model to obtain a collage adjustment action sequence to be executed for this prediction; A collage adjustment unit is configured to execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the current collage adjustment; A cropping unit is configured to crop the collage image after the collage adjustment according to a specified aspect ratio; an adjustment end determination unit configured to determine whether to continue adjusting the collage image, wherein when it is determined to continue adjusting the collage image, the aesthetic feature acquisition unit uses the collage image obtained by this cropping as the current collage image to be adjusted, and acquires the aesthetic features of the current collage image to be adjusted; The training unit is configured to adjust the parameters of the image collage model according to the reward function of executing each predicted collage adjustment action sequence when it is determined not to continue adjusting the collage image, so as to train the image collage model.

20. The training device according to claim 19, characterized in that The aesthetic feature acquisition unit is configured to input the current collage image to be adjusted into the aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and perform fusion processing on the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

21. The training device according to claim 20, characterized in that The aesthetic feature acquisition unit is configured to fuse the aesthetic features of the multiple image blocks based on the distance between the image block and the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

22. The training device according to claim 20, characterized in that The aesthetic feature acquisition unit is configured to perform a fusion process on the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics, so as to obtain the aesthetic features of the current collage image to be adjusted.

23. The training device according to claim 19, characterized in that Types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; The types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; Among them, the type of local adjustment action includes at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

24. The training device according to claim 19, characterized in that The reward function for executing each predicted tile-adjusted action sequence is calculated as follows: Based on the difference between the aesthetic score of the collage image obtained after executing the currently predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the previously predicted collage adjustment action sequence, the reward function for executing the currently predicted collage adjustment action sequence is calculated.

25. Training device according to claim 24, characterized in that The reward function for executing the t-th predicted tile adjustment action sequence is calculated based on the following terms: The value of t; the difference between the aesthetic score of the collage image obtained after executing the t-th predicted collage adjustment action sequence and the aesthetic score of the collage image obtained after executing the t-1-th predicted collage adjustment action sequence.

26. The training device according to claim 24, characterized in that The aesthetic score of the collage image is calculated based on the number of aesthetic boxes of the collage image and / or the blank loss area of ​​the collage image.

27. The training device according to claim 19, characterized in that The cropping unit is configured to crop a plurality of candidate views conforming to the specified aspect ratio from the collage image after the collage adjustment; Inputting the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; The candidate view with the highest aesthetic score among the multiple candidate views is used as the collage image obtained by this cropping.

28. The training device according to claim 21, characterized in that The aesthetic feature acquisition unit is configured to perform weighted processing on the aesthetic features of the multiple image blocks based on the Euclidean distance between the image block and the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted, so as to obtain the aesthetic features of the collage image to be adjusted.

29. An image collage device, characterized in that: The image collage device comprises: An initial collage unit is configured to obtain a plurality of sub-images to be collaged, and collage the plurality of sub-images to be collaged into one image to obtain an initial collage image to be adjusted; an aesthetic feature acquisition unit, configured to acquire the aesthetic feature of the current collage image to be adjusted; A prediction unit is configured to input the acquired aesthetic features into the image collage model to obtain a collage adjustment action sequence to be executed for this prediction; A collage adjustment unit is configured to execute the predicted collage adjustment action sequence on the current collage image to be adjusted to obtain the collage image after the current collage adjustment; A cropping unit, configured to crop the collage image after the collage adjustment according to the aspect ratio specified by the user; an adjustment end determination unit configured to determine whether to continue adjusting the collage image, wherein when it is determined to continue adjusting the collage image, the aesthetic feature acquisition unit uses the collage image obtained by this cropping as the current collage image to be adjusted, and acquires the aesthetic features of the current collage image to be adjusted; an output unit, configured to output the collage image obtained by the current cropping when it is determined that the collage image is not to be adjusted further, The image collage model is trained in the following manner: adjusting the parameters of the image collage model based on a reward function for executing each predicted collage adjustment action sequence of the image collage model on training samples.

30. The image collage device according to claim 29, characterized in that: The aesthetic feature acquisition unit is configured to input the current collage image to be adjusted into the aesthetic feature generation network to obtain aesthetic features of multiple image blocks randomly sampled from the current collage image to be adjusted; and perform fusion processing on the obtained aesthetic features of the multiple image blocks to obtain the aesthetic features of the current collage image to be adjusted, wherein the aesthetic features of the current collage image to be adjusted are used to characterize the image element aesthetic features and composition aesthetic features of the current collage image to be adjusted.

31. The image collage device according to claim 30, characterized in that: The aesthetic feature acquisition unit is configured to fuse the aesthetic features of the multiple image blocks based on the distance between the image block and the center of the collage image to be adjusted and / or the area of ​​the image block to obtain the aesthetic features of the collage image to be adjusted.

32. The image collage device according to claim 30, characterized in that: The aesthetic feature acquisition unit is configured to perform a fusion process on the obtained aesthetic features of the multiple image blocks through an attention mechanism for composition aesthetics, so as to obtain the aesthetic features of the current collage image to be adjusted.

33. The image collage device according to claim 29, characterized in that: Types of collage adjustment actions include: layout adjustment actions and / or local adjustment actions; The types of layout adjustment actions include: exchanging the positions of two sub-images and terminating the layout adjustment action; Among them, the type of local adjustment action includes at least one of the following items: adjusting the placement position of the sub-image, not adjusting the placement position of the sub-image, rotating the placement angle of the sub-image, not adjusting the placement angle of the sub-image, adjusting the layer order of the sub-image, and not adjusting the layer order of the sub-image.

34. The image collage device according to claim 29, characterized in that: The cropping unit is configured to: crop a plurality of candidate views that meet the aspect ratio specified by the user from the collage image after the current collage adjustment; Inputting the plurality of candidate views into an aesthetic scoring network to obtain estimated aesthetic scores of the plurality of candidate views; The candidate view with the highest aesthetic score among the multiple candidate views is used as the collage image obtained by this cropping.

35. The image collage device according to claim 31, characterized in that: The aesthetic feature acquisition unit is configured to perform weighted processing on the aesthetic features of the multiple image blocks based on the Euclidean distance between the image block and the center of the collage image to be adjusted and the ratio of the area of ​​the image block to the area of ​​the collage image to be adjusted, so as to obtain the aesthetic features of the collage image to be adjusted.

36. The image collage device according to claim 29, characterized in that: The image collage model is trained using the training device according to any one of claims 19 to 28.

37. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method of the image collage model as described in any one of claims 1 to 10 and / or the image collage method as described in any one of claims 11 to 18.

38. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image collage model training method as described in any one of claims 1 to 10 and / or the image collage method as described in any one of claims 11 to 18.

39. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the training method of the image collage model according to any one of claims 1 to 10 and / or the image collage method according to any one of claims 11 to 18 are implemented.

Images