Panoramic image generation method, device, program product and electronic device

Abstract

The present disclosure provides a method, device, program product and electronic device for generating a panoramic image, and relates to the field of computer technology. The method comprises: obtaining input information; the input information comprises text prompt information and at least one of a first image; processing the input information through an image generation model to generate a second image; decomposing the second image into a plurality of first image blocks, performing image optimization processing on the first image blocks, and obtaining a plurality of corresponding second image blocks; determining fusion information between different second image blocks, combining a plurality of second image blocks based on the fusion information, and obtaining a panoramic image according to the combination result; the fusion information comprises first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image. The present disclosure can generate a panoramic image without relying on artificial synthesis, and improve the quality of the panoramic image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a panoramic image generation method, a panoramic image generation device, a computer program product, and an electronic device. Background Art

[0002] Panoramic images are images presented from the perspective of human binoculars and are widely used in virtual reality, map street views, housing rental and sales platforms, etc. In panoramic images, users can move the perspective of the image by rotating the virtual reality device, dragging the touch screen, etc., simulating the visual experience in the real world.

[0003] In the related art, panoramic images are synthesized manually, for example, the photographer needs to take multiple spatially continuous images, and then specialized personnel will stitch the multiple images together and perform other post-processing to obtain a panoramic image. This processing method is inefficient, has high labor and time costs, and is difficult to guarantee the quality of the panoramic image. Summary of the invention

[0004] The present disclosure provides a panoramic image generation method, a panoramic image generation device, a computer program product and an electronic device to at least to some extent solve the problem of relying on artificial synthesis of panoramic images in the related art.

[0005] According to a first aspect of the present disclosure, a panoramic image generation method is provided, the method comprising: obtaining input information; the input information comprising text prompt information and at least one of a first image; processing the input information through an image generation model to generate a second image; decomposing the second image into a plurality of first image blocks, performing image optimization processing on the first image blocks to obtain a corresponding plurality of second image blocks; determining fusion information between different second image blocks, combining the plurality of second image blocks based on the fusion information, and obtaining a panoramic image according to the combination result; the fusion information comprises first fusion information between a second image block located at a first edge and a second image block located at a second edge, the first edge and the second edge being two edges opposite to each other in the same direction in the panoramic image.

[0006] According to a second aspect of the present disclosure, a panoramic image generation device is provided, the device comprising: an input information acquisition module, configured to acquire input information; the input information comprises text prompt information and at least one of a first image; a second image generation module, configured to process the input information through an image generation model to generate a second image; an image block processing module, configured to decompose the second image into a plurality of first image blocks, perform image optimization processing on the first image blocks, and obtain a corresponding plurality of second image blocks; an image block combination module, configured to determine fusion information between different second image blocks, combine the plurality of second image blocks based on the fusion information, and obtain a panoramic image according to the combination result; the fusion information comprises first fusion information between a second image block located at a first edge and a second image block located at a second edge, the first edge and the second edge being two edges opposite to each other in the same direction in the panoramic image.

[0007] According to a third aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method of the first aspect and possible implementations thereof are implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method of the above-mentioned first aspect and its possible implementation methods by executing the executable instructions.

[0009] The technical solution disclosed in this disclosure has the following beneficial effects:

[0010] After obtaining the input information, the input information is processed through the image generation model to generate a second image, the second image is decomposed into first image blocks, and the second image blocks are obtained through image optimization processing, and the second image blocks are combined based on the fusion information between the second image blocks to obtain a panoramic image. On the one hand, this solution realizes the automatic generation of panoramic images without the need for manual post-processing, improves the generation efficiency, and reduces labor and time costs. On the other hand, based on the generation of the second image by the image generation model, a block optimization method is adopted to improve the detail information in the image. In addition, by fusing the second image blocks, especially fusing the second image blocks at the opposite edges on both sides, the global consistency of the image and the continuity of the content on both sides are improved, which meets the characteristics of the panoramic image, improves the problem of picture distortion, and improves the quality of the panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart showing a method for generating a panoramic image in this exemplary embodiment is shown;

[0012] Figure 2A flow chart for determining image segmentation parameters in this exemplary embodiment is shown;

[0013] Figure 3 A flowchart of fusing a second image block in this exemplary embodiment is shown;

[0014] Figure 4 A schematic diagram showing a combined second image block in this exemplary embodiment;

[0015] Figure 5 A schematic diagram showing a method of inputting a first image to generate a panoramic image in this exemplary embodiment;

[0016] Figure 6 A schematic structural diagram of a panoramic image generating device in this exemplary embodiment is shown;

[0017] Figure 7 A schematic structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings.

[0019] The accompanying drawings are schematic diagrams of the present disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in hardware modules or integrated circuits, or in networks, processors or microcontrollers. The embodiments can be implemented in various forms and should not be construed as being limited to the examples set forth herein. The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the description below, many specific details are provided to provide a full description of the embodiments of the present disclosure. However, those skilled in the art should appreciate that one or more specific details may be omitted when implementing the technical solution of the present disclosure, or one or more specific details may be replaced by other methods, components, devices, steps, etc.

[0020] In the related technology, it relies on artificial synthesis of panoramic images. For example, the photographer needs to take multiple images that are continuous in space, and then specialized personnel will stitch the multiple images together and perform other post-processing to obtain a panoramic image. This processing method is inefficient, has high labor and time costs, and it is difficult to guarantee the quality of the panoramic image. In addition, there are also solutions that use AI (Artificial Intelligence) models to generate panoramic images, but most of the generated panoramic images have problems such as image distortion.

[0021] In view of the above problems, an exemplary embodiment of the present disclosure provides a method for generating a panoramic image, which can generate a high-quality panoramic image, which can also be called an equidistant cylindrical projection image. Application scenarios include but are not limited to: a user inputs text prompt information, and a panoramic image is generated according to the text prompt information; a user inputs a first image, which is usually a non-panoramic image, and a panoramic image is generated according to the first image; a user inputs text prompt information and a first image, and a panoramic image is generated according to the text prompt information and the first image.

[0022] The following is an explanation of the professional terms involved in this exemplary embodiment:

[0023] Image generation model: A machine learning model used to generate images. It can be any type of model such as a neural network, for example, a model based on an architecture such as a GAN (Generative Adversarial Network).

[0024] Diffusion Model: A model for image generation. The basic idea is to start with a simple distribution (such as a Gaussian distribution) and then gradually add noise until the data becomes completely noise. This is called a diffusion process. The model then learns how to reverse the diffusion process, that is, to recover the original data from the noise. In this way, images are generated.

[0025] Stable Diffusion Model: A diffusion model. The processing method is: first, a clear image gradually becomes blurred and noisy, and then learns how to reverse the noisy state back to a clear image, thereby generating an image. When the user provides a text description, the stable diffusion model uses the knowledge learned in the training phase to determine the noise map that such a description may correspond to, and then gradually removes the noise from the noise map through the reverse diffusion process to restore a detailed and clear image that matches the text description. The image generation model used in this exemplary embodiment can be a diffusion model, and can be a stable diffusion model.

[0026] Conditional Guidance (Classifier Guidance), Unconditional Guidance (Classifier-Free Guidance, CFG): Conditional guidance and unconditional guidance refer to the control methods used in the image generation process. Conditional guidance means taking into full account the conditions provided by the user (such as text prompt information) and generating images under these conditions. This can ensure that images that meet the conditions are generated, but some features may be over-emphasized, causing distortion and other problems. Unconditional guidance means that the conditions provided by the user are not taken into account and images are generated completely freely. The images are more creative, but may deviate from the conditions.

[0027] Guidance Scale: Conditional guidance and unconditional guidance can be considered as two extreme cases. In image generation control, a control method between the two can be used to balance creativity and control. The guidance scale parameter indicates the extent to which the conditions are considered. For example, the image generated by conditional guidance and the image generated by unconditional guidance can be fused according to the guidance scale parameter. The guidance scale parameter can be used as the ratio of the fusion of the two images (for example, the fusion ratio corresponding to the image generated by conditional guidance and the fusion ratio corresponding to the image generated by unconditional guidance can be "1-guidance scale parameter"), thereby obtaining an image between complete creativity and complete control.

[0028] Timestep: It is an important parameter in image generation models, especially diffusion models. The entire image generation process can be divided into multiple timesteps. In each timestep, the features or images are processed to a certain extent. The current timestep can represent the current progress, that is, the current image generation step, so that the image generation model can better capture the stage characteristics and generate appropriate information.

[0029] Process parameters: refers to various parameters involved in the image generation process, usually parameters that control the image generation process, such as hyperparameters of the image generation model, parameters involved in image segmentation, optimization, fusion, etc. Exemplarily, process parameters may include but are not limited to: guidance intensity parameters; re-guidance parameters, which may indicate the degree of guidance re-applied according to conditions in the case of comprehensive conditional guidance and unconditional guidance; noise control intensity, which is a parameter used to control the noise feature processing in the image generation model, and may indicate the degree of intervention in the noise features generated by the model; image segmentation parameters, such as block size, overlap coefficient, etc.; image optimization parameters, such as super-resolution parameters, feature enhancement parameters, etc.; image fusion parameters, such as fusion size, etc.

[0030] The panoramic image generation method is described below. Figure 1 The schematic flow of the panoramic image generation method is shown, which may include the following steps:

[0031] Step S110, obtaining input information; the input information includes at least one of text prompt information and a first image;

[0032] Step S120, processing the input information through the image generation model to generate a second image;

[0033] Step S130, decomposing the second image into a plurality of first image blocks, performing image optimization processing on the first image blocks, and obtaining a plurality of corresponding second image blocks;

[0034] Step S140, determining fusion information between different second image blocks, combining multiple second image blocks based on the fusion information, and obtaining a panoramic image according to the combination result; the fusion information includes first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

[0035] In one embodiment, the processing process of the image generation model can be an iterative process. Specifically, the input information is iteratively processed for the first time through the image generation model to generate a second image; the second image is iteratively processed for the second time through the image generation model, and the input information can be added in the second iterative processing, such as inputting the input information and the second image into the image generation model for the second iterative processing; and so on, any round of iterative processing can be performed to finally obtain a high-quality second image. Alternatively, the input information is iteratively processed for the first time through the image generation model to generate a second image, and a panoramic image is obtained through steps S130 and S140, and the panoramic image is iteratively processed for the second time through the image generation model, and the input information can be added in the second iterative processing, such as inputting the input information and the panoramic image into the image generation model for the second iterative processing to obtain the second image in the second round of iteration, and then the panoramic image in the second round of iteration is obtained through steps S130 and S140; and so on, any round of iterative processing can be performed to finally obtain a high-quality panoramic image. It should be understood that in any round of iterative processing, only partial processing of the image generation model may be performed, that is, the second image in the current iteration is generated by the image generation model, and then the next iterative processing is performed; or partial processing of the image generation model and steps S130 and S140 may be performed, that is, the second image in the current iteration is generated by the image generation model, and then the panoramic image in the current iteration is obtained through steps S130 and S140, and then the next iterative processing is performed. Different rounds of iterations may adopt different processing methods, such as only partial processing of the image generation model in the second round of iterative processing, and partial processing of the image generation model and steps S130 and S140 in the third round of iterative processing. The present disclosure does not limit this.

[0036] based on Figure 1The method shown, after obtaining input information, processes the input information through an image generation model to generate a second image, decomposes the second image into first image blocks, and obtains second image blocks through image optimization processing, and combines the second image blocks based on the fusion information between the second image blocks to obtain a panoramic image. On the one hand, this solution realizes the automatic generation of panoramic images without the need for manual post-processing, improves the generation efficiency, and reduces labor and time costs. On the other hand, based on the generation of the second image by the image generation model, a block optimization method is adopted to improve the detail information in the image. In addition, by fusing the second image blocks, especially fusing the second image blocks at the opposite edges, the global consistency of the image and the continuity of the content on both sides are improved, which meets the characteristics of the panoramic image, improves the problem of picture distortion, and improves the quality of the panoramic image.

[0037] Below Figure 1 Provide detailed instructions for each step.

[0038] refer to Figure 1 In step S110, input information is obtained; the input information includes at least one of text prompt information and a first image.

[0039] The text prompt information may be information input by the user for describing the image. If the user inputs prompt information in other modes such as voice, it can be converted into text prompt information for processing. The first image may be an initial image input by the user for generating a panoramic image, and may include one or more images. The input information obtained in step S110 may include only the text prompt information, or only the first image, or both the text prompt information and the first image.

[0040] Continue to refer Figure 1 In step S120, the input information is processed by the image generation model to generate a second image.

[0041] In this exemplary embodiment, any type of image generation model may be used, such as GAN, diffusion model, stable diffusion model. Input information is input into the image generation model, and after a series of processing by the image generation model, a second image is output. The second image may be a preliminary panoramic image or a non-panoramic image.

[0042] In one implementation, the processing of the input information by the image generation model to generate the second image may include the following steps:

[0043] The image generation model is used to extract features from the input information to obtain initial prompt features, and to generate initial noise features. The initial prompt features and initial noise features are processed to obtain a second image.

[0044] Among them, the image generation model is usually provided with an intermediate layer for extracting features, such as a convolutional layer, etc., which can extract features from the input information. In one embodiment, the image generation model is a diffusion model, and the diffusion model can map the input information to a first feature space to obtain initial prompt features. The first feature space is the space where some feature information in the image generation process is located, such as a latent space, that is, a diffusion model learning and generating feature space. Exemplarily, the diffusion model includes a text encoder and an image encoder, such as a text encoder and an image encoder based on a CLIP (Contrastive Language-Image Pre-training) architecture, using a text encoder to encode text prompt information, and using an image encoder to encode the first image, and after encoding, initial text features and initial image features are obtained respectively, and these two features together constitute the initial prompt features.

[0045] The initial noise feature may be a noise feature that is independent of the input information, such as a randomly initialized noise feature, or a noise feature that is related to the input information, such as a noise feature initialized under the guidance of the input information or the initial prompt feature. Exemplarily, the initial noise feature based on the first feature space may be generated by a diffusion model, or the initial noise information may be generated by a diffusion model, and the initial noise information may be mapped to the first feature space to obtain the initial noise feature.

[0046] In one embodiment, the input information includes a first image, and the initial prompt feature includes an initial image feature. The above-mentioned extraction of features from the input information by the image generation model to obtain the initial prompt feature may include the following steps:

[0047] The image generation model is used to extract features of multiple scales from the first image, and the features of multiple scales are fused to obtain initial image features.

[0048] Among them, the features of different scales can be extracted based on the features of regions of different scales in the first image, representing image information at different scales. Exemplarily, the features of different scales can be extracted by convolutional layers of different scales, or feature layers with different dimensions can be set to extract features of different scales respectively. The features of different scales can be fused by splicing, weighting, etc. to obtain the initial image features.

[0049] The formula for multi-scale feature fusion is shown below:

[0050]

[0051] Among them, w i represents the weight corresponding to the i-th scale, F irepresents the features of the i-th scale extracted from the first image, F fused Represents the initial image features obtained after fusion.

[0052] The above process of extracting and fusing multiple scale features can be referred to the following code:

[0053]

[0054] By extracting and fusing multi-scale features, more comprehensive and sufficient initial image features can be obtained, which is conducive to the subsequent generation of high-quality panoramic images.

[0055] When the initial cue features and the initial noise features are extracted, the initial cue features and the initial noise features are further processed by the image generation model to obtain a second image. For example, the image generation model includes multiple feature layers, and a series of feature processing can be performed on the initial cue features and the initial noise features, such as feature diffusion, feature dimension conversion, feature space mapping, etc., and the processed features can be mapped to a second feature space (such as an image pixel space) to obtain a second image.

[0056] In one implementation, the processing of the initial prompt feature and the initial noise feature may include the following steps:

[0057] Determine the process parameters corresponding to each time step;

[0058] According to the process parameters corresponding to each time step, the initial prompt feature and the initial noise feature are processed in turn according to each time step.

[0059] As mentioned above, a time step represents the progress in the image generation process. The corresponding process parameters can be determined according to each time step, so that at each time step, the process parameters used are adapted to the current process progress. For example, at different stages of the image generation process, the quality of feature processing can be optimized by using appropriate process parameters.

[0060] In one implementation, the process parameters corresponding to each time step may be determined according to the parameter optimization strategy adopted. The parameter optimization strategy includes but is not limited to:

[0061] Linear adaptation (ADAPTIVE_LINEAR): refers to the linear relationship between parameters and time steps, and the characteristic is that the parameters change evenly with the time step (such as uniformly decreasing). It is suitable for scenarios that require stable and predictable parameter changes. The advantage is that the parameter change process is smooth and easy to control.

[0062] Harmonic decay (HARMONIC_DECAY): refers to the harmonic decay relationship between parameters and time steps. The characteristic is that the parameters change slowly at the beginning and change faster in the later stage. For example, in the early stage of the image generation process, the process parameters decrease slowly and decrease rapidly in the later stage. It is suitable for scenes that need to strengthen the adjustment effect in the later stage of the process. The advantage is that the change of parameters conforms to the law of natural decay.

[0063] Exponential growth (EXPONENTIAL_GROWTH): refers to the exponential growth relationship between parameters and time steps, and the characteristic is that the speed of parameter change accelerates as the time step progresses. It is suitable for scenarios that require progressive enhancement effects. The advantage is that it can achieve progressive enhancement effects of process parameters as the process progresses, so that features are also progressively enhanced.

[0064] Cyclic adjustment (CYCLIC_ADJUSTMENT): refers to the periodic change relationship between parameters and time steps. It is suitable for scenarios where parameters need to be adjusted repeatedly. The advantage is that it can be optimized repeatedly at different stages.

[0065] In one embodiment, the parameter optimization strategy to be adopted can be determined according to the complexity of the image content to be generated. For example, if a simple image needs to be generated, a linear adaptation strategy is suitable; if a complex image needs to be generated, a periodic adjustment strategy is suitable; if a detailed image needs to be generated, a harmonic attenuation strategy is suitable; if an image with progressive feature enhancement needs to be generated, an exponential growth strategy is suitable.

[0066] In one embodiment, the parameter optimization strategy to be adopted can be determined according to the feature distribution characteristics (such as the initial prompt feature, the initial noise feature, or the intermediate feature after the initial prompt feature and the initial noise feature are processed). For example, when the feature distribution is relatively uniform, a linear adaptation strategy is used; when the feature distribution is uneven, a nonlinear strategy is used, such as a harmonic attenuation strategy, an exponential growth strategy, etc.; when the feature distribution is multi-peaked, a periodic adjustment strategy is used; when the feature distribution is unimodal, a monotonic change strategy is used, such as a linear adaptation strategy, a harmonic attenuation strategy, an exponential growth strategy, etc.

[0067] In one embodiment, appropriate strategies can be used at different stages of the image generation process. For example, if problems are prone to occur in the early stages of the image generation process, a strategy with large changes in the early stages can be selected, such as a linear adaptation strategy, an exponential growth strategy, a periodic adjustment strategy, etc.; if problems are prone to occur in the later stages of the image generation process, a strategy with large changes in the later stages can be selected, such as a linear adaptation strategy, a harmonic attenuation strategy, a periodic adjustment strategy, etc.; if the entire image generation process needs to be continuously adjusted, a periodic adjustment strategy can be selected; if the entire image generation process needs to be stable and controllable, a linear adaptation strategy can be used. In addition, the characteristics or image conditions at different stages can be monitored in the image generation process, and the parameter optimization strategy can be adjusted when necessary to implement a feedback mechanism for strategy selection. For example, if there is overall oversaturation, a linear adaptation strategy is adopted; if there is local oversaturation, a periodic adjustment strategy is adopted; if there is progressive oversaturation, a harmonic attenuation strategy is adopted; if progressive control is required, an exponential growth strategy is adopted; if there is overall blurring of texture details, an exponential growth strategy is adopted; if there is local distortion in texture details, a periodic adjustment strategy is adopted; if there is progressive degradation in texture details, a harmonic attenuation strategy is adopted; if there is a uniformity problem in texture details, a linear adaptation strategy is adopted.

[0068] The following is an exemplary description of the process of determining process parameters based on time steps and performing feature processing.

[0069] The following noise prediction function can be used:

[0070] P(z t |x 0 )=N(μ θ (z t ,t),Σ θ (z t ,t)) (2)

[0071] Where t represents the time step, z t Indicates the current image state, x 0 represents the desired image, μ θ Represents the main method of noise prediction, Σ θ Represents the uncertainty range of the noise prediction.

[0072] Determine the corresponding noise control strength control_strength according to the time step t.

[0073]

[0074] Where min represents the minimum value of the noise control strength. The image generation process includes a warm-up phase, in which a stronger noise control strength or other control strength parameters are usually used, and various control strength parameters are gradually reduced after the warm-up phase.warmup Indicates the duration of the warm-up phase, that is, the number of time steps included in the warm-up phase.

[0075] In addition, the guidance intensity parameter can be calculated using formula (3), where min in formula (3) is replaced with the minimum value of the guidance intensity parameter, and the calculated result represents the guidance intensity parameter corresponding to the time step t.

[0076] The processing process of noise features can refer to the following code:

[0077]

[0078] In the above manner, by using process parameters adapted to the time step and performing feature processing according to each time step, a high feature processing quality can be ensured, which is conducive to generating high-quality panoramic images.

[0079] In one implementation, the control intensity parameter may be determined by referring to the following formula:

[0080] control_strength=α·ContentComplexity+β·TargetQuality (4)

[0081] The control strength parameter control_strength indicates the degree of control over the image generation process. The larger the control strength parameter, the closer the generated features and images will be to the control conditions (such as text prompt information, the first image, or other user-specified conditions); the smaller the control strength parameter, the more freedom the generated features or images will have. Exemplarily, the control strength parameter can be between 0 and 1.

[0082] ContentComplexity is the content complexity, which indicates the content complexity of the current feature or image. The content complexity can be determined by considering one or more of the following factors: the richness of the details of the feature or image, the degree of texture variation, the diversity of colors, and the density of edges. The system can analyze the above factors and use corresponding quantitative indicators to represent the content complexity, such as by calculating the gradient change and frequency distribution of the image to quantify the content complexity. Generally speaking, the higher the content complexity, the richer the information contained in the current feature or image, and stronger control is required to ensure the generation quality.

[0083] TargetQuality is the target quality, which indicates the desired image quality level. It can be set by the user or automatically determined by the system according to the application scenario. Generally, the higher the target quality, the higher the control intensity parameter needs to be to ensure that the generated result meets the requirements.

[0084] α and β are the weights of content complexity and target quality, respectively. These two weights are used to balance the relative importance of content complexity and target quality. Generally, α+β=1 to achieve the normalization of the overall weight. If the current features or image details are rich, a relatively large α value can be set so that the image generation process pays more attention to content preservation; if the user requires high-quality output, a relatively large β value can be set so that the image generation process pays more attention to quality improvement. In addition, at different stages of the image generation process, the α and β values ​​can change to meet the needs of different stages.

[0085] As can be seen from the above, by calculating the control strength parameter using formula (4), the control strength parameter can be automatically adjusted according to the input information, the current features or characteristics of the image, and the target requirements, and different control strategies can be implemented by adjusting the weights, making the image generation process more flexible. In addition, this parameter control mechanism can better balance the degree of freedom and control accuracy of generation, which is very important for tasks such as panoramic image generation that require both precise control and creativity, and is conducive to generating suitable panoramic images.

[0086] In one embodiment, if the input information includes a first image, the above-mentioned processing of the input information by the image generation model to generate the second image may include: extracting initial image features from the first image by the image generation model, and processing the first image features to obtain first image features, and generating second image features according to the input information (first image, or first image and text prompt information), fusing the first image features and the second image features, and generating the second image according to the fused features. Exemplarily, the area to be expanded can be determined according to the first image, such as determining mask information according to the first image, and the mask information indicates which areas in the first image need to be expanded. The image generation model can generate image content for the area to be expanded, such as generating second image features. The first image features and the second image features can be fused in the first feature space to achieve feature-level fusion, so as to better achieve semantic consistency of the second image and better retain the information in the first image.

[0087] Continue to refer Figure 1 In step S130, the second image is decomposed into a plurality of first image blocks, and image optimization processing is performed on the first image blocks to obtain a plurality of corresponding second image blocks.

[0088] Exemplarily, the size of the first image block can be determined according to the image optimization capability, and the second image can be decomposed into multiple first image blocks according to the size. Each first image block is optimized to obtain a corresponding second image block. Image optimization processing includes but is not limited to: super-resolution processing, feature enhancement processing, noise reduction processing, color enhancement processing, etc.

[0089] In one embodiment, reference Figure 2 As shown, the above decomposing the second image into a plurality of first image blocks may include the following steps S210 to S230:

[0090] Step S210: determining a block size according to the image complexity of the second image and the resource amount of image processing resources.

[0091] Among them, image complexity represents the complexity of information in the image, which can be determined based on one or more of the following factors: image gradient (including gradient changes of the image in the horizontal and vertical directions), image resolution, degree of change of image content (such as the number and complexity of objects and scenes in the image), texture richness (such as can be calculated based on the frequency domain characteristics of the image), color change degree (such as can be determined by statistical distribution of colors in the image). Exemplarily, image complexity can be a parameter between 0-1. Image processing resources include various hardware resources for processing images, such as video memory resources of GPU (Graphics Processing Unit, graphics processor). Block size refers to the calculated size of the first image block. Exemplarily, the block size can be calculated using the following formula:

[0092]

[0093] Among them, AvailableMemory refers to the available memory size, which can refer to the GPU video memory or memory size that can be used to generate images in the system. Specifically, the system can monitor the available video memory status of the current device in real time. For GPU devices, a portion of the video memory is usually reserved for system use. Therefore, AvailableMemory represents the actual available video memory. The actual available video memory = total video memory - system reserved video memory - video memory occupied by the basic model. The unit can be GB or MB, etc. ImageComplexity represents the image complexity of the second image. BlockSizeoptimal represents the block size, and MaxBlockSize represents the set maximum block size, which can be determined based on the amount of resources, the size of the second image, the model used for image optimization processing, etc., or can be set manually.

[0094] In one implementation, the maximum block size is first determined based on available memory, and then the block size is adjusted based on the image complexity. Generally, the higher the image complexity, the smaller the block size is used to ensure the processing quality of the single first image block. The larger the amount of resources, the larger the block size is used to improve processing efficiency.

[0095] Step S220, determining the overlap size according to the block size and the overlap coefficient.

[0096] The overlap coefficient may refer to the overlap ratio between adjacent first image blocks, and the overlap size may refer to the size of the overlap area between adjacent first image blocks. The overlap coefficient may be determined empirically, or may be determined based on the size of the second image, image complexity, resource quantity, etc. Generally, the higher the image complexity and the larger the resource quantity, the larger the overlap coefficient may be, so as to ensure sufficient processing of the image. Exemplarily, the overlap size may be determined by the following formula:

[0097] OverlapSize = BlockSize K2ti1aL OverlapFactor (6)

[0098] Among them, OverlapSize represents the overlap size, and OverlapFactor represents the overlap coefficient.

[0099] For example, the size of the second image is 6144×3072, the image complexity is 0.8 (high), and the available video memory is 8 GB. It can be determined that the block size is 256×256 and the overlap factor is 50%.

[0100] Step S230 : decompose the second image into a plurality of first image blocks according to the block size and the overlap size, so that the size of the overlap area between adjacent first image blocks is the overlap size.

[0101] According to the block size and the overlap size, the coordinate range of each first image block in the horizontal and vertical directions can be calculated, thereby decomposing each first image block from the second image. Exemplarily, the process of decomposing the image block can refer to the following code:

[0102]

[0103]

[0104] based on Figure 2 The method adopts the block size and other block parameters that are adaptive to the second image and the amount of resources. Its advantages are: it can automatically adjust the processing method according to the hardware conditions to ensure that images of any size can be processed without insufficient resources, ensure the balance between processing quality and efficiency, and improve the adaptability and stability of the system. In particular, it enables the system to efficiently process the generation task of panoramic images under limited hardware resources while ensuring processing quality.

[0105] In one implementation, performing image optimization processing on the first image block to obtain a corresponding plurality of second image blocks may include the following steps:

[0106] Super-resolution processing is performed on the first image block, and feature enhancement processing is performed on the first image block after the super-resolution processing to obtain corresponding multiple second image blocks.

[0107] Among them, feature enhancement processing is to enhance the image at the feature level (mathematical representation of visual elements such as texture, edge, color, etc.). Here, "feature" refers to the mathematical representation of visual elements such as texture, edge, color, etc. in the image. One or more of super-resolution processing and feature enhancement processing can be completed using models such as Real-ESRGAN (a super-resolution model), which not only increases the image resolution, but also improves the visual quality of the image through deep learning models. This process takes into account the local and global features of the image to ensure that the enhanced image not only maintains the original visual information, but also presents more details.

[0108] The super-resolution process may increase the size of the first image block. The scaling factor may be calculated based on the desired size of the second image block and the size of the first image block:

[0109]

[0110] Among them, Scale factor represents the scaling factor, target_size represents the desired target size (ie, the size of the second image block), and InputSize represents the size of the first image block.

[0111] Feature enhancement processing can be achieved through feature enhancement functions. The feature enhancement functions are as follows:

[0112] F enhanced (x,y)=G(F original (x,y),Scale_factor) (8)

[0113] Exemplarily, the process of performing image optimization processing on the first image block may refer to the following code:

[0114]

[0115]

[0116] By decomposing image blocks and performing image optimization processing on a block-by-block basis, the optimization effect can be improved and problems such as image distortion and blur can be alleviated.

[0117] Continue to refer Figure 1 In step S140, fusion information between different second image blocks is determined, multiple second image blocks are combined based on the fusion information, and a panoramic image is obtained according to the combination result; the fusion information includes first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

[0118] When the second image is decomposed into first image blocks, there may be overlapping areas between different first image blocks, and after the first image blocks are processed by image optimization to form second image blocks, there may be content discontinuity between adjacent second image blocks. If the second image blocks are directly spliced ​​and combined, the resulting image as a whole will have problems such as inconsistent pictures, poor content continuity, and sudden picture changes at the splicing points. Therefore, fusion areas can be created between different second image blocks, and the fusion areas can achieve picture transition effects and improve picture consistency and continuity. Fusion information is used to indicate how different second image blocks are fused, and may include but is not limited to information such as fusion areas and fusion weights.

[0119] In this exemplary embodiment, the second image blocks at the first edge and the second edge that are opposite to each other are fused. For example, the first edge is the left edge of the panoramic image, and the second edge is the right edge of the panoramic image, or the first edge is the top edge of the panoramic image, and the second edge is the bottom edge of the panoramic image. A first fusion area is set between the second image blocks at the first edge and the second edge, and the second image blocks at the first edge and the second edge are fused to obtain an image of the first fusion area. The first fusion area can be set at the first edge of the panoramic area, or at the second edge, or at the first edge and the second edge. It should be understood that by fusing the second image blocks at the first edge and the second edge, the first edge and the second edge of the panoramic image have picture continuity, which meets the characteristics of the panoramic image.

[0120] In one implementation, feature points may be extracted from the second image blocks, and feature point matching may be performed on second image blocks to be fused (such as adjacent second image blocks, or second image blocks at the first edge and the second edge), and fusion information may be determined based on the matching results.

[0121] In one embodiment, the fusion information includes first fusion information and second fusion information between adjacent second image blocks. The first fusion information includes a first fusion area and first fusion weights corresponding to different positions in the first fusion area. The second fusion information includes a second fusion area and second fusion weights corresponding to different positions in the second fusion area. Figure 3 As shown, the above-mentioned determination of fusion information between different second image blocks and combination of multiple second image blocks based on the fusion information may include the following steps S310 to S330:

[0122] Step S310, determining a first fusion size, and determining a first fusion area between a second image block located at a first edge and a second image block located at a second edge according to the first fusion size; determining a second fusion size, and determining a second fusion area between adjacent second image blocks according to the second fusion size.

[0123] The first fusion area refers to the fusion area between the second image blocks at the first edge and the second edge, and the first fusion size refers to the size of the first fusion area. The second fusion area refers to the fusion area between adjacent second image blocks, and the second fusion size refers to the size of the second fusion area. If the second image block is fused along the image width direction (i.e., the horizontal direction), the first fusion size or the second fusion size includes the fusion width (blend_width). If the second image block is fused along the image height direction (i.e., the vertical direction), the first fusion size or the second fusion size includes the fusion height (blend_height).

[0124] The first fusion size and the second fusion size can be determined according to one or more of factors such as the size of the second image block, the size of the second image, and the image complexity of the second image. Exemplarily, the fusion ratio can be set manually or determined according to the image complexity of the second image, and the second fusion size is determined according to the fusion ratio and the size of the second image block. For example, the fusion ratio can be 10%, and the second fusion size is 10% of the size of the second image block. The first fusion size is determined according to the size of the second image. In one embodiment, the first fusion size can be smaller than the second fusion size to avoid transitional fusion of image edges and maintain edge information.

[0125] In one embodiment, the first fusion size or the second fusion size may be determined by the following formula:

[0126]

[0127] The first fusion area is determined according to the first fusion size, including the position of the first fusion area, the portion of the second image block at the first edge and the second edge corresponding to the first fusion area (i.e., determining which portions of the second image block at the first edge and the second edge participate in the fusion), etc. The second fusion area is determined according to the second fusion size, including the position of the second fusion area, the portion of the adjacent second image block corresponding to the second fusion area (i.e., determining which portions of the adjacent second image block participate in the fusion), etc.

[0128] Step S320, determining first fusion weights corresponding to different positions in the first fusion area according to a preset first function, and determining second fusion weights corresponding to different positions in the second fusion area according to a preset second function.

[0129] The first fusion weight refers to the fusion weight of the two second image blocks corresponding to the first edge and the second edge at each position of the first fusion area. The second fusion weight refers to the fusion weight corresponding to the two adjacent second image blocks at each position of the second fusion area. Both the first function and the second function can be continuous functions, so that the first fusion weight in the first fusion area changes continuously with the change of position, and the second fusion weight in the second fusion area also changes continuously with the change of position, so that a smooth transition of the first fusion weight and the second fusion weight is achieved in the first fusion area and the second fusion area, and progressive fusion is achieved.

[0130] Several function examples are provided below, and the first function and the second function can be any of these functions.

[0131] ①Linear function:

[0132]

[0133] Among them, x and y are the horizontal and vertical coordinates respectively, W v (y) represents the fusion weight in the vertical direction, W h (x) represents the fusion weight in the horizontal direction, I 1 (x,y),I 2 (x, y) are the values ​​of (x, y) in the two second image blocks to be fused, I blend (x,y) represents the fused value.

[0134] ②Power function or polynomial function

[0135]

[0136]

[0137] Wherein, p represents the relative coordinate value in the first or second fusion area, and W(p) represents the fusion weight in the horizontal or vertical direction.

[0138] ③Trigonometric functions

[0139]

[0140] In one embodiment, the first function and the second function may be different functions, for example, the first function is a nonlinear function and the second function is a linear function, so that different fusion methods are used in the edge area and the middle adjacent area to meet the image content characteristics of different areas in the panoramic image.

[0141] In one implementation, the determining of the first fusion weights corresponding to different positions in the first fusion area according to the preset first function may include the following steps:

[0142] Substituting the position parameter of the target position in the first fusion area into the first function to obtain the position weight corresponding to the target position;

[0143] A first fusion weight corresponding to the target position is determined according to a position weight corresponding to the target position, a structural similarity of the target position, and a content complexity of the target position.

[0144] Among them, the target position can be any position in the first fusion area. Exemplarily, any one of the above formulas (10) to (13) can be used as the first function, and the position parameters of the target position (such as position coordinates x, y, or relative coordinates p in the first fusion area) are substituted into the first function to calculate the corresponding weight, which is called the position weight. In addition, the structural similarity and content complexity of the image can also be considered to determine the first fusion weight by comprehensively considering the three factors. Generally, the higher the structural similarity, the larger the corresponding weight, and the higher the content complexity, the smaller the corresponding weight. In this exemplary embodiment, the larger the weight, it can be understood that the weights corresponding to the two second image blocks used for fusion are closer, the higher the degree of fusion, and the smaller the weight. It can be understood that the greater the difference in weights corresponding to the two second image blocks used for fusion, the tendency is to make the fusion result retain more of the content in one of the second image blocks.

[0145] You can refer to the following formula:

[0146] W adptive (x,y)=W(p)·S(x,y)·C(x,y) (13)

[0147] Among them, (x, y) represents the target position, S(x, y) represents the weight corresponding to the structural similarity, which is usually positively correlated with the structural similarity, C(x, y) represents the weight corresponding to the content complexity, which is usually negatively correlated with the content complexity, and W adaptive (x,y) represents the first fusion weight calculated by comprehensively considering three factors.

[0148] You can refer to the following code to determine the first fusion weight:

[0149]

[0150]

[0151] The final first fusion weight is the comprehensive result of three factors: position, structural similarity, and content complexity. For example, if it is near the fusion boundary, the structure is similar, and the regional content is simple, then the position weight is larger, the weight corresponding to the structural similarity is larger, and the weight corresponding to the content complexity is smaller or always, and a larger first fusion weight is calculated. If it is far away from the boundary, the position weight is smaller, even if the weight corresponding to the structural similarity and the weight corresponding to the content complexity are larger, the calculated first fusion weight will be smaller. If the area is very complex, the weight corresponding to the content complexity is smaller, so that the details can be protected.

[0152] The above-mentioned adaptive weight scheme enables the fusion process to be automatically adjusted according to the characteristics of the image content and adapt to different image scenes, thereby obtaining a better fusion effect.

[0153] Step S330, combining multiple second image blocks, fusing the second image block located at the first edge with the second image block located at the second edge according to the first fusion weight to obtain an image of the first fusion area, and fusing adjacent second image blocks according to the second fusion weight to obtain an image of the second fusion area.

[0154] Figure 4 A schematic diagram of combining the second image blocks is shown. For the first fusion region, the second image block located at the first edge (such as the leftmost side) and the second image block located at the second edge (such as the rightmost side) can be weighted fused according to the first fusion weight to obtain the image of the first fusion region. For the second fusion region, two adjacent second image blocks (which can be left-right adjacent second image blocks or top-bottom adjacent second image blocks) can be weighted fused according to the second fusion weight to obtain the image of the second fusion region.

[0155] Exemplarily, the process of fusing the second image block may refer to the following code:

[0156]

[0157]

[0158] based on Figure 3 The method can eliminate the splicing traces between the second image blocks, maintain the continuity of the texture, and enhance the naturalness of the visual effect.

[0159] In one implementation, combining the plurality of second image blocks based on the fusion information and obtaining the panoramic image according to the combination result may include the following steps:

[0160] Combining the plurality of second image blocks based on the fusion information to obtain a third image;

[0161] The third image is globally optimized for consistency based on constraint conditions, and the optimized third image is used as a panoramic image; the constraint conditions include: minimizing the gradient of the optimized third image and the difference between the optimized third image and a reference image; the reference image includes one or more of the first image, the second image, and the third image.

[0162] The fusion of the second image block ensures the image continuity of the local area. On this basis, the third image is globally optimized for consistency, further optimizing the semantic consistency and picture continuity of the image from the global level.

[0163] Iterative optimization can be performed based on constraints. For example, the constraints can refer to the following expression:

[0164]

[0165] Where I(x,y) represents any pixel point of the optimized third image. represents the gradient of the optimized third image, I_ref represents the reference image, and the second image or the third image before optimization can be used as the reference image. Ω represents the overlapping area between the optimized third image and the reference image, and λ is the coefficient for balancing the two constraints, which can be set based on experience.

[0166] The first part of the above expression, i.e., the part about the gradient, aims to minimize the overall change of the image. The second part of the expression, i.e., the part about the image difference, aims to minimize the difference between the optimized third image and the reference image.

[0167] After each round of iterative optimization of the third image, the constraint value can be calculated by expression (14). If the constraint value converges, the iteration can be terminated and the final third image, i.e., the panoramic image, can be output. Of course, the condition for the end of the iteration can also include other conditions, such as the constraint value converges for multiple consecutive rounds of iterations, reaches a predetermined number of iterations, and the constraint value is less than a preset threshold.

[0168] The process of global consistency optimization can refer to the following code:

[0169]

[0170]

[0171] Through global consistency optimization, we can ensure the natural transition of image boundaries, eliminate obvious stitching marks, maintain the continuity of image content, and improve the overall visual effect.

[0172] Figure 5The schematic diagram of generating a panoramic image when a first image is input is shown. Based on the first image, new image content can be expanded and generated and fused with the first image. Based on the block optimization method, the panoramic image has better detail information, and by fusing image blocks and optimizing global consistency, the panoramic image is guaranteed to have global semantic consistency and picture continuity.

[0173] The exemplary embodiment of the present disclosure also provides a panoramic image generation device. Figure 6 As shown, the panoramic image generating device 600 may include the following program modules:

[0174] The input information acquisition module 610 is configured to acquire input information; the input information includes at least one of text prompt information and a first image;

[0175] The second image generation module 620 is configured to process the input information through an image generation model to generate a second image;

[0176] An image block processing module 630 is configured to decompose the second image into a plurality of first image blocks, and perform image optimization processing on the first image blocks to obtain a plurality of corresponding second image blocks;

[0177] The image block combination module 640 is configured to determine fusion information between different second image blocks, combine the multiple second image blocks based on the fusion information, and obtain a panoramic image according to the combination result; the fusion information includes first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

[0178] In one embodiment, the processing of the input information by an image generation model to generate a second image includes: extracting features from the input information by the image generation model to obtain initial prompt features, and generating initial noise features, and processing the initial prompt features and the initial noise features to obtain the second image.

[0179] In one embodiment, the input information includes the first image, and the initial prompt feature includes an initial image feature; extracting features from the input information through the image generation model to obtain the initial prompt feature includes: extracting features of multiple scales from the first image through the image generation model, and fusing the features of the multiple scales to obtain the initial image feature.

[0180] In one embodiment, the processing of the initial prompt feature and the initial noise feature includes: determining the process parameters corresponding to each time step; and processing the initial prompt feature and the initial noise feature in turn according to the process parameters corresponding to each time step.

[0181] In one embodiment, decomposing the second image into multiple first image blocks includes: determining a block size based on image complexity of the second image and the amount of image processing resources; determining an overlap size based on the block size and an overlap coefficient; and decomposing the second image into multiple first image blocks according to the block size and the overlap size, so that the size of the overlap area between adjacent first image blocks is the overlap size.

[0182] In one embodiment, performing image optimization processing on the first image block to obtain corresponding multiple second image blocks includes: performing super-resolution processing on the first image block, and performing feature enhancement processing on the first image block after super-resolution processing to obtain corresponding multiple second image blocks.

[0183] In one embodiment, the fusion information includes the first fusion information and the second fusion information between adjacent second image blocks; the first fusion information includes the first fusion weights corresponding to different positions in the first fusion area and the first fusion area; the second fusion information includes the second fusion weights corresponding to different positions in the second fusion area and the second fusion area; the determining of the fusion information between different second image blocks and combining the multiple second image blocks based on the fusion information includes: determining a first fusion size, and determining a first fusion area between a second image block located at a first edge and a second image block located at a second edge according to the first fusion size; determining a second fusion size, and determining a second fusion area between adjacent second image blocks according to the second fusion size; determining the first fusion weights corresponding to different positions in the first fusion area according to a preset first function, and determining the second fusion weights corresponding to different positions in the second fusion area according to a preset second function; combining the multiple second image blocks, fusing the second image block located at the first edge with the second image block located at the second edge according to the first fusion weight to obtain an image of the first fusion area, and fusing the adjacent second image blocks according to the second fusion weight to obtain an image of the second fusion area.

[0184] In one embodiment, determining the first fusion weights corresponding to different positions in the first fusion area according to a preset first function includes: substituting the position parameters of the target position in the first fusion area into the first function to obtain the position weight corresponding to the target position; determining the first fusion weight corresponding to the target position according to the position weight corresponding to the target position, the structural similarity of the target position, and the content complexity of the target position.

[0185] In one embodiment, combining the multiple second image blocks based on the fusion information and obtaining a panoramic image according to the combination result includes: combining the multiple second image blocks based on the fusion information to obtain a third image; performing global consistency optimization on the third image based on constraints, and using the optimized third image as the panoramic image; the constraints include: minimizing the gradient of the optimized third image and the difference between the optimized third image and a reference image; the reference image includes one or more of the first image, the second image, and the third image.

[0186] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0187] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0188] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0189] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0190] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0191] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0192] The computer program can be carried or transmitted by electrical, magnetic, optical, electromagnetic, infrared and other signals. The electronic device can convert the signal carrying the computer program into a digital signal, and then run the computer program. When the computer program is running on the electronic device, its code is used to make the electronic device execute (more specifically, the processor of the electronic device can execute) the method steps of various exemplary embodiments of the present disclosure, such as the following steps: step S110, obtaining input information; the input information includes at least one of the text prompt information and the first image; step S120, processing the input information through the image generation model to generate a second image; step S130, decomposing the second image into a plurality of first image blocks, performing image optimization processing on the first image block, and obtaining a plurality of corresponding second image blocks; step S140, determining the fusion information between different second image blocks, combining the plurality of second image blocks based on the fusion information, and obtaining a panoramic image according to the combination result; the fusion information includes the first fusion information between the second image block located at the first edge and the second image block located at the second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

[0193] The above method steps are implemented by a computer program. After obtaining the input information, the input information is processed by the image generation model to generate a second image, the second image is decomposed into the first image blocks, and the second image blocks are obtained through image optimization processing. The second image blocks are combined based on the fusion information between the second image blocks to obtain a panoramic image. On the one hand, this scheme realizes the automatic generation of panoramic images without the need for manual post-processing, improves the generation efficiency, and reduces the labor and time costs. On the other hand, based on the generation of the second image by the image generation model, a block optimization method is adopted to improve the detail information in the image. In addition, by fusing the second image blocks, especially fusing the second image blocks at the opposite edges, the global consistency of the image and the continuity of the content on both sides are improved, the characteristics of the panoramic image are met, and the problems such as picture distortion are improved, thereby improving the quality of the panoramic image.

[0194] The exemplary embodiments of the present disclosure also provide an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions.

[0195] Reference below Figure 7 , the electronic device is exemplified in the form of a general-purpose computing device. It should be understood that Figure 7 The electronic device 700 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0196] like Figure 7 As shown, the electronic device 700 may include: a processor 710 , a memory 720 , a bus 730 , an I / O (input / output) interface 740 , and a network adapter 750 .

[0197] The memory 720 may include a volatile memory, such as a RAM 721, a cache unit 722, and may also include a non-volatile memory, such as a ROM 723. The memory 720 may also include one or more program modules 724, such program modules 724 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 724 may include each module in the above-mentioned device.

[0198] The processor 710 may include one or more processing units. For example, the processor 710 may include an AP (Application Processor), a modem processor, a GPU, an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit) and other processing units.

[0199] The processor 710 can be used to execute executable instructions stored in the memory 720, such as executing the following steps: Step S110, obtaining input information; the input information includes text prompt information and at least one of the first image; Step S120, processing the input information through an image generation model to generate a second image; Step S130, decomposing the second image into multiple first image blocks, performing image optimization processing on the first image blocks, and obtaining corresponding multiple second image blocks; Step S140, determining fusion information between different second image blocks, combining multiple second image blocks based on the fusion information, and obtaining a panoramic image according to the combination result; the fusion information includes first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

[0200] The processor 710 executes the above method steps, obtains the input information, processes the input information through the image generation model, generates the second image, decomposes the second image into the first image block, and obtains the second image block through image optimization processing, and combines the second image blocks based on the fusion information between the second image blocks to obtain the panoramic image. On the one hand, this scheme realizes the automatic generation of panoramic images without the need for manual post-processing, improves the generation efficiency, and reduces the labor and time costs. On the other hand, based on the generation of the second image by the image generation model, the block optimization method is adopted to improve the detail information in the image. In addition, by fusing the second image blocks, especially fusing the second image blocks at the opposite edges on both sides, the global consistency of the image and the continuity of the content on both sides are improved, the characteristics of the panoramic image are met, and the problems such as picture distortion are improved, thereby improving the quality of the panoramic image.

[0201] The bus 730 is used to realize the connection between different components of the electronic device 700, and may include a data bus, an address bus, and a control bus.

[0202] The electronic device 700 can communicate with one or more external devices 800 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 740 .

[0203] The electronic device 700 can communicate with one or more networks through the network adapter 750. For example, the network adapter 750 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 750 can communicate with other modules of the electronic device 700 through the bus 730.

[0204] although Figure 7 Not shown, other hardware and / or software modules may also be provided in the electronic device 700, including but not limited to: a display, a microcode, a device driver, a redundant processor, an external disk drive array, a tape drive, and a data backup storage system.

[0205] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0206] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A method for generating a panoramic image, characterized in that: The method comprises: Acquire input information; the input information includes at least one of text prompt information and a first image; Processing the input information through an image generation model to generate a second image; Decomposing the second image into a plurality of first image blocks, performing image optimization processing on the first image blocks, and obtaining a plurality of corresponding second image blocks; Determine fusion information between different second image blocks, combine the multiple second image blocks based on the fusion information, and obtain a panoramic image according to the combination result; the fusion information includes first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

2. The method according to claim 1, characterized in that The step of processing the input information by using an image generation model to generate a second image includes: The image generation model is used to extract features from the input information to obtain initial prompt features, and to generate initial noise features. The initial prompt features and the initial noise features are processed to obtain the second image.

3. The method according to claim 2, characterized in that The input information includes the first image, and the initial prompt feature includes an initial image feature; The extracting features from the input information by the image generation model to obtain initial prompt features includes: The image generation model is used to extract features of multiple scales from the first image, and the features of the multiple scales are fused to obtain the initial image features.

4. The method according to claim 2, characterized in that: The processing of the initial prompt feature and the initial noise feature includes: Determine the process parameters corresponding to each time step; According to the process parameters corresponding to each time step, the initial prompt feature and the initial noise feature are processed in turn according to each time step.

5. The method according to claim 1, characterized in that The step of decomposing the second image into a plurality of first image blocks comprises: determining a block size according to an image complexity of the second image and a resource amount of image processing resources; Determining an overlap size according to the block size and the overlap coefficient; The second image is decomposed into a plurality of first image blocks according to the block size and the overlap size, so that the size of the overlap area between adjacent first image blocks is the overlap size.

6. The method according to claim 1, characterized in that The performing image optimization processing on the first image block to obtain a corresponding plurality of second image blocks includes: The first image block is subjected to super-resolution processing, and feature enhancement processing is performed on the first image block after the super-resolution processing to obtain a corresponding plurality of second image blocks.

7. The method according to claim 1, characterized in that The fusion information includes the first fusion information and second fusion information between adjacent second image blocks; the first fusion information includes a first fusion area and first fusion weights corresponding to different positions in the first fusion area; the second fusion information includes the second fusion area and second fusion weights corresponding to different positions in the second fusion area; The determining of fusion information between different second image blocks, and combining the plurality of second image blocks based on the fusion information, comprises: Determine a first fusion size, and determine a first fusion area between a second image block located at a first edge and a second image block located at a second edge according to the first fusion size; determine a second fusion size, and determine a second fusion area between adjacent second image blocks according to the second fusion size; Determine first fusion weights corresponding to different positions in the first fusion area according to a preset first function, and determine second fusion weights corresponding to different positions in the second fusion area according to a preset second function; Combine the multiple second image blocks, fuse the second image block located at the first edge with the second image block located at the second edge according to the first fusion weight to obtain the image of the first fusion area, and fuse the adjacent second image blocks according to the second fusion weight to obtain the image of the second fusion area.

8. The method according to claim 7, characterized in that The determining, according to a preset first function, first fusion weights corresponding to different positions in the first fusion area includes: Substituting the position parameter of the target position in the first fusion area into the first function to obtain a position weight corresponding to the target position; A first fusion weight corresponding to the target position is determined according to a position weight corresponding to the target position, a structural similarity of the target position, and a content complexity of the target position.

9. The method according to any one of claims 1 to 8, characterized in that: The combining the plurality of second image blocks based on the fusion information and obtaining a panoramic image according to the combination result includes: Combining the plurality of second image blocks based on the fusion information to obtain a third image; The third image is globally optimized for consistency based on constraint conditions, and the optimized third image is used as the panoramic image; the constraint conditions include: minimizing the gradient of the optimized third image and the difference between the optimized third image and a reference image; the reference image includes one or more of the first image, the second image, and the third image.

10. A panoramic image generating device, characterized in that: The device comprises: An input information acquisition module is configured to acquire input information; the input information includes at least one of text prompt information and a first image; A second image generation module is configured to process the input information through an image generation model to generate a second image; an image block processing module, configured to decompose the second image into a plurality of first image blocks, and perform image optimization processing on the first image blocks to obtain a corresponding plurality of second image blocks; The image block combination module is configured to determine fusion information between different second image blocks, combine the multiple second image blocks based on the fusion information, and obtain a panoramic image according to the combination result; the fusion information includes first fusion information between a second image block located at a first edge and a second image block located at a second edge, and the first edge and the second edge are two edges opposite to each other in the same direction in the panoramic image.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

Images