Panoramic image generation method and apparatus, device, and storage medium

Abstract

The present disclosure provides a panoramic image generation method and apparatus, a device, and a storage medium. The method comprises: obtaining text description information, and extracting text features of the text description information; generating an initial feature index, and extending the initial feature index into a feature index to be processed that conforms to panoramic image features; on the basis of the text features, generating a predicted feature index corresponding to the feature index to be processed; and converting the predicted feature index into a converted feature index that conforms to the panoramic image features, and, on the basis of the converted feature index, generating a panoramic image corresponding to the text description information. According to the technical solutions provided by one or more embodiments of the present disclosure, a panoramic image can be conveniently generated.

Description

Method, device, equipment and storage medium for generating panoramic images

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 16, 2023, with application number 202311035518.4 and invention name “A method, device, equipment and storage medium for generating panoramic images”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for generating a panoramic image. Background Art

[0003] Panoramic images play a certain role in high-realism rendering. They not only provide a natural skybox background for the rendering program, but also can be used as a light source to illuminate the rendered model in certain scenes, giving the rendered model realistic light and shadow effects.

[0004] Panoramic images are typically acquired through photography. Specifically, a metal ball is placed in the target scene to reflect the surroundings. The ball is then photographed at varying exposures, and the resulting panoramic image is synthesized. This method requires the deployment of specialized equipment and devices in the actual scene, which poses significant limitations in terms of convenience, operability, and scalability.

[0005] In view of this, a more convenient method for generating panoramic images is currently needed.

[0006] Summary of the Invention

[0007] One or more embodiments of the present disclosure provide a method, apparatus, device, and storage medium for generating a panoramic image, which can conveniently generate a panoramic image.

[0008] On the one hand, the present disclosure provides a method for generating a panoramic image, which includes: obtaining text description information and extracting text features of the text description information; generating an initial feature index and expanding the initial feature index into a feature index to be processed that conforms to panoramic image features; based on the text features, generating a predicted feature index corresponding to the feature index to be processed; converting the predicted feature index into a converted feature index that conforms to panoramic image features, and generating a panoramic image corresponding to the text description information based on the converted feature index.

[0009] On the other hand, the present disclosure also provides a device for generating a panoramic image, which includes: a text feature extraction unit for obtaining text description information and extracting text features of the text description information; an initialization unit for generating an initial feature index and expanding the initial feature index into a feature index to be processed that conforms to the panoramic image features; a prediction unit for generating a predicted feature index corresponding to the feature index to be processed based on the text features; and a panoramic image generation unit for converting the predicted feature index into a converted feature index that conforms to the panoramic image features, and generating a panoramic image corresponding to the text description information based on the converted feature index.

[0010] On the other hand, the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned method for generating a panoramic image is implemented.

[0011] On the other hand, the present disclosure further provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the above-mentioned method for generating a panoramic image is implemented.

[0012] The technical solutions provided by one or more embodiments of the present disclosure can generate a corresponding panoramic image based on text description information input by a user. Specifically, by extracting text features from the text description information, these text features can be used as prior conditions for generating the panoramic image. Under the constraints of these prior conditions, a corresponding predicted feature index can be generated. By processing the predicted feature index, a panoramic image that matches the text description information can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The features and advantages of the various embodiments of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0014] FIG1 shows a schematic diagram of a data processing flow of a panoramic image generation model in one embodiment of the present disclosure;

[0015] FIG2 is a schematic diagram showing steps of a method for training a panoramic image generation model in one embodiment of the present disclosure;

[0016] FIG3 is a schematic diagram showing steps of a method for generating a panoramic image in one embodiment of the present disclosure;

[0017] FIG4 shows a schematic diagram of generating a panoramic image based on text description information in one embodiment of the present disclosure;

[0018] FIG5 is a schematic diagram showing the functional modules of a device for generating a panoramic image in one embodiment of the present disclosure; and

[0019] FIG6 shows a schematic structural diagram of an electronic device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0021] With the continuous development of deep learning technology, related technologies can restore panoramic images through input flat images. However, the flat images in this technology also need to be photographed or collected from the actual scene. If the actual scene is not easy to find or cannot be realized, the required panoramic image cannot be generated. In view of this, one embodiment of the present disclosure can generate a corresponding panoramic image based only on the text description information input by the user. In this way, on the one hand, it can get rid of the limitations of the actual scene, making the panoramic image generation process more convenient. On the other hand, it can meet the user's imagination and actual needs as much as possible, making the generated panoramic image more diverse and in line with user needs.

[0022] Based on the above ideas, if conventional deep learning technology is used to achieve the conversion from text description information to panoramic images, a large amount of training data is usually required. This training data can include matching text description information and panoramic images. In this way, by training the mapping relationship between text description information and panoramic images with a large amount of training data, an accurate panoramic image generation model can be obtained. However, the above-mentioned massive training data does not currently exist in the relevant field, resulting in this method being unable to train a highly accurate panoramic image generation model. In addition, panoramic images are essentially spherical signals represented by equidistant projections, which have certain image distortions. At the same time, the images at the left and right boundaries of the image need to be consistent. How to generate panoramic images that meet these conditions is also a current technical problem.

[0023] In light of this, one embodiment of the present disclosure provides a method for generating panoramic images. This method, by combining discretized encoding and a prediction module (e.g., a transformer), constructs a training set that characterizes the mapping relationship between text description information and panoramic images. Based on this training set, a qualified panoramic image generation model is trained. Subsequently, by inputting the text description information into the panoramic image generation model, the corresponding panoramic image can be output.

[0024] Please refer to Figure 1. In a specific application example, the panoramic image generation model to be trained may include an encoder, a codebook, a decoder, and a prediction module (transformer). The panoramic image generation model can be trained through the multiple steps shown in Figure 2.

[0025] S11: Obtain panoramic image training samples, generate text description samples of the panoramic image training samples, and extract text features of the text description samples.

[0026] In this embodiment, considering that panoramic images can be viewed as images expanded from a unit sphere, they are essentially spherical signals represented by equidistant projections, exhibiting some image distortion. Furthermore, the images at the left and right boundaries of the panoramic image are consistent. Therefore, panoramic images can be used as panoramic image training samples. After obtaining the panoramic image training samples, text description information for the panoramic image training samples can be generated, and this text description information can serve as the corresponding text description samples. Through the above method, training sample pairs consisting of panoramic image training samples and text description samples can be constructed.

[0027] Specifically, when generating text description samples for panoramic image training samples, the panoramic image training samples can be dedistorted, thereby converting them from panoramic images into two-dimensional images. The images in the two-dimensional images are more consistent with the real scene. By intercepting one or more semantic images from the two-dimensional images and then extracting text descriptions for each semantic image, the combination of the extracted text descriptions can be used as the text description sample for the panoramic image training samples.

[0028] There are various ways to dedistort the panoramic image training samples. For example, the panoramic image training samples can be converted into images in a cube map format. In this way, in the cube map format image, the upper surface can correspond to the sky area at the top of the panoramic image training sample, the lower surface can correspond to the ground area at the bottom of the panoramic image training sample, and the four horizontal surfaces can contain most of the information in the panoramic image training sample. Therefore, these four horizontal surfaces can be used as semantic images in the cube map format image. Through image captioning or other image description technologies, text descriptions of each semantic image can be extracted. Finally, the combination of these text descriptions can be used as a text description sample of the panoramic image training sample.

[0029] Of course, converting panoramic image training samples into images in cubemap format is just one way to generate text description samples. In practical applications, panoramic image training samples can correspond to a camera model, which can be used to perform dedistortion correction on the panoramic image training samples. For example, for a fisheye camera, its camera model can be a KBCM model, which can be used to perform dedistortion correction on panoramic images captured by the fisheye camera. After dedistortion correction, a planar image corresponding to the panoramic image training sample can be obtained. By intercepting a semantic image with rich content from this planar image, text description samples for the panoramic image training sample can be generated based on the text description of the semantic image.

[0030] Through the above-mentioned method, a large number of panoramic image training samples can be processed to generate training sample pairs of panoramic image training samples and text description samples, thereby constructing a training set with a rich number of samples.

[0031] In this embodiment, after generating a text description sample, text features of the text description sample can be extracted. Specifically, the text features can be CLIP features (Text CLIP), which can be used to train a prediction module in a panoramic image generation model.

[0032] S13: Using the panoramic image training samples, the encoder, the encoding directory, and the decoder in the panoramic image generation model are trained.

[0033] Referring to FIG1 , in this embodiment, the encoder in the panoramic image generation model can extract sample features of the panoramic image training samples. In practical applications, the sample features can be viewed as a three-dimensional matrix of M*L*K (K is not shown in FIG1 ), where K represents the number of feature extraction channels of the encoder and M*L represents the feature dimension extracted by each channel. For example, in FIG1 , both M and L can be 3, and the features extracted by each channel are a 3*3 matrix. After the sample features are extracted, each discrete feature in the sample features can be quantized into a feature index based on the codebook.

[0034] Specifically, the coding directory may include a mapping relationship between feature indexes and discrete features. Taking Figure 1 as an example, the serial numbers 0 to N-1 in front can be used as feature indexes, and the color block after each serial number can represent discrete features. When quantizing the discrete features in the sample features into feature indexes, the similarity between the discrete features in the sample features and the discrete features in the coding directory can be calculated, and the feature index corresponding to the most similar discrete feature in the coding directory can be used as the feature index of the discrete feature in the sample features. In the above manner, each discrete feature in the sample features can be matched one by one in the coding directory, so that each discrete feature in the sample features can be quantized to obtain the sample feature index corresponding to the sample feature. In Figure 1, the 3*3 matrix composed of 9 numbers such as 5, 3, 21, etc. can be used as the quantized sample feature index.

[0035] In this embodiment, after obtaining the sample feature index, a decoder can be used to decode the sample feature index, thereby restoring the sample feature index to an image. In this way, a decoded image corresponding to the sample feature index can be generated. It can then be determined whether the decoded image is a true image. This determination process can be performed by calculating the error between the decoded image and the input panoramic image training sample. If the error is greater than a specified threshold, the decoded image is considered not to be a true image. If the error is less than the specified threshold, the decoded image is considered to be a true image.

[0036] In the above process, in the process of converting sample features into quantized sample feature indexes, since similarity calculation is involved, there is a certain error between the sample features and the sample feature indexes, and this error can be represented by a feature quantization error function. In addition, there will also be an error between the decoded image and the panoramic image training sample, and this error can be represented by a network adversarial loss function. In this way, based on the judgment result of whether the decoded image is a real image, an error function can be generated. The error function can be the above-mentioned feature quantization error function and the network adversarial loss function, so that the encoder, encoding directory and decoder are corrected by the error function. After training with a large number of training samples, the encoder, encoding directory and decoder can restore the corresponding panoramic image after processing the input panoramic image.

[0037] S15: Based on the trained encoder and encoding directory, a standard feature index of the panoramic image training sample is generated.

[0038] S17: Using the text features of the text description sample as a priori conditions, driving the prediction module in the panoramic image generation model to generate a prediction feature index, and training the prediction module in the panoramic image generation model by comparing the prediction feature index with the standard feature index.

[0039] In this embodiment, after the encoder, encoding directory, and decoder are trained, the prediction module in the panoramic image generation model can be trained. The text features of the text description samples of the panoramic image training samples can serve as prior conditions to drive the prediction module in the panoramic image generation model to generate a predicted feature index. Specifically, the prediction module can be an iterative process. First, based on the text features, a first index prediction value (e.g., 5 in FIG1 ) can be generated. Then, based on the text features and the first index prediction value, a second index prediction value can be generated, and so on, until all the index prediction values ​​(9 index prediction values ​​in FIG1 ) are generated. In this way, each index prediction value can constitute a predicted feature index. To determine whether the predicted feature index generated by the prediction module is correct, a reference is required. This reference can be generated by the trained encoder and encoding directory. Specifically, after the encoder and encoding directory are trained, the encoder can again extract sample features of the panoramic image training samples and then convert the sample features into a standard feature index based on the encoding directory. Since the encoder and encoding catalog have already been trained, the converted standard feature index can be used as an accurate reference. By comparing the predicted feature index with the standard feature index, the prediction module in the panoramic image generation model can be trained based on the comparison results. In this process, the prediction module can be trained using an autoregressive method.

[0040] It should be noted that during the training of the prediction module, the original M*L (assuming the number of channels is 1) matrix is ​​flattened into an ML*1 vector. For example, in Figure 1, the standard feature index is a 3*3 matrix, while the predicted feature index obtained by the prediction module is a 9*1 vector. In this case, two feature indices of different dimensions cannot be accurately compared. In view of this, during the training of the prediction module, feature position encoding can be introduced. Through position encoding, feature indices of different dimensions can be aligned. Specifically, for any pixel in the panoramic image training sample, the spherical coordinates of the pixel can be used as the position encoding. The longitude on the sphere can be mapped to the 360-degree angular space of [0, 2π], and the latitude can be mapped to the 180-degree angular space of [0, π]. In this way, the spherical coordinates can be represented by the angle in the mapped angular space. The left and right boundaries of the panoramic image training sample are originally adjacent on the sphere, so the position encoding represented by the spherical coordinates is also relatively similar, satisfying the constraint condition of consistency between the left and right boundaries of the image.

[0041] After completing the training phase of the panoramic image generation model, the panoramic image generation model can be used to process the text description information to generate the corresponding panoramic image.

[0042] Referring to FIG. 3 , an embodiment of the present disclosure provides a method for generating a panoramic image. The method may include the following steps.

[0043] S21: Acquire text description information and extract text features of the text description information.

[0044] In this embodiment, the text description information can be a paragraph of text entered by the user, and the text description information can describe the scene represented by the panoramic image. After obtaining the text description information, CLIP (Contrastive Language-Image Pre-training) features of the text description information can be extracted, and the extracted CLIP features can be used as text features of the text description information. The CLIP feature extraction process can be found in the relevant art and will not be repeated here.

[0045] S23: Generate an initial feature index, and expand the initial feature index into a feature index to be processed that meets the panoramic image feature.

[0046] In this embodiment, each value in the initial feature index can be randomly generated, for example, the initial feature index shown in FIG4 can be randomly generated. The dimension of the initial feature index can be flexibly set according to actual needs, but in general, the aspect ratio of the initial feature index is maintained at 2:1 to conform to the characteristics of the expanded panoramic image.

[0047] In this embodiment, after the initial feature index is randomly generated, in order to meet the constraint condition of left and right boundary continuity, the boundary of the initial feature index may be expanded to generate a feature index to be processed that meets the characteristics of the panoramic image.

[0048] Specifically, referring to FIG4 , a first boundary index and a second boundary index can be determined in the initial feature index. The first boundary index can be a boundary index consisting of 11 and 6 on the left side, and the second boundary index can be a boundary index consisting of 8 and 5 on the right side. The first boundary index can then be added as a first extended index to a position adjacent to the second boundary index, and the second boundary index can be added as a second extended index to a position adjacent to the first boundary index, thereby obtaining the feature index to be processed. In this way, the first extended index and the second extended index can serve as two new boundary indexes for the feature index to be processed.

[0049] S25: Based on the text feature, a predicted feature index corresponding to the feature index to be processed is generated.

[0050] In this embodiment, after the feature index to be processed is generated, the prediction module in the panoramic image generation model can be used to predict the predicted values ​​of each discrete feature in the feature index to be processed, using the text feature generated in step S21 as a priori condition. In the process of generating the predicted value, not only the text feature is required as a priori condition, but also the numerical values ​​of multiple discrete features near the position to be predicted in the feature index to be processed are required as input. Taking Figure 4 as an example, when predicting the predicted value of the discrete feature at the upper left corner position in the feature index to be processed, the numerical values ​​of 8, 11, 5, and 6 are required as inputs to the prediction module. In this way, after iterative prediction by the prediction module, the predicted values ​​of each discrete feature in the feature index to be processed can be generated, thereby obtaining a predicted feature index composed of each predicted value.

[0051] S27: Convert the predicted feature index into a converted feature index that conforms to panoramic image features, and generate a panoramic image corresponding to the text description information according to the converted feature index.

[0052] In this embodiment, the predicted feature index obtained by the prediction module may not conform to the feature of the coherence of the left and right boundaries of the panoramic image. In view of this, the predicted feature index can be converted into a conversion feature index that conforms to the panoramic image feature in a similar manner to step S23.

[0053] Specifically, a third boundary index and a fourth boundary index can be determined in the predicted feature index. Taking Figure 4 as an example, the third boundary index can be a boundary index consisting of 24 and 11, and the fourth boundary index can be a boundary index consisting of 18 and 6. Then, a first target index adjacent to the third boundary index can be determined in the predicted feature index, and a second target index adjacent to the fourth boundary index can be determined. Again taking Figure 4 as an example, the first target index can be an index consisting of 10 and 39, and the second target index can be an index consisting of 27 and 16. Then, the fourth boundary index can be replaced with the first target index, and the third boundary index can be replaced with the second target index, thereby obtaining the converted feature index.

[0054] In this embodiment, after obtaining the conversion feature index, the conversion feature index can be decoded by the decoder in the panoramic image generation model to obtain the generated image corresponding to the conversion feature index. However, due to the previous boundary expansion, the generated image does not meet the requirement of an aspect ratio of 2:1. In view of this, it is necessary to intercept the effective area of ​​the generated image, which can meet the requirement of an aspect ratio of 2:1, and use the image of the effective area as the panoramic image corresponding to the text description information. Specifically, when intercepting the effective area from the generated image, part of the left and right boundaries of the generated image can be discarded, and the remaining image can be used as the image of the effective area.

[0055] In one embodiment, considering that high dynamic range (HDR) panoramic images are difficult to obtain in practical applications, low dynamic range (LDR) panoramic images are often used as training samples during the training phase, and the final generated panoramic image is also an LDR panoramic image.

[0056] If the user ultimately requires an HDR panoramic image, the generated LDR panoramic image must be optimized to obtain the corresponding HDR panoramic image. Specifically, the LDR panoramic image can first be processed using a super-resolution algorithm to increase the resolution of the LDR panoramic image. In practical applications, the super-resolution algorithm can be, for example, the Real-ESRGAN algorithm or other feasible algorithms. After obtaining the LDR panoramic image with improved resolution, it can be converted into an HDR panoramic image based on the LDR panoramic image in various ways. For example, the LDR panoramic image can be processed using Deep Reverse Tone Mapping technology. Specifically, if the dynamic range of the panoramic image corresponding to the generated text description information does not meet specified conditions, the panoramic image can be exposed to different degrees to generate a texture sequence. Then, based on the texture sequence, a panoramic image with a dynamic range that meets the specified conditions can be synthesized. In practical applications, the LDR panoramic image can be used as input, and the up-exposure network can output N images with increased exposure, and the down-exposure network can output N images with reduced exposure. Combined with the input LDR panoramic image, there are a total of 2N+1 LDR images, from which k LDR images can be selected to construct the final HDR panoramic image. For another example, the LDR panoramic image can be converted into an HDR panoramic image using the Single-Image HDR Reconstruction algorithm.

[0057] The technical solution provided by one or more embodiments of the present disclosure can generate a corresponding panoramic image based on the text description information input by the user. Specifically, by extracting the text features of the text description information, the text features can be used as a priori conditions for generating a panoramic image. Under the constraints of the prior conditions, for each discrete feature in the feature index to be processed, a predicted value that meets the text features can be predicted, thereby generating a predicted feature index composed of the predicted values. By processing the predicted feature index, a panoramic image that matches the text description information can be generated.

[0058] As can be seen, the technical solution provided by this disclosure can generate panoramic images based solely on text description information. This not only does not rely on the actual scene, but also does not require the deployment of complex equipment and devices, greatly improving convenience. Furthermore, users can express their needs through text description information, and the panoramic images generated based on this text description information are more closely aligned with the user's actual needs.

[0059] Please refer to Figure 5. An embodiment of the present disclosure also provides a device for generating a panoramic image, which includes: a text feature extraction unit 100, used to obtain text description information and extract text features of the text description information; an initialization unit 200, used to generate an initial feature index and expand the initial feature index into a feature index to be processed that conforms to the panoramic image feature; a prediction unit 300, used to generate a predicted feature index corresponding to the feature index to be processed based on the text feature; and a panoramic image generation unit 400, used to convert the predicted feature index into a converted feature index that conforms to the panoramic image feature, and generate a panoramic image corresponding to the text description information based on the converted feature index.

[0060] The specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0061] Please refer to FIG6 . The present disclosure further provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, the above-mentioned method for generating a panoramic image is implemented.

[0062] The various units described in the above embodiments can be implemented by computer chips or products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0063] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0064] The present disclosure also provides a computer-readable storage medium for storing a computer program. When the computer program is executed by a processor, the above-mentioned method for generating a panoramic image is implemented.

[0065] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0066] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the methods in the aforementioned method embodiments.

[0067] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0068] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0069] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, equipment, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant portions, refer to the descriptions of the method embodiments.

[0070] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

[0071] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for generating a panoramic image, the method comprising: Acquire text description information, and extract text features of the text description information; Generate an initial feature index, and expand the initial feature index into a feature index to be processed that meets the panoramic image feature; Based on the text feature, generating a predicted feature index corresponding to the feature index to be processed; The predicted feature index is converted into a converted feature index that conforms to the panoramic image feature, and a panoramic image corresponding to the text description information is generated according to the converted feature index.

2. The method according to claim 1, wherein expanding the initial feature index to a feature index to be processed that conforms to the panoramic image feature comprises: Determining a first boundary index and a second boundary index in the initial feature index; Using the first boundary index as a first extended index, adding it to a position adjacent to the second boundary index, and using the second boundary index as a second extended index, adding it to a position adjacent to the first boundary index, to obtain the feature index to be processed; The first extended index and the second extended index serve as two boundary indexes of the feature index to be processed.

3. The method according to claim 1, wherein converting the predicted feature index into a converted feature index that conforms to the panoramic image feature comprises: For a third boundary index and a fourth boundary index of the predicted feature index, determining a first target index adjacent to the third boundary index in the predicted feature index, and determining a second target index adjacent to the fourth boundary index; The fourth boundary index is replaced by the first target index, and the third boundary index is replaced by the second target index to obtain the conversion feature index.

4. The method according to claim 1 or 3, wherein generating a panoramic image corresponding to the text description information according to the conversion feature index comprises: Decoding the conversion feature index to obtain a generated image corresponding to the conversion feature index; A valid area of ​​the generated image is intercepted, and an image of the valid area is used as a panoramic image corresponding to the text description information.

5. The method according to claim 1, wherein after generating the panoramic image corresponding to the text description information, the method further comprises: If the dynamic range of the panoramic image corresponding to the text description information does not meet the specified conditions, the panoramic image is exposed to different degrees to generate a texture sequence, and based on the texture sequence, a panoramic image with a dynamic range that meets the specified conditions is synthesized.

6. The method according to claim 1, wherein generating a predicted feature index corresponding to the feature index to be processed based on the text feature comprises: The text feature is used as a priori condition to predict the predicted value of each discrete feature in the feature index to be processed, so as to generate a predicted feature index corresponding to the feature index to be processed.

7. The method according to claim 6, wherein the step of predicting the predicted value of each discrete feature in the feature index to be processed is implemented by a trained panoramic image generation model, and the panoramic image generation model is trained in the following manner: Acquire a panoramic image training sample, generate a text description sample of the panoramic image training sample, and extract text features of the text description sample; Using the panoramic image training samples to train the encoder, encoding directory and decoder in the panoramic image generation model; Generating a standard feature index of the panoramic image training sample based on the trained encoder and the encoding directory; The text features of the text description samples are used as prior conditions to drive the prediction module in the panoramic image generation model to generate a prediction feature index, and the prediction module in the panoramic image generation model is trained by comparing the prediction feature index with the standard feature index.

8. The method according to claim 7, wherein generating a text description sample of the panoramic image training sample comprises: Converting the panoramic image training sample into an image in a cube map format, and determining a semantic image from the image in the cube map format; The text descriptions of the semantic images are extracted, and a combination of the extracted text descriptions is used as a generated text description sample.

9. The method according to claim 7, wherein using the panoramic image training sample to train the encoder, the encoding directory and the decoder in the panoramic image generation model comprises: Extracting sample features of the panoramic image training sample using an encoder, and generating a sample feature index corresponding to the sample feature based on a coding directory; Decoding the sample feature index using a decoder to generate a decoded image corresponding to the sample feature index; It is determined whether the decoded image is a true image, and an error function is generated based on the determination result, so as to correct the encoder, the encoding directory and the decoder through the error function.

10. The method according to claim 7, wherein in the process of training the prediction module in the panoramic image generation model, the method further comprises: For any pixel point in the panoramic image training sample, the spherical coordinates of the pixel point are used as the position encoding of the pixel point, wherein the longitude on the sphere is mapped to an angular space of 360 degrees, and the latitude on the sphere is mapped to an angular space of 180 degrees, and the spherical coordinates are represented by the angle in the mapped angular space.

11. A device for generating a panoramic image, the device comprising: A text feature extraction unit, used to obtain text description information and extract text features of the text description information; An initialization unit, used to generate an initial feature index, and expand the initial feature index into a feature index to be processed that conforms to the panoramic image feature; A prediction unit, configured to generate a predicted feature index corresponding to the feature index to be processed based on the text feature; The panoramic image generation unit is used to convert the predicted feature index into a converted feature index that conforms to the panoramic image feature, and generate a panoramic image corresponding to the text description information according to the converted feature index.

12. An electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images