Panoramic image generation method and device, equipment and storage medium
Through deep learning technology, the panoramic image is generated based on text description information, and the problem of the need for special equipment and equipment in the prior art is solved, and the matching degree of user needs is achieved.
Patent Information
- Application Number
- CN202311035518.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2025-02-21
AI Technical Summary
Acquisition of panoramic images in the prior art requires the deployment of special equipment and equipment in actual scenes, which limits convenience, operability and scalability.
By obtaining text description information, extracting text features, and using deep learning technology to build a panoramic image generation model, and generating panoramic images based on text features.
It realizes the generation of panoramic images without actual scenes and complex equipment, improves convenience and makes the generated panoramic images more in line with user needs.
Smart Images
Figure CN119494966A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating a panoramic image. Background Art
[0002] Panoramic images play a certain role in high-realistic rendering. They not only provide a natural skybox background for the rendering program, but also can be used as a light source to illuminate the rendered model in some scenes, giving the rendered model realistic light and shadow effects.
[0003] In the related art, panoramic images are usually obtained by photography. Specifically, a metal ball can be placed in the target scene, and the metal ball can be used to reflect the environment of the target scene. Then, the metal ball is photographed with different exposures, and a panoramic image is synthesized based on the photographed results. This method requires the deployment of special equipment and devices in the actual scene, which has great limitations in terms of convenience, operability, and scalability.
[0004] In view of this, a more convenient method for generating panoramic images is currently needed. Summary of the invention
[0005] One or more embodiments of the present disclosure provide a method, apparatus, device and storage medium for generating a panoramic image, which can conveniently generate a panoramic image.
[0006] On the one hand, the present disclosure provides a method for generating a panoramic image, the method comprising: acquiring text description information and extracting text features of the text description information; generating an initial feature index and expanding the initial feature index into a feature index to be processed that conforms to panoramic image features; based on the text features, generating a predicted feature index corresponding to the feature index to be processed; converting the predicted feature index into a converted feature index that conforms to panoramic image features, and generating a panoramic image corresponding to the text description information according to the converted feature index.
[0007] On the other hand, the present disclosure also provides a device for generating a panoramic image, the device comprising: a text feature extraction unit, used to obtain text description information and extract text features of the text description information; an initialization unit, used to generate an initial feature index and expand the initial feature index to a feature index to be processed that conforms to the panoramic image feature; a prediction unit, used to generate a predicted feature index corresponding to the feature index to be processed based on the text feature; and a panoramic image generation unit, used to convert the predicted feature index into a converted feature index that conforms to the panoramic image feature, and generate a panoramic image corresponding to the text description information based on the converted feature index.
[0008] On the other hand, the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned method for generating a panoramic image is implemented.
[0009] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating a panoramic image is implemented.
[0010] The technical solution provided by one or more embodiments of the present disclosure can generate a corresponding panoramic image based on the text description information input by the user. Specifically, by extracting the text features of the text description information, the text features can be used as a priori conditions for generating a panoramic image. Under the constraints of the prior conditions, a corresponding predicted feature index can be generated. By processing the predicted feature index, a panoramic image matching the text description information can be generated.
[0011] It can be seen that the technical solution provided by the present disclosure can generate a panoramic image based only on the text description information, which is not only independent of the actual scene, but also does not require the deployment of complex equipment and devices, greatly improving the convenience. At the same time, the user's needs can be expressed through the text description information, and the panoramic image generated based on the text description information is more in line with the user's actual needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The features and advantages of the embodiments of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:
[0013] Figure 1 A schematic diagram of a data processing flow of a panoramic image generation model in one embodiment of the present disclosure is shown;
[0014] Figure 2 A schematic diagram of the steps of a training method for a panoramic image generation model in one embodiment of the present disclosure is shown;
[0015] Figure 3 A schematic diagram of the steps of a method for generating a panoramic image in one embodiment of the present disclosure is shown;
[0016] Figure 4 A schematic diagram showing a method of generating a panoramic image based on text description information in one embodiment of the present disclosure is shown;
[0017] Figure 5 A schematic diagram showing the functional modules of a device for generating a panoramic image in one embodiment of the present disclosure is shown;
[0018] Figure 6A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0020] With the continuous development of deep learning technology, related technologies can restore panoramic images through input flat images. However, the flat images in this technology also need to be photographed or collected from actual scenes. If the actual scene is not easy to find or cannot be realized, the required panoramic image cannot be generated. In view of this, an embodiment of the present disclosure can generate a corresponding panoramic image based only on the text description information input by the user. In this way, on the one hand, it can get rid of the limitations of the actual scene, making the panoramic image generation process more convenient. On the other hand, it can meet the user's imagination and actual needs as much as possible, making the generated panoramic image more diversified and in line with user needs.
[0021] Based on the above ideas, if conventional deep learning technology is used to realize the conversion from text description information to panoramic images, a large amount of training data is usually required, which may include matching text description information and panoramic images. In this way, by training the mapping relationship between text description information and panoramic images with massive training data, an accurate panoramic image generation model can be obtained. However, the above-mentioned massive training data does not exist in the relevant field at present, which makes it impossible for this method to train a panoramic image generation model with high accuracy. In addition, panoramic images are essentially spherical signals represented by equidistant projections, which have certain image distortions. At the same time, the images at the left and right boundaries of the image need to be consistent. How to generate panoramic images that meet these conditions is also a technical problem that currently exists.
[0022] In view of this, an embodiment of the present disclosure provides a method for generating a panoramic image. The method can construct a training set that characterizes the mapping relationship between text description information and a panoramic image by combining discretization coding and a prediction module (such as a transformer), and obtain a panoramic image generation model that meets the requirements based on the training set. Subsequently, by inputting the text description information into the panoramic image generation model, the corresponding panoramic image can be output.
[0023] See also Figure 1In a specific application example, the panoramic image generation model to be trained may include an encoder, a codebook, a decoder, and a prediction module (transformer). The panoramic image generation model can be Figure 2 The training is performed in multiple steps as shown.
[0024] S11: Obtain panoramic image training samples, generate text description samples of the panoramic image training samples, and extract text features of the text description samples.
[0025] In this embodiment, considering that the panoramic image can be regarded as an image expanded by a unit sphere, the panoramic image is essentially a spherical signal represented by an equidistant projection, which also has a certain image distortion, and the images at the left and right boundaries of the panoramic image are also coherent. In view of this, the panoramic image can be used as a panoramic image training sample. After obtaining the panoramic image training sample, the text description information of the panoramic image training sample can be generated, and the text description information can be used as the corresponding text description sample. In the above manner, a training sample pair of the panoramic image training sample and the text description sample can be constructed.
[0026] Specifically, when generating text description samples of panoramic image training samples, the panoramic image training samples can be dedistorted, so as to convert the panoramic image training samples from panoramic images to planar images. The pictures in the planar images are more consistent with the real scenes. By intercepting one or more semantic images from the planar images and then extracting the text descriptions of each semantic image, the combination of the extracted text descriptions can be used as the text description samples of the panoramic image training samples.
[0027] There are various ways to dedistort the panoramic image training samples. For example, the panoramic image training samples can be converted into images in a cube map format, so that in the image in the cube map format, the upper surface can correspond to the sky area at the top of the panoramic image training samples, the lower surface can correspond to the ground area at the bottom of the panoramic image training samples, and the four horizontal surfaces can contain most of the information in the panoramic image training samples. Therefore, these four horizontal surfaces can be used as semantic images in the image in the cube map format. Through image captioning or other image description technologies, text descriptions of each semantic image can be extracted, and finally, the combination of these text descriptions can be used as a text description sample of the panoramic image training samples.
[0028] Of course, converting the panoramic image training samples into images in cube map format is only one way to generate text description samples. In practical applications, the panoramic image training samples may correspond to a certain camera model, through which the panoramic image training samples may be dedistorted. For example, for a fisheye camera, its camera model may be a KBCM model, through which the panoramic image captured by the fisheye camera may be dedistorted. After dedistortion correction, a plane image corresponding to the panoramic image training sample may be obtained, and by intercepting a semantic image with richer content in the plane image, a text description sample of the panoramic image training sample may be generated based on the text description of the semantic image.
[0029] Through the above method, a large number of panoramic image training samples can be processed to generate training sample pairs of panoramic image training samples and text description samples, thereby constructing a training set with a rich number of samples.
[0030] In this embodiment, after the text description sample is generated, the text feature of the text description sample can be extracted. Specifically, the text feature can be a CLIP feature (Text CLIP), which can be used to train the prediction module in the panoramic image generation model.
[0031] S13: Using the panoramic image training samples, the encoder, the encoding directory and the decoder in the panoramic image generation model are trained.
[0032] See also Figure 1 In this embodiment, the encoder in the panoramic image generation model can extract sample features of the panoramic image training samples. In practical applications, the sample features can be regarded as a three-dimensional matrix of M*L*K (in Figure 1 K is not shown in the figure), where K represents the number of feature extraction channels of the encoder, and M*L represents the feature dimension extracted by each channel. Figure 1 In , both M and L can be 3, and the features extracted by each channel are 3*3 matrices. After extracting the sample features, each discrete feature in the sample features can be quantized into a feature index based on the codebook.
[0033] Specifically, the encoding directory may include a mapping relationship between feature indexes and discrete features. Figure 1For example, the serial numbers from 0 to N-1 in front can be used as feature indexes, and the color block after each serial number can represent discrete features. When quantizing the discrete features in the sample features into feature indexes, the similarity between the discrete features in the sample features and the discrete features in the coding directory can be calculated, and the feature index corresponding to the most similar discrete feature in the coding directory can be used as the feature index of the discrete feature in the sample features. Through the above method, each discrete feature in the sample features can be matched one by one in the coding directory, so that each discrete feature in the sample features can be quantified to obtain the sample feature index corresponding to the sample feature. Figure 1 In the example, the 3*3 matrix consisting of 9 numbers such as 5, 3, and 21 can be used as the quantized sample feature index.
[0034] In this embodiment, after obtaining the sample feature index, the sample feature index can be decoded by a decoder, so as to restore the sample feature index to an image, and thus a decoded image corresponding to the sample feature index can be generated. Then, it can be determined whether the decoded image is a real image, and the determination process can be to calculate the error between the decoded image and the input panoramic image training sample, and when the error is higher than a specified threshold, it is considered that the decoded image is not a real image, and if the error is lower than the specified threshold, it is considered that the decoded image is a real image.
[0035] In the above process, in the process of converting sample features into quantized sample feature indexes, since the calculation of similarity is involved, there is a certain error between the sample features and the sample feature indexes, and the error can be represented by a feature quantization error function. In addition, there will also be an error between the decoded image and the panoramic image training sample, and the error can be represented by a network adversarial loss function. In this way, based on the judgment result of whether the decoded image is a real image, an error function can be generated, and the error function can be the above-mentioned feature quantization error function and the network adversarial loss function, so that the encoder, encoding directory and decoder are corrected by the error function. After training with a large number of training samples, the encoder, encoding directory and decoder can restore the corresponding panoramic image after processing the input panoramic image.
[0036] S15: Based on the trained encoder and encoding directory, a standard feature index of the panoramic image training sample is generated.
[0037] S17: Using the text features of the text description sample as a priori conditions, driving the prediction module in the panoramic image generation model to generate a prediction feature index, and training the prediction module in the panoramic image generation model by comparing the prediction feature index with the standard feature index.
[0038] In this embodiment, after the training of the encoder, the encoding directory and the decoder is completed, the prediction module in the panoramic image generation model can be trained. The text features of the text description samples of the panoramic image training samples can be used as a priori conditions to drive the prediction module in the panoramic image generation model to generate a prediction feature index. Specifically, the prediction module can be a cyclic iterative process. First, the first index prediction value (for example, Figure 1 5), then based on the text features and the first index prediction value, the second index prediction value can be generated, and so on, until all index prediction values are generated ( Figure 1 There are 9 index prediction values in it). In this way, each index prediction value can constitute a prediction feature index. To determine whether the prediction feature index generated by the prediction module is correct, a reference object needs to be provided. The reference object can be generated by an encoder and a coding directory that have completed training. Specifically, after the encoder and the coding directory have completed training, the encoder can extract sample features of the panoramic image training samples again, and then convert the sample features into standard feature indexes based on the coding directory. Since the encoder and the coding directory have completed training, the standard feature index obtained by conversion can be regarded as an accurate reference. By comparing the prediction feature index and the standard feature index, the prediction module in the panoramic image generation model can be trained based on the comparison result. In this process, the prediction module can be trained in an autoregressive manner.
[0039] It should be noted that in the process of training the prediction module, the original M*L (assuming the number of channels is 1) matrix is flattened into a ML*1 vector. Figure 1 In the example, the standard feature index is a 3*3 matrix, while the predicted feature index predicted by the prediction module is a 9*1 vector. In this case, two feature indexes of different dimensions cannot be accurately compared. In view of this, in the training process of the prediction module, the position encoding of the feature can be introduced, and the feature indexes of different dimensions can be aligned through the position encoding. Specifically, for any pixel point in the panoramic image training sample, the spherical coordinates of the pixel point can be used as the position encoding. Among them, the longitude on the sphere can be mapped to the 360-degree angle space of [0, 2π], and the latitude can be mapped to the 180-degree angle space of [0, π]. In this way, the spherical coordinates can be represented by the angle in the mapped angle space. The left and right boundaries of the panoramic image training sample were originally adjacent on the sphere, so the position encoding represented by the spherical coordinates is also relatively similar, satisfying the constraint condition that the left and right boundaries of the image are consistent.
[0040] After completing the training phase of the panoramic image generation model, the panoramic image generation model can be used to process the text description information to generate the corresponding panoramic image.
[0041] See also Figure 3 An embodiment of the present disclosure provides a method for generating a panoramic image, which may include the following steps.
[0042] S21: Acquire text description information, and extract text features of the text description information.
[0043] In this embodiment, the text description information may be a paragraph of text input by the user, and the text description information may describe the picture represented by the panoramic image. After the text description information is obtained, the CLIP (Contrastive Language-Image Pre-training, pre-training based on contrastive text-image pairs) feature of the text description information may be extracted, and the extracted CLIP feature may be used as the text feature of the text description information. For the extraction process of the CLIP feature, please refer to the description in the relevant technology in the art, which will not be repeated here.
[0044] S23: Generate an initial feature index, and expand the initial feature index into a feature index to be processed that meets the panoramic image feature.
[0045] In this embodiment, each value in the initial feature index may be randomly generated, for example, Figure 4 The dimension of the initial feature index can be flexibly set according to actual needs, but in general, the aspect ratio of the initial feature index is kept at 2:1 to meet the characteristics of the expanded panoramic image.
[0046] In this embodiment, after the initial feature index is randomly generated, in order to meet the constraint condition of left and right boundary continuity, the boundary of the initial feature index may be expanded to generate a feature index to be processed that meets the panoramic image feature.
[0047] Specifically, see Figure 4 , a first boundary index and a second boundary index can be determined in the initial feature index. The first boundary index can be a boundary index composed of 11 and 6 on the left, and the second boundary index can be a boundary index composed of 8 and 5 on the right. Then, the first boundary index can be added as a first extended index to a position adjacent to the second boundary index, and the second boundary index can be added as a second extended index to a position adjacent to the first boundary index to obtain the feature index to be processed. In this way, the first extended index and the second extended index can be used as two new boundary indexes of the feature index to be processed.
[0048] S25: Based on the text feature, generate a predicted feature index corresponding to the feature index to be processed.
[0049] In this embodiment, after the feature index to be processed is generated, the prediction module in the panoramic image generation model can be used to predict the predicted values of each discrete feature in the feature index to be processed, taking the text feature generated in step S21 as a priori condition. In the process of generating the predicted value, not only the text feature is required as a priori condition, but also the values of multiple discrete features near the position to be predicted in the feature index to be processed are required as input. Figure 4 For example, when predicting the predicted value of the discrete feature at the upper left corner of the feature index to be processed, the values 8, 11, 5, and 6 need to be used as the input of the prediction module. In this way, after iterative prediction by the prediction module, the predicted value of each discrete feature in the feature index to be processed can be generated, thereby obtaining a predicted feature index composed of each predicted value.
[0050] S27: Convert the predicted feature index into a converted feature index that conforms to the panoramic image feature, and generate a panoramic image corresponding to the text description information according to the converted feature index.
[0051] In this embodiment, the predicted feature index obtained by the prediction module may not conform to the feature of the coherence of the left and right boundaries of the panoramic image. In view of this, the predicted feature index can be converted into a conversion feature index that conforms to the panoramic image feature in a similar manner to step S23.
[0052] Specifically, the third boundary index and the fourth boundary index can be determined in the prediction feature index. Figure 4 For example, the third boundary index may be a boundary index composed of 24 and 11, and the fourth boundary index may be a boundary index composed of 18 and 6. Then, a first target index adjacent to the third boundary index may be determined in the predicted feature index, and a second target index adjacent to the fourth boundary index may be determined. Figure 4 For example, the first target index may be an index consisting of 10 and 39, and the second target index may be an index consisting of 27 and 16. Then, the fourth boundary index may be replaced by the first target index, and the third boundary index may be replaced by the second target index, thereby obtaining the conversion feature index.
[0053] In this embodiment, after obtaining the conversion feature index, the conversion feature index can be decoded by the decoder in the panoramic image generation model to obtain the generated image corresponding to the conversion feature index. However, due to the previous boundary expansion, the generated image does not meet the requirement of an aspect ratio of 2:1. In view of this, it is necessary to intercept the effective area of the generated image, which can meet the requirement of an aspect ratio of 2:1, and use the image of the effective area as the panoramic image corresponding to the text description information. Specifically, when intercepting the effective area from the generated image, part of the left and right borders of the generated image can be discarded, and the remaining image can be used as the image of the effective area.
[0054] In one embodiment, considering that high dynamic range (HDR) panoramic images are difficult to obtain in practical applications, low dynamic range (LDR) panoramic images are often used as training samples during the training phase, so the panoramic image finally generated is also an LDR panoramic image.
[0055] If the user's final demand is an HDR panoramic image, the generated LDR panoramic image must be optimized to obtain the corresponding HDR panoramic image. Specifically, the LDR panoramic image can be processed by a super-resolution algorithm to improve the resolution of the LDR panoramic image. In practical applications, the super-resolution algorithm can be, for example, a Real-ESRGAN algorithm or other feasible algorithms. After obtaining the LDR panoramic image with improved resolution, it can be converted into an HDR panoramic image based on the LDR panoramic image in a variety of ways. For example, the LDR panoramic image can be processed by a deep reverse tone mapping technique. Specifically, if the dynamic range of the panoramic image corresponding to the generated text description information does not meet the specified conditions, the panoramic image can be exposed to different degrees to generate a texture sequence, and then a panoramic image with a dynamic range that meets the specified conditions can be synthesized based on the texture sequence. In practical applications, the LDR panoramic image can be used as input, and N images with increased exposure can be output through an up-exposure network, and N images with reduced exposure can be output through a down-exposure network. Combined with the input LDR panoramic image, there are a total of 2N+1 LDR images, from which k LDR images can be selected to construct the final HDR panoramic image. For another example, the LDR panoramic image can be converted into an HDR panoramic image through the Single-Image HDRReconstruction algorithm.
[0056] The technical solution provided by one or more embodiments of the present disclosure can generate a corresponding panoramic image based on the text description information input by the user. Specifically, by extracting the text features of the text description information, the text features can be used as a priori conditions for generating a panoramic image. Under the constraints of the prior conditions, for each discrete feature in the feature index to be processed, a predicted value that meets the text features can be predicted, thereby generating a predicted feature index composed of the predicted values. By processing the predicted feature index, a panoramic image that matches the text description information can be generated.
[0057] It can be seen that the technical solution provided by the present disclosure can generate a panoramic image based only on the text description information, which is not only independent of the actual scene, but also does not require the deployment of complex equipment and devices, greatly improving the convenience. At the same time, the user's needs can be expressed through the text description information, and the panoramic image generated based on the text description information is more in line with the user's actual needs.
[0058] See also Figure 5 In one embodiment of the present disclosure, a device for generating a panoramic image is provided, the device comprising:
[0059] The text feature extraction unit 100 is used to obtain text description information and extract text features of the text description information;
[0060] An initialization unit 200 is used to generate an initial feature index and expand the initial feature index into a feature index to be processed that meets the panoramic image feature;
[0061] A prediction unit 300, configured to generate a predicted feature index corresponding to the feature index to be processed based on the text feature;
[0062] The panoramic image generating unit 400 is configured to convert the predicted feature index into a converted feature index that conforms to panoramic image features, and generate a panoramic image corresponding to the text description information according to the converted feature index.
[0063] The specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.
[0064] See also Figure 6 The present disclosure also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned method for generating a panoramic image is implemented.
[0065] Each unit described in the above embodiments may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0066] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0067] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating a panoramic image is implemented.
[0068] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.
[0069] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.
[0070] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0071] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.
[0072] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0073] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
[0074] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for generating a panoramic image, characterized in that: The method comprises: Acquire text description information, and extract text features of the text description information; Generate an initial feature index, and expand the initial feature index into a feature index to be processed that meets the panoramic image feature; Based on the text feature, generating a predicted feature index corresponding to the feature index to be processed; The predicted feature index is converted into a converted feature index that conforms to the panoramic image feature, and a panoramic image corresponding to the text description information is generated according to the converted feature index.
2. The method according to claim 1, characterized in that Expanding the initial feature index to a feature index to be processed that conforms to the panoramic image feature includes: Determining a first boundary index and a second boundary index in the initial feature index; Using the first boundary index as a first extended index, adding it to a position adjacent to the second boundary index, and using the second boundary index as a second extended index, adding it to a position adjacent to the first boundary index, to obtain the feature index to be processed; The first extended index and the second extended index serve as two boundary indexes of the feature index to be processed.
3. The method according to claim 1, characterized in that: Converting the predicted feature index into a converted feature index that conforms to the panoramic image feature includes: For a third boundary index and a fourth boundary index of the predicted feature index, determining a first target index adjacent to the third boundary index in the predicted feature index, and determining a second target index adjacent to the fourth boundary index; The fourth boundary index is replaced by the first target index, and the third boundary index is replaced by the second target index to obtain the conversion feature index.
4. The method according to claim 1 or 3, characterized in that: Generating a panoramic image corresponding to the text description information according to the conversion feature index includes: Decoding the conversion feature index to obtain a generated image corresponding to the conversion feature index; A valid area of the generated image is intercepted, and an image of the valid area is used as a panoramic image corresponding to the text description information.
5. The method according to claim 1, characterized in that After generating the panoramic image corresponding to the text description information, the method further includes: If the dynamic range of the panoramic image corresponding to the text description information does not meet the specified conditions, the panoramic image is exposed to different degrees to generate a texture sequence, and based on the texture sequence, a panoramic image with a dynamic range that meets the specified conditions is synthesized.
6. The method according to claim 1, characterized in that Based on the text feature, generating a predicted feature index corresponding to the feature index to be processed includes: The text feature is used as a priori condition to predict the predicted value of each discrete feature in the feature index to be processed, so as to generate a predicted feature index corresponding to the feature index to be processed.
7. The method according to claim 6, characterized in that The step of predicting the predicted value of each discrete feature in the feature index to be processed is implemented by completing the trained panoramic image generation model, and the panoramic image generation model is trained in the following manner: Acquire a panoramic image training sample, generate a text description sample of the panoramic image training sample, and extract text features of the text description sample; Using the panoramic image training samples to train the encoder, encoding directory and decoder in the panoramic image generation model; Generating a standard feature index of the panoramic image training sample based on the trained encoder and the encoding directory; The text features of the text description samples are used as prior conditions to drive the prediction module in the panoramic image generation model to generate a prediction feature index, and the prediction module in the panoramic image generation model is trained by comparing the prediction feature index with the standard feature index.
8. The method according to claim 7, characterized in that Generating the text description sample of the panoramic image training sample includes: Converting the panoramic image training sample into an image in a cube map format, and determining a semantic image from the image in the cube map format; The text descriptions of the semantic images are extracted, and a combination of the extracted text descriptions is used as a generated text description sample.
9. The method according to claim 7, characterized in that: Using the panoramic image training sample to train the encoder, the encoding directory, and the decoder in the panoramic image generation model includes: Extracting sample features of the panoramic image training sample using an encoder, and generating a sample feature index corresponding to the sample feature based on a coding directory; Decoding the sample feature index using a decoder to generate a decoded image corresponding to the sample feature index; It is determined whether the decoded image is a true image, and an error function is generated based on the determination result, so as to correct the encoder, the encoding directory and the decoder through the error function.
10. The method according to claim 7, characterized in that In the process of training the prediction module in the panoramic image generation model, the method further includes: For any pixel point in the panoramic image training sample, the spherical coordinates of the pixel point are used as the position encoding of the pixel point, wherein the longitude on the sphere is mapped to an angular space of 360 degrees, and the latitude on the sphere is mapped to an angular space of 180 degrees, and the spherical coordinates are represented by the angle in the mapped angular space.
11. A device for generating a panoramic image, characterized in that: The device comprises: A text feature extraction unit, used to obtain text description information and extract text features of the text description information; An initialization unit, used to generate an initial feature index, and expand the initial feature index into a feature index to be processed that conforms to the panoramic image feature; A prediction unit, configured to generate a predicted feature index corresponding to the feature index to be processed based on the text feature; The panoramic image generation unit is used to convert the predicted feature index into a converted feature index that conforms to the panoramic image feature, and generate a panoramic image corresponding to the text description information according to the converted feature index.
12. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.