A method and system for generating tour images based on semantic analysis

Abstract

The present invention discloses a method and system for generating a tour image based on semantic analysis, including: S1: obtaining tourist photos and scenic spot location pictures, and preprocessing the tourist photos to obtain preprocessed tourist photos; S2: constructing a picture feature extraction network to extract features from tourist photos and scenic spot location pictures; S3: analyzing tourist intentions, performing multimodal entity correspondence, and outputting prompt words and sentences; S4: fusing multimodal data to generate a tour image; S5: adjusting the generated tour image based on the personalized choice of tourists. The present invention aims to solve the problems of time-consuming and labor-intensive traditional tour image generation methods, high technical thresholds, and difficulty in subsequent adjustments through multimodal information understanding and generation technology. The present invention aims to integrate tourist photos with scenic spot poster styles, generate personalized pictures, and automatically adjust according to the personalized requirements of tourists to enhance the tourists' scenic spot tour experience.

Description

Technical Field

[0001] The present invention relates to the technical field of tourism services, and in particular to a method and system for generating a tour image based on semantic analysis. Background Art

[0002] Image generation technology is a technology that creates or modifies images from scratch through artificial intelligence algorithms and models; tour image generation is based on image generation technology, which is a technology that integrates tourist photos with the style of scenic spot posters and generates personalized pictures. Traditional tour image generation methods usually require tourists' photos and scenic spot posters to be imported into image editing software, and then edited and adjusted by professional photo editors. This process is not only time-consuming and labor-intensive, but also has a high technical threshold; in addition, although some software has preset filter functions that can adjust photos to a specific style with one click, this method lacks flexibility and creativity, and the generated pictures are often stereotyped, which makes it difficult to truly reflect the personalized needs of tourists. At the same time, when tourists adjust their position, style, light, etc., they cannot interact with the system directly through natural language, and can only rely on cumbersome manual operations, which has a poor experience. Summary of the invention

[0003] In view of this, the present invention aims to provide a method and system for generating a tour image based on semantic analysis, aiming to solve the problems of cumbersome procedures and difficult subsequent adjustments in traditional tour image generation methods.

[0004] A method for generating a tour image based on semantic analysis comprises the following steps:

[0005] S1: Obtain tourist photos and scenic spot location pictures, and pre-process the tourist photos to obtain pre-processed tourist photos;

[0006] S2: Build an image feature extraction network to extract features from tourist photos and scenic spot location pictures;

[0007] S3: Analyze tourists’ intentions, perform multimodal entity correspondence, and output prompt words and sentences;

[0008] S4: Fuse multimodal data to generate tour images;

[0009] S5: Based on tourists’ personalized choices, the generated tour image is adjusted.

[0010] Furthermore, the step S1 acquires tourist photos and scenic spot location pictures, and pre-processes the tourist photos to obtain pre-processed tourist photos, including:

[0011] S11: using a histogram matching method, adjusting the color and brightness of the tourist photo to match the color tone and light conditions of the scenic spot location picture; the scenic spot location picture is selected from a pre-made scenic spot scenery photo library; the tourist photo is a photo containing the tourist's portrait taken at any time and place by the tourist's mobile phone;

[0012] The scenic spot photo library includes scenic photos of different locations, angles, light conditions and seasonal changes in the scenic spot; for example, mountain tops, lakes, waterfalls, characteristic vegetation, etc., so as to provide sufficient material selection for subsequent tourist photo matching and synthesis; each scenic photo is accompanied by metadata annotations, including shooting location, shooting time, shooting season, and light intensity;

[0013] S12: Identify the location of the tourist and automatically crop the photo to ensure that the tourist is in the center of the image. The calculation method is:

[0014] target1 = YOLOv7(I visitor )

[0015] Among them, target1 is the result of the YOLOv7 network output, including the target box position and target box confidence, YOLOv7() is the YOLOv7 network operation, I visitor The tourist photos after histogram matching;

[0016]

[0017] target2=Cut(target1)

[0018] Among them, x center is the horizontal coordinate of the center of the cropped target box, y center is the ordinate of the center of the cropped target frame, x is the abscissa of the upper left corner of the target frame, y is the ordinate of the upper left corner of the target frame, w is the width of the target frame, h is the height of the target frame, H2 is the width of the tourist photo, H1 is the height of the tourist photo; target2 is the cropped tourist photo, and Cut() is the cropping operation;

[0019] Traditional image cropping methods rely on fixed cropping frames or require users to manually select cropping areas, which is inefficient. After the present invention identifies the subject of the photo, it ensures that the tourist is in the center of the image through cropping operations. This automatic cropping method based on target detection is highly flexible and can handle various complex tourist photos. For the subsequent image understanding model, when the model receives an input picture with a consistent subject position, it can focus more on learning the characteristics of the subject itself without allocating resources to deal with the interference caused by the change of the subject position. This ensures that when the tour image is generated, the tourist and the scenic spot location picture are better integrated, generating a more natural and realistic tour image.

[0020] S13: Remove the background of the tourist photos and keep only the person part. The calculation method is:

[0021] target3=UNet(target2)

[0022] Among them, target3 is the tourist photo with the background removed, and UNet() is the image segmentation network operation.

[0023] Furthermore, the step S2 constructs an image feature extraction network to extract features from scenic spot location images and pre-processed tourist photos, including:

[0024] S21: Construct a feature map weight correction mechanism, the calculation method is:

[0025]

[0026] s c =tanh(W2·ReLU(W1·z c ))

[0027]

[0028] Among them, z c is the channel compression vector, i is the vertical index, j is the horizontal index, c is the channel index, and f is the channel index. i,j,c is the value of the cth channel at position (i, j) in the original feature map, s c is the feature map weight, tanh() is the inverse tangent function, W1 and W2 are weight matrices, ReLU() is the linear rectification function, is the weight-corrected feature map, F c is the original feature map;

[0029] Step S21 compresses the spatial information of each channel into a real number through global average pooling, uses two fully connected layers and ReLU and tanh activation functions to learn the nonlinear relationship between channels, and outputs the weight of each channel feature map. Finally, the weight obtained in the excitation step is applied to the original feature map, and multiplication is performed to obtain the final output feature map. In this way, the model can dynamically adjust the importance of feature channels, thereby improving the representation ability of features.

[0030] In traditional ResNet, the importance of each feature channel is the same, but different feature channels have different importance for tasks. The method of dynamically adjusting the importance of feature channels enables the model to pay more attention to those feature channels that are more critical to the task and ignore unimportant features, thereby improving the extraction effect of portrait features and scenic spot location image features.

[0031] S22: The feature map in the residual convolution network is weighted to obtain the attention residual convolution network, and the attention residual convolution network is used to extract the tourist portrait features. The calculation method is:

[0032] F visitor =AttentionResNet(I visitor )

[0033] Among them, F visitor is the tourist portrait feature map, AttentionResNet() is the attention residual convolutional network operation;

[0034] S23: Use the attention residual convolutional network to extract the characteristics of the scenic spot location image. The calculation method is:

[0035] F location =AttentionResNet(I location )

[0036] Among them, F location is the feature map of the scenic spot location image, I location For scenic spot location pictures;

[0037] S24: Design a training method for region discarding and train the attention residual convolutional network. The calculation method is:

[0038]

[0039] Among them, RegionDrop(F) is the feature map after region discarding, M is the region discarding binary mask, pro is the region discarding probability, and ⊙ is the element-by-element multiplication.

[0040] Furthermore, the step S3 analyzes the tourist's intention, performs multimodal entity correspondence, and outputs a prompt word sentence, including:

[0041] S31: Collect tourists’ language descriptions of generated images and build a semantic understanding model to analyze keywords in the language, identify language entities and analyze tourists’ intentions. The calculation method is:

[0042] Entities1,Intention=BART(Input1,Input1)

[0043] Among them, Entities1 is the language entity, Intention is the tourist intention, BART() is the bidirectional autoregressive language model operation, and Input1 is the tourist’s language description of the generated image;

[0044] Step S31 copies the tourist's language description of the generated image to construct a double input for the bidirectional autoregressive language model; when the model receives repeated language descriptions, it can obtain more contextual information, especially when the text is short or the contextual information is not rich enough, this repeated information can help the model understand and analyze the text more deeply; in addition, the bidirectional autoregressive language model is a model based on the Transformer architecture, which uses an attention mechanism to focus on different parts of the input. When the input is copied, the model will focus on the learned part again, and the effect of the attention mechanism will be amplified, which will help the model to more accurately identify and understand the key information in the text;

[0045] S32: Build an image understanding model and correspond language entities to image entities. The calculation method is:

[0046] head=sigmoid(W3·Concat(Input1,target3,I location )+b head )

[0047] Entities2=CLIP(Input1,target3,I location ,head)

[0048] Among them, head is the image understanding identification head, sigmoid() is the sigmodi function, Concat() is the splicing operation, W3 is the identification head weight matrix, b head is the identification head bias vector, Entities2 is the image entity, and CLIP() is the multimodal representation model operation;

[0049] Traditional correspondence between images and language entities often relies on simple text matching or rule-based mapping, which is often limited in effect when processing complex scenes or fuzzy descriptions, and is prone to correspondence errors. The present invention constructs an image understanding model and calculates an image understanding identifier to guide the correspondence process between images and language entities; and through multimodal representation model operations, simultaneously processes image and text data, and maps the two to the same semantic space, thereby achieving semantic correspondence between images and language entities. On this basis, the model can deeply understand the entities in the image and their relationships, which is helpful for the subsequent generation of more accurate prompt word sentences.

[0050] S33: Combine language entities, image entities, and tourist intentions to output prompt words and sentences. The calculation method is:

[0051] v k =BiLSTM(Input2)

[0052] V = sigmoid(W v ·v k +b v )

[0053] Prompt=ViLBERT(Entities1,Entities2,Intention,V)

[0054] Among them, v k is the sentence vector of the prompt word, BiLSTM() is a bidirectional recurrent neural network operation, Input2 is the historical prompt word sentence data, V is the prompt word sentence chapter structure vector, W v is the chapter structure weight matrix, b v is the text structure bias vector, Prompt is the generated prompt word sentence, and ViLBERT() is the multimodal joint representation model operation;

[0055] Most traditional prompt word sentence generation methods directly generate prompt word sentences based on input information. However, prompt word sentences are semi-structured information. Directly using them as input will waste their structural information. The present invention introduces BiLSTM to calculate sentence vectors and further calculates paragraph vectors to extract semi-structured information and integrate it into the generation process of prompt word sentences. This can better capture the structural and semantic relationships of the input information, thereby generating more structured and coherent prompt word sentences.

[0056] Furthermore, the step S4 fuses the multimodal data to generate a tour image, including:

[0057] S41: Perform feature fusion on the tourist portrait features and the scenic spot location picture features to obtain fusion features. The calculation method is:

[0058] F fusion =Concat((F visitor +F location ),(F visitor ⊙F location ),F visitor ,F location )

[0059] Among them, F fusion is the fusion feature map;

[0060] S42: Generate a tour image by combining the prompt word sentence, the pre-processed tourist photos, the scenic spot location pictures, the language entity and the image entity, and the fusion features. The calculation method is:

[0061] Picture=CLIP(Prompt,target3,I location ,Entities1,Entities2,F fusion )

[0062] Among them, Picture is the generated tour image;

[0063] S43: Adjust the light and shadow of the generated tour image by the following calculation method:

[0064] Picture′=UNet′(Picture)

[0065] Among them, Picture′ is the tour image after adjusting the light and shadow.

[0066] Furthermore, the step S5 adjusts the generated tour image based on the personalized selection of the tourist, including:

[0067] S51: Analyze tourists’ personalized choices through semantic understanding model. The calculation method is:

[0068] Intention′=BART(Input3,Input3)

[0069] Among them, Intention′ is the personalized selection intention of the tourist, and Input3 is the text description of the personalized selection of the tourist; the personalized selection includes: style adjustment, portrait adjustment, and position adjustment; the specific adjustment method is determined according to the natural language description method of the tourist, for example: the tourist selects style adjustment and provides a natural language description: "The whole picture is brighter, and I want the portrait to be sharper";

[0070] S52: Through the image understanding model, the personalized choices of tourists are matched with image entities. The calculation method is:

[0071] d=CLIP(Intention′,Picture′)

[0072] Among them, d is the image-text relationship vector;

[0073] S53: Generate a new version of the tour image by combining the personalized selection of the tourist, the generated tour image, and the correspondence between the personalized selection and the image entity. The calculation method is:

[0074] Picture″=CLIP(Input3,Picture′,d)

[0075] Among them, "Picture" is the new version of the tour image;

[0076] For the personalized choices proposed by tourists, traditional image adjustment methods rely on fixed editing tools and templates, require manual intervention, and have high technical barriers. The present invention realizes automatic analysis of tourists' personalized choices and correspondence with image entities by introducing semantic understanding models and image understanding models. This method can more accurately understand the intentions of tourists and regenerate the generated tour images in a targeted manner according to the intentions. This process does not require professional technicians to perform long-term image adjustments through professional software, providing tourists with a better experience.

[0077] Compared with the prior art, the advantages of the present invention are:

[0078] (1) In terms of the overall solution, the present invention combines tourist photos and scenic spot location pictures to identify language entities, image entities and analyze tourist intentions, and fuses multimodal data to achieve intelligent generation of images combined with text information, and reduces manual intervention in the subsequent image adjustment process, greatly improving the efficiency of business processing.

[0079] (2) In terms of algorithm improvement, the present invention introduces the paragraph structure vector and integrates it into the generation process of the prompt word sentence, so that the system can better utilize the unstructured characteristics of the prompt word sentence; the present invention constructs a dual input for the semantic understanding model, amplifies the effect of the attention mechanism, and helps the model to understand the text more deeply, especially when the input text is short, the effect is more obvious;

[0080] (3) In image generation, the present invention constructs an image understanding identification head to guide the correspondence process between images and language entities, achieving accurate modeling of the image-text relationship, helping the system to more accurately understand the intentions and needs of tourists and generate tour images that meet the requirements.

[0081] (4) In data preprocessing, the present invention proposes an automatic cropping method. After the subject of the photo is identified, the cropping operation is used to ensure that the tourist is in the center of the image; this enables the subsequent image understanding model to receive input images with a consistent subject position and focus more on learning the characteristics of the subject itself without allocating resources to deal with the interference caused by changes in the subject position. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A schematic diagram of the flow of the method for generating a tour image based on semantic analysis provided by the present invention.

[0083] Figure 2 A schematic diagram of the data preprocessing process provided by the present invention.

[0084] Figure 3 A schematic diagram of the algorithm flow of the image feature extraction network provided by the present invention.

[0085] Figure 4 A schematic diagram of the algorithm flow for generating prompt word sentences provided by the present invention.

[0086] Figure 5 A schematic diagram of the process of modifying the tour image provided by the present invention. DETAILED DESCRIPTION

[0087] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0088] Embodiment 1: A method for generating a tour image based on semantic analysis, such as Figure 1 As shown, the following steps are included:

[0089] S1: Obtain tourist photos and scenic spot location pictures, and pre-process the tourist photos to obtain pre-processed tourist photos;

[0090] S11: using a histogram matching method, adjusting the color and brightness of the tourist photo to match the color tone and light conditions of the scenic spot location picture; the scenic spot location picture is selected from a pre-made scenic spot scenery photo library; the tourist photo is a photo containing the tourist's portrait taken at any time and place by the tourist's mobile phone;

[0091] The scenic spot photo library includes scenic photos of different locations, angles, light conditions and seasonal changes in the scenic spot; for example, mountain tops, lakes, waterfalls, characteristic vegetation, etc., so as to provide sufficient material selection for subsequent tourist photo matching and synthesis; each scenic photo is accompanied by metadata annotations, including shooting location, shooting time, shooting season, and light intensity;

[0092] S12: Identify the location of the tourist and automatically crop the photo to ensure that the tourist is in the center of the image. The calculation method is:

[0093] target1 = YOLOv7(I visitor )

[0094] Among them, target1 is the result of the YOLOv7 network output, including the target box position and target box confidence, YOLOv7() is the YOLOv7 network operation, I visitor The tourist photos after histogram matching;

[0095]

[0096] target2=Cut(target1)

[0097] Among them, x center is the horizontal coordinate of the center of the cropped target box, y center is the ordinate of the center of the cropped target frame, x is the abscissa of the upper left corner of the target frame, y is the ordinate of the upper left corner of the target frame, w is the width of the target frame, h is the height of the target frame, H2 is the width of the tourist photo, H1 is the height of the tourist photo; target2 is the cropped tourist photo, and Cut() is the cropping operation;

[0098] Traditional image cropping methods rely on fixed cropping frames or require users to manually select cropping areas, which is inefficient. After the present invention identifies the subject of the photo, it ensures that the tourist is in the center of the image through cropping operations. This automatic cropping method based on target detection is highly flexible and can handle various complex tourist photos. For the subsequent image understanding model, when the model receives an input picture with a consistent subject position, it can focus more on learning the characteristics of the subject itself without allocating resources to deal with the interference caused by the change of the subject position. This ensures that when the tour image is generated, the tourist and the scenic spot location picture are better integrated, generating a more natural and realistic tour image.

[0099] S13: Remove the background of the tourist photos and keep only the person part. The calculation method is:

[0100] target3=UNet(target2)

[0101] Among them, target3 is the tourist photo with the background removed, and UNet() is the image segmentation network operation.

[0102] S2: Build an image feature extraction network to extract features from tourist photos and scenic spot location pictures;

[0103] S21: Construct a feature map weight correction mechanism, the calculation method is:

[0104]

[0105] s c =tanh(W2·ReLU(W1·z c ))

[0106]

[0107] Among them, z c is the channel compression vector, i is the vertical index, j is the horizontal index, c is the channel index, and f is the channel index. i,j,c is the value of the cth channel at position (i, j) in the original feature map, s c is the feature map weight, tanh() is the inverse tangent function, W1 and W2 are weight matrices, ReLU() is the linear rectification function, is the weight-corrected feature map, F c is the original feature map;

[0108] Step S21 compresses the spatial information of each channel into a real number through global average pooling, uses two fully connected layers and ReLU and tanh activation functions to learn the nonlinear relationship between channels, and outputs the weight of each channel feature map. Finally, the weight obtained in the excitation step is applied to the original feature map, and multiplication is performed to obtain the final output feature map. In this way, the model can dynamically adjust the importance of feature channels, thereby improving the representation ability of features.

[0109] In traditional ResNet, the importance of each feature channel is the same, but different feature channels have different importance for tasks. The method of dynamically adjusting the importance of feature channels enables the model to pay more attention to those feature channels that are more critical to the task and ignore unimportant features, thereby improving the extraction effect of portrait features and scenic spot location image features.

[0110] S22: The feature map in the residual convolution network is weighted to obtain the attention residual convolution network, and the attention residual convolution network is used to extract the tourist portrait features. The calculation method is:

[0111] F visitor =AttentionResNet(I visitor )

[0112] Among them, F visitor is the tourist portrait feature map, AttentionResNet() is the attention residual convolutional network operation;

[0113] S23: Use the attention residual convolutional network to extract the characteristics of the scenic spot location image. The calculation method is:

[0114] F location =AttentionResNet(I location )

[0115] Among them, F location is the feature map of the scenic spot location image, I location For scenic spot location pictures;

[0116] S24: Design a training method for region discarding and train the attention residual convolutional network. The calculation method is:

[0117]

[0118] Among them, RegionDrop(F) is the feature map after region drop, M is the region drop binary mask, pro is the region drop probability, and ⊙ is the element-by-element multiplication;

[0119] The specific training method of region discarding is:

[0120] (1) Determine the size of the continuous region to be discarded, for example, Region_size=7;

[0121] (2) Calculate retention probability: Similar to Dropout, set a retention probability, for example, 0.9 means that 10% of the image area will not be updated based on the information;

[0122] (3) Randomly select a starting point on the feature map, and set the cells in the Region_size area centered on the point to 0, and set other locations to 1;

[0123] (4) Apply the generated mask to the feature map and discard the area covered by the mask;

[0124] (5) In order to keep the expected value of the feature map unchanged, the feature map after the mask is applied is divided by pro.

[0125] S3: Analyze tourists’ intentions, perform multimodal entity correspondence, and output prompt words and sentences;

[0126] S31: Collect tourists’ language descriptions of generated images and build a semantic understanding model to analyze keywords in the language, identify language entities and analyze tourists’ intentions. The calculation method is:

[0127] Entities1,Intention=BART(Input1,Input1)

[0128] Among them, Entities1 is the language entity, Intention is the tourist intention, BART() is the bidirectional autoregressive language model operation, and Input1 is the tourist’s language description of the generated image;

[0129] Step S31 copies the tourist's language description of the generated image to construct a double input for the bidirectional autoregressive language model; when the model receives repeated language descriptions, it can obtain more contextual information, especially when the text is short or the contextual information is not rich enough, this repeated information can help the model understand and analyze the text more deeply; in addition, the bidirectional autoregressive language model is a model based on the Transformer architecture, which uses an attention mechanism to focus on different parts of the input. When the input is copied, the model will focus on the learned part again, and the effect of the attention mechanism will be amplified, which will help the model to more accurately identify and understand the key information in the text;

[0130] For example, after visiting a scenic spot, tourists are using the tools provided by the tour image generation system to merge their selfies with scenic spot scenery photos, and have uploaded tourist photos and scenic spot location pictures;

[0131] After uploading their photos, visitors are prompted to provide a brief description of their photo, which they might describe like this:

[0132] “I hope my photo can be naturally integrated into this beautiful sunset background. Standing on the top of the mountain, facing the magnificent sunset, I hope that the mountains and clouds in the background can be clearly seen, while my portrait part can also remain clear and true.”

[0133] The system collects and saves the fusion description provided by tourists and performs word segmentation on it as a guide for subsequent analysis and fusion operations.

[0134] S32: Build an image understanding model and correspond language entities to image entities. The calculation method is:

[0135] hesd=sigmoid(W3·Concat(Input1,target3,I location )+b hesd )

[0136] Entities2=CLIP(Input1,target3,I location ,hesd)

[0137] Among them, hesd is the image understanding identification head, sigmoid() is the sigmoid function, Concat() is the splicing operation, W3 is the identification head weight matrix, b headis the identification head bias vector, Entities2 is the image entity, and CLIP() is the multimodal representation model operation;

[0138] Traditional correspondence between images and language entities often relies on simple text matching or rule-based mapping, which is often limited in effect when processing complex scenes or fuzzy descriptions, and is prone to correspondence errors. The present invention constructs an image understanding model and calculates an image understanding identifier to guide the correspondence process between images and language entities; and through multimodal representation model operations, simultaneously processes image and text data, and maps the two to the same semantic space, thereby achieving semantic correspondence between images and language entities. On this basis, the model can deeply understand the entities in the image and their relationships, which is helpful for the subsequent generation of more accurate prompt word sentences.

[0139] S33: Combine language entities, image entities, and tourist intentions to output prompt words and sentences. The calculation method is:

[0140] v k =BiLSTM(Input2)

[0141] V = sigmoid(W v ·v k +b v )

[0142] Prompt=ViLBERT(Entities1,Entities2,Intention,V)

[0143] Among them, v k is the sentence vector of the prompt word, BiLSTM() is a bidirectional recurrent neural network operation, Input2 is the historical prompt word sentence data, V is the prompt word sentence chapter structure vector, W v is the chapter structure weight matrix, b v is the text structure bias vector, Prompt is the generated prompt word sentence, and ViLBERT() is the multimodal joint representation model operation;

[0144] Most traditional prompt word sentence generation methods directly generate prompt word sentences based on input information. However, prompt word sentences are semi-structured information. Directly using them as input will waste their structural information. The present invention introduces BiLSTM to calculate sentence vectors and further calculates paragraph vectors to extract semi-structured information and integrate it into the generation process of prompt word sentences. This can better capture the structural and semantic relationship of the input information, thereby generating prompt word sentences with more structure and coherence.

[0145] For example:

[0146] The original language description of the tourists is:

[0147] I want to combine myself with this scenic spot poster into one photo, but I want me to stand on the steps in the poster, and I want my portrait to be a little higher so that it looks more spectacular.

[0148] The generated prompt word sentence is:

[0149] You are an experienced landscape photo retouching expert with superb image processing skills and a keen eye for detail. Please make a high-quality composite of me and this scenic spot poster, making sure that I stand in a prominent position on the steps, and the portrait part needs to be appropriately stretched and raised to highlight my main position and enhance the visual effect.

[0150] In terms of the picture style, I hope to maintain the original naturalness and realism of the poster. In terms of light and shadow processing, please fine-tune the light and shadow to ensure that the light and shadow of the portrait and the background are coordinated, creating a sense of three-dimensionality and space, making the overall picture more vivid and natural.

[0151] S4: Fuse multimodal data to generate tour images;

[0152] S41: Perform feature fusion on the tourist portrait features and the scenic spot location picture features to obtain fusion features. The calculation method is:

[0153] F fusion =Concat((F visitor +F location ),(F visitor ⊙F location ),F visitor ,F location )

[0154] Among them, F fusion is the fusion feature map;

[0155] S42: Generate a tour image by combining the prompt word sentence, the pre-processed tourist photos, the scenic spot location pictures, the language entity and the image entity, and the fusion features. The calculation method is:

[0156] Picture=CLIP(Prompt,target3,I location ,Entities1,Entities2,F fusion )

[0157] Among them, Picture is the generated tour image;

[0158] Traditional methods of generating tourist images are based on only a single input data, such as tourist photos or scenic spot location pictures, or simply superimpose multiple input data without effective feature fusion and semantic association, resulting in the lack of realism in the generated tourist images, which cannot accurately reflect the intentions of tourists and the characteristics of scenic spots;

[0159] Step S42 takes multiple different modal data such as prompt words, tourist photos, scenic spot location pictures, language entities and image entities as input, fully utilizes the complementarity and semantic association of various data, extracts key information and features, realizes the comprehensive integration of multimodal data, and helps to generate a tour image that better meets the expectations of tourists;

[0160] S43: Adjust the light and shadow of the generated tour image by the following calculation method:

[0161] Picture′=UNet′(Picture)

[0162] Among them, Picture′ is the tour image after adjusting the light and shadow.

[0163] S5: Adjust the generated tour image based on the tourists’ personalized choices;

[0164] S51: Analyze tourists’ personalized choices through semantic understanding model. The calculation method is:

[0165] Intention′=BART(Input3,Input3)

[0166] Among them, Intention′ is the personalized selection intention of the tourist, and Input3 is the text description of the personalized selection of the tourist; the personalized selection includes: style adjustment, portrait adjustment, and position adjustment; the specific adjustment method is determined according to the natural language description method of the tourist, for example: the tourist selects style adjustment and provides a natural language description: "The whole picture is brighter, and I want the portrait to be sharper";

[0167] S52: Through the image understanding model, the personalized choices of tourists are matched with image entities. The calculation method is:

[0168] d=CLIP(Intention′,Picture′)

[0169] Among them, d is the image-text relationship vector;

[0170] S53: Generate a new version of the tour image by combining the personalized selection of the tourist, the generated tour image, and the correspondence between the personalized selection and the image entity. The calculation method is:

[0171] Picture″=CLIP(Input3,Picture′,d)

[0172] Among them, "Picture" is the new version of the tour image;

[0173] For the personalized choices proposed by tourists, traditional image adjustment methods rely on fixed editing tools and templates, require manual intervention, and have high technical barriers. The present invention realizes automatic analysis of tourists' personalized choices and correspondence with image entities by introducing semantic understanding models and image understanding models. This method can more accurately understand the intentions of tourists and regenerate the generated tour images in a targeted manner according to the intentions. This process does not require professional technicians to perform long-term image adjustments through professional software, providing tourists with a better experience.

[0174] The present invention is particularly suitable for scenic spots with a large number of tourists. At popular tourist attractions, it is difficult for tourists to capture a satisfactory background due to the large flow of people. Through the present invention, tourists can combine their own photos with photos of the scenic spot location to generate a tour image of the scenic spot. In addition, at scenic spots with a large number of tourists, manual photo editing is often time-consuming and labor-intensive, and it is difficult to meet the needs of a large number of tourists. The present invention can quickly generate high-quality composite images through automated and intelligent processing methods, greatly improving processing efficiency and meeting the immediate needs of tourists.

[0175] Embodiment 2: The present invention also discloses a tour image generation system based on semantic analysis, which is used to implement the tour image generation method based on semantic analysis, comprising:

[0176] Data acquisition and preprocessing module: obtain tourist photos, scenic spot location pictures, and preprocess the tourist photos to obtain preprocessed tourist photos;

[0177] Image feature extraction network construction module: Build an image feature extraction network to extract features from tourist photos and scenic spot location pictures;

[0178] Prompt word sentence output module: analyzes tourists' intentions, performs multimodal entity correspondence, and outputs prompt word sentences;

[0179] Tour image generation module: fuses multimodal data to generate tour images;

[0180] Image modification module: adjusts the generated tour images based on tourists' personalized choices.

[0181] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0183] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating a tour image based on semantic analysis, characterized in that: The following steps are involved: S1: Obtain tourist photos and scenic spot location pictures, and pre-process the tourist photos to obtain pre-processed tourist photos; the scenic spot location pictures are selected from a pre-made scenic spot scenery photo library; the tourist photos are photos containing tourist portraits taken at any time and place by tourists' mobile phones; perform histogram matching on tourist photos and scenic spot location pictures, and automatically crop and remove the background of tourist photos in combination with target detection; S2: Build an image feature extraction network to extract features from tourist photos and scenic spot location pictures; including: S21: Construct a feature map weight correction mechanism, the calculation method is: S c =tanh(W2·ReLU(W1·z c )) Among them, z c is the channel compression vector, i is the vertical index, j is the horizontal index, c is the channel index, and f is the channel index. i,j,c is the value of the cth channel at position (i, j) in the original feature map, s c is the feature map weight, tanh() is the inverse tangent function, W1 and W2 are weight matrices, ReLU() is the linear rectification function, is the weight-corrected feature map, F c is the original feature map, H1 is the height of the tourist photo, and H2 is the width of the tourist photo; S22: The feature map in the residual convolution network is weighted to obtain the attention residual convolution network, and the attention residual convolution network is used to extract the tourist portrait features. The calculation method is: F visitor =AttentionResNet(I visitor ) Among them, F visitor is the tourist portrait feature map, AttentionResNet() is the attention residual convolution network operation, I visitor The tourist photos after histogram matching; S23: Use the attention residual convolutional network to extract the characteristics of the scenic spot location image. The calculation method is: F location =AttentionResNet(I location ) Among them, F location is the feature map of the scenic spot location image, I location For scenic spot location pictures; S24: Design a training method for region discarding and train the attention residual convolutional network. The calculation method is: Among them, RegionDrop(F) is the feature map after region drop, M is the region drop binary mask, pro is the region drop probability, and ⊙ is the element-by-element multiplication; S3: Analyze the tourists’ intentions, perform multimodal entity correspondence, and output prompt words and sentences; including: S31: Collect tourists’ language descriptions of generated images and build a semantic understanding model to analyze keywords in the language, identify language entities and analyze tourists’ intentions. The calculation method is: Entities1,Intention=BART(Input1,Input1) Among them, Entities1 is the language entity, Intention is the tourist intention, BART() is the bidirectional autoregressive language model operation, and Input1 is the tourist’s language description of the generated image; S32: Build an image understanding model and correspond language entities to image entities. The calculation method is: head=sigmoid(W3·Concat(Input1,target3,I location )+b head ) Entities2=CLIP(Input1,target3,I location ,head) Among them, head is the image understanding identification head, sigmoid() is the sigmodi function, Concat() is the splicing operation, W3 is the identification head weight matrix, b head is the identification head bias vector, Entities2 is the image entity, CLIP() is the multimodal representation model operation, and target3 is the tourist photo with the background removed; S33: Combine language entities, image entities, and tourist intentions to output prompt words and sentences. The calculation method is: v k =BiLSTM(Input2) V=sigmoid(W v ·v k +b v ) Prompt=ViLBERT(Entities1,Entities2,Intention,V) Among them, v k is the sentence vector of the prompt word, BiLSTM() is a bidirectional recurrent neural network operation, Input2 is the historical prompt word sentence data, V is the prompt word sentence chapter structure vector, W v is the chapter structure weight matrix, b v is the text structure bias vector, Prompt is the generated prompt word sentence, and ViLBERT() is the multimodal joint representation model operation; S4: Fusing multimodal data to generate tour images; including: S41: Perform feature fusion on the tourist portrait features and the scenic spot location picture features to obtain fusion features. The calculation method is: F fusion =Concat((F visitor +F location ),(F visitor ⊙F location ),F visitor ,F location ) Among them, F fusion is the fusion feature map; S42: Generate a tour image by combining the prompt word sentence, the pre-processed tourist photos, the scenic spot location pictures, the language entity and the image entity, and the fusion features. The calculation method is: Picture=CLIP(Prompt,target3,I location ,Entities1,Entities2,F fusion ) Among them, Picture is the generated tour image; S43: Adjust the light and shadow of the generated tour image by the following calculation method: Picture′=UNet′(Picture) Among them, Picture′ is the tour image after adjusting the light and shadow; S5: Adjust the generated tour image based on tourists’ personalized choices.

2. The method for generating a tour image based on semantic analysis according to claim 1, characterized in that: The step S1 comprises: S11: Using a histogram matching method, adjust the color and brightness of the tourist photos to match the tone and light conditions of the scenic spot location picture; S12: Identify the location of the tourist and automatically crop the photo to ensure that the tourist is in the center of the image. The calculation method is: target1=YOLOv7(I visitor ) Among them, target1 is the result of the YOLOv7 network output, including the target box position and target box confidence, and YOLOv7() is the YOLOv7 network operation; target2=Cut(target1) Among them, x center is the horizontal coordinate of the center of the cropped target box, y center is the ordinate of the center of the cropped target frame, x is the abscissa of the upper left corner of the target frame, y is the ordinate of the upper left corner of the target frame, w is the width of the target frame, and h is the height of the target frame; target2 is the cropped tourist photo, and Cut() is the cropping operation; S13: Remove the background of the tourist photos and keep only the person part. The calculation method is: tar get3=UNet(target2) Among them, UNet() is the image segmentation network operation.

3. The method for generating a tour image based on semantic analysis according to claim 1, characterized in that: The step S5 comprises: S51: Analyze tourists’ personalized choices through semantic understanding model. The calculation method is: Intention′=BART(Input3,Input3) Among them, Intention′ is the personalized choice intention of tourists, and Input3 is the text description of the personalized choice of tourists; S52: Through the image understanding model, the personalized choices of tourists are matched with image entities. The calculation method is: d=CLIP(Intention′,Picture′) Among them, d is the image-text relationship vector; S53: Generate a new version of the tour image by combining the personalized selection of the tourist, the generated tour image, and the correspondence between the personalized selection and the image entity. The calculation method is: Picture″=CLIP(Input3,Picture′,d) Among them, "Picture" is the new version of the tour image.

4. A tour image generation system based on semantic analysis, characterized in that: include: Data acquisition and preprocessing module: obtain tourist photos, scenic spot location pictures, and preprocess the tourist photos to obtain preprocessed tourist photos; Image feature extraction network construction module: Build an image feature extraction network to extract features from tourist photos and scenic spot location pictures; Prompt word sentence output module: analyzes tourists' intentions, performs multimodal entity correspondence, and outputs prompt word sentences; Tour image generation module: fuses multimodal data to generate tour images; Image modification module: adjusts the generated tour images based on tourists’ personalized choices; To realize the method for generating a tour image based on semantic analysis as described in any one of claims 1 to 3.

Images