Model training method, address positioning method, electronic device, storage medium and computer program product

Abstract

The application discloses a model training method, an address positioning method, an electronic device, a storage medium and a computer program product, relates to the technical field of large models and image positioning, and the method comprises the following steps: acquiring a training data set, wherein the training data set comprises a training scene image and a visual question and answer data set of corresponding address information of the training scene image; performing cross-view alignment training on an initial multi-modal address positioning model by using the training data set, to generate an intermediate multi-modal address positioning model; and performing address positioning training on the intermediate multi-modal address positioning model by using the training data set, to generate a target multi-modal address positioning model, wherein the target multi-modal address positioning model is used for performing address positioning analysis on a target scene image to be processed and an address question, to obtain an address answer. The application solves the technical problems of the image address positioning scheme provided in the related art, such as coarse positioning granularity, poor positioning accuracy and poor flexibility of question and answer interaction.

Description

Technical Field

[0001] This application relates to the fields of large model technology and image address localization technology, and more specifically, to a model training method, an address localization method, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In related technical fields, constructing multimodal address localization models can achieve the conversion between images, geographic coordinates, and specific addresses, thereby expanding the model's application in macroscopic space. However, while current visual language models can achieve coarse-grained region localization (e.g., countries, cities) based on visual cues in images, they still struggle with fine-grained image address localization, such as street blocks or locations. In other words, current visual language models have relatively coarse granularity and poor accuracy in image address localization, resulting in limited flexibility in the image localization interaction provided by the models. Improving the accuracy of image address localization and enhancing the flexibility of image localization interaction has become one of the important technical challenges in related technical fields.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a model training method, an address positioning method, an electronic device, a storage medium, and a computer program product to at least solve the technical problems of coarse positioning granularity, poor positioning accuracy, and poor question-and-answer interaction flexibility in the image address positioning schemes provided in the related art.

[0005] According to one aspect of the embodiments of this application, a model training method is provided, comprising: acquiring a training dataset, wherein the training dataset includes: a training scene image and a visual question-answering dataset containing address information corresponding to the training scene image; using the training dataset to perform cross-view alignment training on an initial multimodal address localization model to generate an intermediate multimodal address localization model; using the training dataset to perform address localization training on the intermediate multimodal address localization model to generate a target multimodal address localization model, wherein the target multimodal address localization model is used to perform address localization analysis on the target scene image to be processed and the address question to obtain an address answer.

[0006] According to one aspect of the embodiments of this application, an address localization method is provided, comprising: acquiring a target scene image and an address question to be processed; performing address localization analysis on the target scene image and the address question using a target multimodal address localization model to obtain an address answer; wherein the target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset, and the intermediate multimodal address localization model is generated by training an initial multimodal address localization model using a training dataset for cross-view alignment training, and the training dataset includes: a training scene image and a visual question-answering dataset containing address information corresponding to the training scene image.

[0007] According to one aspect of the embodiments of this application, an address localization method is provided, comprising: acquiring a city street scene image and a city street address question; performing address localization analysis on the city street scene image and the city street address question using a target multimodal address localization model to obtain a city street address answer; wherein, the target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset, and the intermediate multimodal address localization model is generated by training an initial multimodal address localization model using a training dataset for cross-view alignment training, and the training dataset includes: a training scene image and a visual question-and-answer dataset containing address information corresponding to the training scene image.

[0008] According to one aspect of the embodiments of this application, an address location method is provided, comprising: obtaining an address location request through a first application programming interface (API), wherein the request data carried in the address location request includes: a target scene image and an address question; and returning an address location response through a second API, wherein the response data carried in the address location response includes: an address answer, wherein the address answer is obtained by performing address location analysis on the target scene image and the address question using a target multimodal address location model, the target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset, the intermediate multimodal address location model is generated by performing cross-view alignment training on an initial multimodal address location model using a training dataset, and the training dataset includes: a training scene image and a visual question-and-answer dataset containing address information corresponding to the training scene image.

[0009] According to one aspect of the embodiments of this application, an address localization method is provided, comprising: acquiring a currently input address localization dialogue request, wherein the request data carried in the address localization dialogue request includes: a target scene image and an address question; responding to the address localization dialogue request and returning an address localization dialogue response, wherein the information carried in the address localization dialogue response includes: an address answer, wherein the address answer is obtained by performing address localization analysis on the target scene image and the address question using a target multimodal address localization model, the target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset, the intermediate multimodal address localization model is generated by performing cross-view alignment training on an initial multimodal address localization model using a training dataset, the training dataset including: a training scene image and a visual question-and-answer dataset of address information corresponding to the training scene image; and playing the address answer within a graphical user interface.

[0010] According to one aspect of the embodiments of this application, an electronic device is provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes either the model training method or the address location method of any of the above-mentioned methods during runtime.

[0011] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to execute the model training method or the address location method of any of the above-mentioned methods.

[0012] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the model training method or the address location method of any of the above-mentioned methods.

[0013] In this embodiment, a training dataset is obtained, comprising: a training scene image and a visual question-and-answer dataset containing corresponding address information of the training scene image; the initial multimodal address localization model is trained for cross-view alignment using the training dataset to generate an intermediate multimodal address localization model; the intermediate multimodal address localization model is then trained for address localization using the training dataset to generate a target multimodal address localization model, wherein the target multimodal address localization model is used to perform address localization analysis on the target scene image and address question to obtain an address answer. Thus, in this embodiment, based on a multimodal training dataset including training scene images and address information, the multimodal address localization model undergoes two stages of training, respectively enhancing the model's cross-view alignment performance and address localization performance, thereby enabling the trained target multimodal address localization model to flexibly generate more accurate address answers. In other words, this application achieves the goal of training a more accurate and flexible question-and-answer target multimodal address localization model, thereby improving the model's image address localization accuracy and enhancing the flexibility of image localization interaction, and solving the technical problems of coarse localization granularity, poor localization accuracy, and poor question-and-answer interaction flexibility in related technologies.

[0014] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of a model training method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a model training method according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of an optional cross-view alignment training phase according to an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of an optional address location training phase according to an embodiment of this application;

[0020] Figure 5 This is a flowchart of an address location method according to an embodiment of this application;

[0021] Figure 6This is a flowchart of another address location method according to an embodiment of this application;

[0022] Figure 7 This is a flowchart of another address location method according to an embodiment of this application;

[0023] Figure 8 This is a flowchart of another address location method according to an embodiment of this application;

[0024] Figure 9 This is a structural block diagram of a model training device according to an embodiment of this application;

[0025] Figure 10 This is a structural block diagram of an address positioning device according to an embodiment of this application;

[0026] Figure 11 This is a structural block diagram of another address positioning device according to an embodiment of this application;

[0027] Figure 12 This is a structural block diagram of another address positioning device according to an embodiment of this application;

[0028] Figure 13 This is a structural block diagram of another address positioning device according to an embodiment of this application;

[0029] Figure 14 This is a structural block diagram of a computing device according to an embodiment of this application;

[0030] Figure 15 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0034] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, image recognition using a multimodal address localization model in an image address localization scenario is used as an example for explanation.

[0035] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows.

[0036] Large Vision-Language Model (LVLM) is an artificial intelligence model that combines images and natural language to understand and generate visual and linguistic information, and to associate and interact between these two modalities. LVLM uses a large language model as its foundation for vision-language related tasks. An LVLM typically includes an image encoder, a large language model, and a vision-language mapper. After pre-training on massive amounts of data and fine-tuning with instructions, LVLM can achieve semantic understanding of image content and generate highly interactive natural language responses.

[0037] Contrastive Language-Image Pre-training (CLIP) models: CLIP refers to a large-scale multimodal learning framework, and CLIP models are models trained using this framework. CLIP models can understand the relationship between visual content and its corresponding textual descriptions. During training, CLIP models use contrastive learning methods to pair visual content (such as images) with corresponding textual descriptions, while simultaneously distinguishing between visual content and mismatched textual descriptions. CLIP models have strong generalization capabilities and can be transferred to different downstream tasks.

[0038] Image address localization (ADL) refers to the process of identifying and determining the specific address (such as street name, house number, etc.) of a scene presented in an image by analyzing the image. The process typically involves extracting geographically significant features (such as road signs, shop signs, specific buildings, etc.) and other clues that may reveal location information from the image. Image address localization can be modeled as an image-based address classification or address alignment problem, and it has high application value in personalized recommendation scenarios in social media and travel, address verification scenarios for internet information, and urban planning and management scenarios.

[0039] Image geo-localization, also known as location identification, is used to determine the specific geographic coordinates of an image. By meticulously analyzing the geographic features within an image and matching them with a geographic information database, the location where the image was taken can be inferred. Image geo-localization has significant applications in news reporting, event location, and tourism information retrieval.

[0040] Supervised fine-tuning refers to the process of further training a pre-trained model using labeled training data.

[0041] According to an embodiment of this application, a model training method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the method provided in this application embodiment can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1 In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0043] In this embodiment, a system comprising a client device and a server can execute the above method. The client device transmits a training dataset to the server, wherein the training dataset includes a visual question-answering dataset containing training scene images and corresponding address information of the training scene images. After acquiring the training dataset, the server uses the training dataset to perform cross-view alignment training on an initial multimodal address localization model to generate an intermediate multimodal address localization model, and uses the training dataset to perform address localization training on the intermediate multimodal address localization model to generate a target multimodal address localization model. The target multimodal address localization model is used to perform address localization analysis on the target scene image and address question to obtain the address answer. Further, the server returns the target multimodal address localization model to the client, or the server provides the client with the calling interface of the target multimodal address localization model.

[0044] It should be noted that with the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-machine systems in other application scenarios. In one optional embodiment, the model-in-machine system has multiple built-in models, and users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-machine system can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the large model-in-machine system has a pre-trained model built-in, and the high-performance computing unit built into the model-in-machine system can then directly call that model to execute the methods provided in this application embodiment.

[0045] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0046] Under the aforementioned operating environment, this application provides the following: Figure 2 The model training method shown. Figure 2 This is a flowchart of a model training method according to an embodiment of this application. Figure 2 As shown, the method may include the following steps S201 to S203.

[0047] Step S201: Obtain the training dataset, which includes: a visual question-answering dataset of training scene images and the address information corresponding to the training scene images.

[0048] The training dataset can be a multimodal dataset. For example, it could include training scene images for the image modality and address information for the text modality. The training scene images could be street view images (such as photographs or satellite images showing street scenes), capturing detailed environmental information such as city streets, buildings, and signs. The address information could be the block name, street name, house number, etc., corresponding to the training scene images.

[0049] The address information mentioned above can be organized in a question-and-answer format. For example, in the address information corresponding to the training scene image P1, the question is "On which street was this photo taken?", and the answer is "This photo was taken on street c1".

[0050] Therefore, the training dataset not only contains scene images, but also provides the corresponding geographical location information for the scene images, which provides important information for subsequent model training.

[0051] Step S202: Use the training dataset to perform cross-view alignment training on the initial multimodal address location model to generate an intermediate multimodal address location model.

[0052] The aforementioned initial multimodal address localization model can be a pre-trained visual language large model, such as a visually enhanced generative pre-trained Transformer-Vision (GPT-V) model, a Large Language and Vision Architecture (LLaVA) model, etc.

[0053] The aforementioned training dataset was used to train the initial multimodal address localization model using cross-view alignment. Specifically, during the training of the initial multimodal address localization model using the cross-view alignment training strategy, the multi-view and multimodal information corresponding to the training dataset was integrated, enabling the model to learn the consistency and complementarity characteristics among the multi-view and multimodal information, thereby obtaining an intermediate multimodal address localization model.

[0054] The aforementioned multi-view, multi-modal information can be provided by data samples contained in the training dataset, or it can be jointly provided by the data samples and other data sources (such as satellite maps, remote sensing datasets, etc.).

[0055] The training strategies for cross-view alignment mentioned above may include, but are not limited to: feature matching training strategy (i.e., finding corresponding feature points between different views and achieving cross-view alignment through feature point matching), transfer learning training strategy (i.e., using the alignment information of existing views to help train the new view alignment model, which can improve the model's generalization ability to new views), and generative adversarial training strategy (i.e., using generative adversarial networks to learn the alignment mapping between views and achieving cross-view alignment through adversarial training between the generator network and the discriminator network), etc.

[0056] As shown above, cross-view alignment training can improve the accuracy of the model in fine-grained address localization, and the generated intermediate multimodal address localization model has preliminary address localization capabilities.

[0057] Step S203: The intermediate multimodal address localization model is trained using the training dataset to generate the target multimodal address localization model. The target multimodal address localization model is used to perform address localization analysis on the target scene image and address problem to obtain the address answer.

[0058] During the process of training the intermediate multimodal address localization model using the training dataset, we focus on improving the model's ability to answer questions related to fine-grained addresses (such as city street and block addresses).

[0059] For example, in address localization training, the model will further learn to answer specific questions such as "Please describe the location in this photo, including the specific block and street name," explaining the basis for address localization, rather than simply identifying the geographical area corresponding to the image.

[0060] As shown above, through address localization training, the target multimodal address localization model can not only locate images but also provide detailed address descriptions, demonstrating stronger address localization capabilities. The target multimodal address localization model possesses the ability to flexibly handle various address-related issues and interact with users. It can provide accurate text address answers for input scene images, significantly improving the practicality of address localization technology and user experience.

[0061] The address localization method provided in this application embodiment can be used to provide a question-and-answer service that locates an address based on an input image in a preset application scenario. The preset application scenarios include, but are not limited to, scenarios involving image address localization in fields such as e-commerce, education, healthcare, conferences, social networks, financial products, logistics, and navigation. The target multimodal image localization model can be a model trained for the preset application scenario, and this target multimodal image localization model can exhibit high performance on specific tasks within the corresponding preset application scenario.

[0062] In this embodiment, a training dataset is obtained, comprising: a training scene image and a visual question-and-answer dataset containing corresponding address information of the training scene image; the initial multimodal address localization model is trained for cross-view alignment using the training dataset to generate an intermediate multimodal address localization model; the intermediate multimodal address localization model is then trained for address localization using the training dataset to generate a target multimodal address localization model, wherein the target multimodal address localization model is used to perform address localization analysis on the target scene image and address question to obtain an address answer. Thus, in this embodiment, based on a multimodal training dataset including training scene images and address information, the multimodal address localization model undergoes two stages of training, respectively enhancing the model's cross-view alignment performance and address localization performance, thereby enabling the trained target multimodal address localization model to flexibly generate more accurate address answers. In other words, this application achieves the goal of training a more accurate and flexible question-and-answer target multimodal address localization model, thereby improving the model's image address localization accuracy and enhancing the flexibility of image localization interaction, and solving the technical problems of coarse localization granularity, poor localization accuracy, and poor question-and-answer interaction flexibility in related technologies.

[0063] The following description, in conjunction with exemplary application scenarios, further illustrates other optional embodiments included in the model training method described in this application.

[0064] In an exemplary application scenario, consider an address hierarchy from highest to lowest, including: country, state, county, city, block, street, and house number. A training dataset is constructed for the block and street levels within the address hierarchy. That is, the training dataset may include block images and their corresponding question-and-answer information, or it may include street images and their corresponding question-and-answer information.

[0065] In the above application scenario, the initial multimodal address location model is trained in two stages using a training dataset. In the first stage, cross-view alignment training is performed on the initial multimodal address location model to generate an intermediate multimodal address location model. In the second stage, address location training is performed on the intermediate multimodal address location model to generate the target multimodal address location model.

[0066] For example, in the above application scenario, the initial multimodal address localization model is an untrained Address Vision-Language Model (Address VLM), and the target multimodal address localization model is a trained Address VLM.

[0067] In an optional embodiment, the training scene images include: a macroscopic view image and a first microscopic view image; the visual question-answering dataset includes at least: a first question text; the initial multimodal address localization model includes: an initial visual encoder, an initial visual language adapter, and an initial language model; in step S202, the initial multimodal address localization model is trained across views using the training dataset to generate an intermediate multimodal address localization model, including the following method steps:

[0068] Step S221: Visually encode the target grafting image using an initial visual encoder to obtain the first visual feature. The target grafting image is obtained by grafting the macroscopic view image and the first microscopic view image.

[0069] Step S222: The first visual feature is mapped to a first visual feature vector representation that matches the first question language embedding vector representation using an initial visual language adapter, wherein the first question language embedding vector representation is generated based on the first question text;

[0070] Step S223: The initial language model is used to perform cross-view alignment training on the language embedding vector representation of the first problem and the first visual feature vector representation to generate an intermediate multimodal address localization model.

[0071] The aforementioned macroscopic view images can be satellite images or aerial images. Macroscopic view images provide a broad view of the target area (such as a city). Macroscopic view images can be used to determine macroscopic information such as street layouts and landmark buildings. The first microscopic view image can be a street view image, used to capture detailed information such as streets, shops, and traffic signs.

[0072] The first question text mentioned above is the question data used in the cross-view training phase of the training dataset. The first question text is used to guide the model to learn how to extract spatial location information from the fusion result of the macroscopic view image and the first microscopic view image (i.e., the target grafted image).

[0073] The aforementioned target grafted image can be an image generated by combining a macroscopic view image (such as a satellite image) with a first microscopic view image (such as a street view image) using image grafting technology.

[0074] The aforementioned initial visual encoder can be a component in a large visual language model used to convert images into feature vectors. This initial visual encoder is used to perform visual encoding processing on the target grafted image to extract first visual features, which may include color, shape, texture, architectural structure, etc. These first visual features are a fusion of macroscopic and microscopic viewpoints.

[0075] The aforementioned initial visual language adapter can be a component in a large visual language model used to transform visual features into feature vector representations that match language embedding vectors. In other words, cross-modal association between images and text is achieved through the initial visual language adapter.

[0076] After generating a first question language embedding vector representation based on the first question text, an initial visual language adapter is used to map the first visual features into a first visual feature vector representation, which matches the first question language embedding vector representation in form. This mapping process, implemented through the initial visual language adapter, helps the model understand how to associate visual cues with address information, thereby more accurately locating locations in the image.

[0077] Furthermore, the initial language model described above can be a pre-trained LLM with natural language understanding and generation capabilities. This initial language model will work in conjunction with a visual encoder and a visual language adapter to enable the model to achieve cross-modal understanding of images and text.

[0078] The aforementioned cross-modal alignment training can be a training strategy that, by jointly using information from macroscopic view images, first microscopic view images, and relevant question text, enables the model to understand the relationships between images from different perspectives and generate accurate address descriptions. For example, based on the cross-modal alignment training strategy, the model will learn how to infer the street and block location of the street view image from the grafted image, and how to explain the basis for making the above inference.

[0079] In the process of cross-view alignment training of the first question language embedding vector representation and the first visual feature vector representation using the initial language model, the first question language embedding vector representation can be used as the text input for the cross-view alignment fine-tuning task, and the first visual feature vector representation can be used as the visual input for the cross-view alignment fine-tuning task. The text input and visual data can be concatenated, compressed, and then input into the initial language model. Through the training strategy of cross-modal alignment training and supervised fine-tuning described above, an intermediate multimodal address localization model is obtained.

[0080] In an exemplary application scenario, the macroscopic view image could be a satellite image, and the first microscopic view image could be a street photograph. The first question text could be, “Based on the visual cues provided in the image, the street view in the upper right appears to have been taken from a location near the intersection of streets c2 and c3. The reflective facade of a large, curved building suggests that this may be a…”

[0081] The information in the text of the first question mentioned above includes:

[0082] The spatial location information suggests that "near the intersection of street c2 and street c3" is a descriptive location on the satellite image, which also implies that the street view image should be associated with this location;

[0083] Visual features provide guidance; for example, the reflective facade of a large, curved building is a visual cue in a street view image. This helps Address VLM identify and match specific features in the image to determine the correlation between the street view image and a location in the satellite image.

[0084] The task requires Address VLM not only to determine the location where the street view image was taken, but also to explain what visual cues led to this conclusion. This helps to train Address VLM to understand and express the reasoning process that connects images and addresses.

[0085] Based on the guidance of the first question text above, the training objectives of Address VLM can include:

[0086] (1) Learning the positional relationships between images: By fusing satellite images and street view images, Address VLM learns the spatial layout of city streets and buildings, as well as the relationship between these layouts and specific addresses;

[0087] (2) Enhance address location capabilities based on visual cues: Address VLM needs to learn to identify and utilize specific visual features in images, such as the shape, color, texture and environmental features of buildings, to determine the location where the image was taken;

[0088] (3) Enhanced explanatory power: Address VLM not only needs to provide the address location result, but also needs to be able to describe which visual cues were used and what reasoning process was carried out, so as to achieve more accurate and detailed address location.

[0089] Based on this, in the exemplary application scenario, through the above cross-view alignment training, Address VLM can better understand and associate visual information in the image with the specific address, providing a stronger foundation for subsequent address localization and question answering tasks (such as address localization fine-tuning in the second stage).

[0090] Through steps S221 to S223 above, this embodiment of the application optimizes the address positioning capability of the large visual language model by integrating image information from both macroscopic and microscopic perspectives.

[0091] In one optional embodiment, the model training method further includes the following method steps:

[0092] Step S204: Based on the preset grafting mechanism, the macroscopic view image and the first microscopic view image are grafted to obtain the target grafted image. The preset grafting mechanism is used to merge the macroscopic view image and the first microscopic view image into a single input image under the premise of determining the master-slave relationship between the macroscopic view image and the first microscopic view image and keeping the aspect ratio of the macroscopic view image and the first microscopic view image unchanged.

[0093] The aforementioned preset grafting mechanism can be an image processing technique used to fuse a macroscopic view image with a first microscopic view image to generate a target grafted image. The preset grafting mechanism considers two key elements: the master-slave relationship between the images and aspect ratio preservation. The purpose of the preset grafting mechanism is to ensure that the macroscopic view image serves as the background and dominant information, while the first microscopic view image serves as the foreground and specific information, maintaining the same aspect ratio between the macroscopic view image and the first microscopic view image during fusion to avoid information distortion.

[0094] In an exemplary application scenario, during the generation of the target grafted image, firstly, a pre-defined grafting mechanism clarifies the master-slave relationship between the macroscopic view image and the first microscopic view image. The macroscopic view image serves as the background and global information, while the first microscopic view image serves as the foreground and specific visual cues. Secondly, the pre-defined grafting mechanism maintains the aspect ratio of the macroscopic view image and the first microscopic view image unchanged. That is, precise image processing technology ensures the proportional coordination between the satellite image and the street view image, avoiding image distortion or discrepancies and guaranteeing the integrity and accuracy of the information. Furthermore, the macroscopic view image and the first microscopic view image are merged into a single input image to obtain the target grafted image.

[0095] The grafting process described above, based on a pre-defined grafting mechanism, ensures both the global perspective of the macro view and highlights the specific details of the micro view, enabling Address VLM to utilize information from both macro and micro perspectives during training, thereby enhancing Address VLM's understanding of urban street distribution.

[0096] Through the above step S204, in the model training of this application embodiment, the grafting process not only integrates image information from macroscopic and microscopic perspectives, but also significantly improves the learning effect and prediction accuracy of the model in urban street-level address positioning by optimizing the image display and processing methods, thus providing a foundation for subsequent training and application.

[0097] In an optional embodiment, in step S204, based on a preset grafting mechanism, the macroscopic view image and the first microscopic view image are grafted to obtain the target grafted image, including the following method steps:

[0098] Step S241: Based on the preset grafting mechanism, the fill position and deletion position in the macroscopic view image are determined by binary mask to obtain the first processing result, and the fill position and deletion position in the first microscopic view image are determined by binary mask to obtain the second processing result.

[0099] Step S242: The target grafting image is obtained by grafting the first processing result and the second processing result.

[0100] The aforementioned binary mask is a mask composed of 0s and 1s. The binary mask is used to determine the regions in the image that need to be deleted (i.e., the regions corresponding to 0s in the mask) and the regions that need to be retained (i.e., the regions corresponding to 1s in the mask). In the above optional embodiments, the binary mask is used to determine the fill and deletion positions in the macroscopic view image, and also to determine the fill and deletion positions in the first microscopic view image.

[0101] The first processing result mentioned above is the processing result of the macroscopic view image. That is, the processing result obtained after applying the binary mask to the macroscopic view image, determining and processing the image pixels at the filling and deletion positions.

[0102] The second processing result mentioned above is the processing result of the first microscopic view image. That is, the processing result obtained after applying the binary mask to the first microscopic view image, determining and processing the image pixels at the filling and deletion positions.

[0103] In an exemplary application scenario, based on a preset grafting mechanism, binary masks are used to determine which parts of the satellite image need to be deleted and which positions need to be filled. For example, a preset area in a satellite image is determined to be the area where a street view image is placed. The information of the preset area on the satellite image is marked as deleted (0), while the information of other areas is marked as retained (1).

[0104] Furthermore, binary masks are used to determine which parts of the street view image need to be filled into specific locations in the satellite image, and which information needs to be deleted to ensure image scale consistency. For example, some areas of the street view image may need to be cropped to fit a preset area in the satellite image. The information of these areas in the street view image will be marked as deleted (0), while the information of other areas will be marked as retained (1).

[0105] Furthermore, based on the first and second processing results, the satellite image and the street image are integrated into a single input image using a grafting technique to obtain the target grafted image. Specifically, the street view image is precisely placed within a preset area of ​​the satellite image, ensuring that the aspect ratio of both the street view and satellite images remains unchanged. In addition, the processing corresponding to the deleted and filled areas can also be performed according to preset rules to achieve seamless fusion between the images.

[0106] Through the above steps S241 to S242, the image grafting processing in this embodiment of the application not only achieves efficient integration of macroscopic and microscopic perspective information, but also improves the learning efficiency and prediction accuracy of the model in urban street-level address localization by maintaining the image ratio and optimizing information display, providing the model with more detailed and comprehensive training materials, and further enhancing the model's performance in multimodal spatial localization tasks.

[0107] In one alternative embodiment, in the model training method, in the target grafting image, based on the long side overlap rate of the image, a first microscopic view image is set in a designated display area of ​​the macroscopic view image.

[0108] The designated display area can be a preset area in the macroscopic view image used to display the first microscopic view image. The image long-side overlap rate is used to characterize the degree of overlap between the long sides of the macroscopic view image and the first microscopic view image during image grafting. By maintaining a certain overlap rate during image grafting, it can be ensured that the street and building information on the macroscopic view image and the first microscopic view image correspond, thereby ensuring that key spatial correlation information is not lost when fusing the macroscopic view image and the first microscopic view image.

[0109] In an exemplary application scenario, when generating the target grafted image, firstly, the specific location of the street image on the satellite image (e.g., the upper right corner) is determined based on the long side overlap rate of the image. Subsequently, the street image is placed in the display area corresponding to the aforementioned specific location on the satellite image, while the size of the street image is adjusted or cropped to ensure that the adjusted street image maintains the same aspect ratio as the satellite image.

[0110] By employing the aforementioned target image grafting technique, this application not only optimizes the fusion method of macroscopic and microscopic perspective images, but also ensures that the model can more accurately understand spatial information and features in urban street-level address localization tasks by maintaining the long side overlap rate of the images, thus providing a foundation for subsequent training and model application.

[0111] In an optional embodiment, the visual question-answering dataset further includes: a first answer text matched with the first question text. In step S223, an initial language model is used to perform cross-view alignment training on the first question language embedding vector representation and the first visual feature vector representation to generate an intermediate multimodal address localization model, including the following method steps:

[0112] Step S2231: The initial language model is used to perform cross-view alignment label generation processing on the first question language embedding vector representation, the first answer language embedding vector representation corresponding to the first answer text, and the first visual feature vector representation to obtain fine-tuning labels. The fine-tuning labels are used to explain the matching reasons between the macroscopic view image and the first microscopic view image and to predict the matching address of the first microscopic view image.

[0113] Step S2232: Based on the fine-tuning labels, perform cross-view alignment fine-tuning on the model parameters of the initial visual encoder, the model parameters of the initial visual language adapter, and the model parameters of the initial language model to generate an intermediate multimodal address localization model.

[0114] The first answer text mentioned above can be the correct answer corresponding to the first question text. For example, if the first question text is a question related to an address, such as "In which block was this photo taken?", then the first answer text is the corresponding precise description of the address, such as "This photo was taken on street c3 in area B of city A".

[0115] The fine-tuned labels obtained through the cross-view alignment label generation process described above can not only contain address information and explain in detail the reasons for the matching between the macroscopic view image and the first microscopic view image, but also explain in detail how the model predicts the matching address of the first microscopic view image based on visual cues. In other words, the cross-view alignment label generation process can provide the model with rich learning material.

[0116] The matching reasons described above can be used to determine which visual cues in the macroscopic view image and the first microscopic view image support the judgment that the macroscopic view image matches the first microscopic view image. The matching address described above can be the specific address of the first microscopic view image match predicted based on the description of the visual cues.

[0117] In an exemplary application scenario, the initial language model in Address VLM performs cross-view alignment label generation processing on the language embedding vector representation of the first question, the language embedding vector representation of the first answer text, and the first visual feature vector representation to obtain fine-tuned labels. In this process, the initial language model generates detailed labels based on visual features and features to explain the association between the image and the address, and can also determine which visual cues in the image support the judgment of address location.

[0118] For example, in training for area B in city A, Address VLM might generate fine-tuned labels based on visual features such as building appearance, signs, and street layout in street images, as well as macroscopic information of the corresponding locations in satellite images, such as: "Based on the street signs and surrounding building styles of street c4 in the street image, and the location and direction of travel of street c4 in the satellite image, it can be determined that this street image was taken at the intersection of street c4 and street c5."

[0119] Furthermore, based on the fine-tuning labels, the model parameters of the initial visual encoder, initial visual language adapter, and initial language model are fine-tuned across view alignment. This cross-view alignment fine-tuning process is conducted in a supervised manner. The purpose of the fine-tuning is to optimize the Address VLM's integration performance of information from both macroscopic and microscopic perspectives, enabling it to more accurately understand and locate address information in images. For example, during training, Address VLM learns how to match specific visual cues in street view images with location information in satellite images, and makes address location judgments based on these cues, while providing the rationale for these judgments.

[0120] In one specific implementation, such as Figure 3 The diagram shown illustrates the cross-view alignment training phase, as follows: Figure 3 As shown, based on the fused image of the macroscopic view image and the first microscopic view image, and the first question text corresponding to the fused image, combined with the first answer text corresponding to the first question text, cross-view alignment training is performed on the initial visual encoder, initial visual language adapter, and initial language model in the initial multimodal address localization model. Figure 3 The initial visual encoder, initial visual language adapter, and initial language model all contain learnable parameters, which are fine-tuned during the cross-view alignment training. After cross-view alignment training, the initial visual encoder becomes the intermediate visual encoder, the initial visual language adapter becomes the intermediate visual language adapter, and the initial language model becomes the intermediate language model, thus obtaining the intermediate multimodal address localization model.

[0121] Therefore, by fine-tuning the generation of labels and optimizing model parameters, Address VLM can not only identify address information in images, but also understand the spatial relationship between addresses and image content, thus improving the accuracy of Address VLM in fine-grained address localization.

[0122] Furthermore, the use of generative models and the introduction of fine-tuned labels enable Address VLM to generate detailed address location explanations, enhancing its interactive question-and-answer capabilities. Users can ask Address VLM how it makes location judgments, and Address VLM can provide explicit answers based on image features. Through cross-view alignment fine-tuning, not only are the visual features of the image considered, but also the textual information of the address question and answer are combined. This allows Address VLM to understand the image content while mastering the linguistic description of the address information, thus providing more accurate and detailed answers to various address-related questions.

[0123] Through the above steps S2231 to S2232, the embodiments of this application not only significantly improve the fine-grained address localization capability of the visual language model through cross-view alignment fine-tuning, providing a technical foundation for building a large target multimodal address localization model that is easy to interact with, but also provide richer and more detailed address localization solutions by enhancing the flexibility and interactivity of the model, thus promoting the application and development of multimodal spatial localization technology in scenarios.

[0124] In one optional embodiment, the training scene image includes: a second microscopic viewpoint image; the visual question-answering dataset includes at least: a second question text; the intermediate multimodal address localization model includes: an intermediate visual encoder, an intermediate visual language adapter, and an intermediate language model; in step S203, the intermediate multimodal address localization model is trained for address localization using the training dataset to generate the target multimodal address localization model, including the following method steps:

[0125] Step S231: Visually encode the second microscopic view image using an intermediate visual encoder to obtain the second visual features. The intermediate visual encoder is obtained by fine-tuning the model parameters of the initial visual encoder through cross-view alignment.

[0126] Step S232: The second visual features are mapped to the second visual feature vector representation that matches the second question language embedding vector representation using an intermediate visual language adapter. The second question language embedding vector representation is generated based on the second question text. The intermediate visual language adapter is obtained by fine-tuning the model parameters of the initial visual language adapter through cross-view alignment.

[0127] Step S233: The intermediate language model is used to train the address localization of the second problem language embedding vector representation and the second visual feature vector representation to generate a target multimodal address localization model. The intermediate language model is obtained by fine-tuning the model parameters of the initial language model through cross-view alignment.

[0128] Similar to the first microscopic view image, the second microscopic view image can also be a street view image, used to capture detailed information such as streets, shops, and traffic signs. However, unlike the first microscopic view image, in the second stage of training, the model will focus more on fine-tuning the address localization based on the second microscopic view image, which no longer needs to be grafted onto the macroscopic view image.

[0129] The second question text mentioned above can be a specific question related to address location, such as "What street is this?" or "Please describe the name of the current block." This second question text serves as input for the second stage of model training, helping the model learn how to generate address information from images.

[0130] In an exemplary application scenario, the Address VLM (i.e., intermediate multimodal address localization model) obtained in the first stage of training includes an intermediate visual encoder, an intermediate visual language adapter, and an intermediate language model. This intermediate multimodal address localization model serves as the starting point for the second stage of model training (i.e., address localization training).

[0131] In the second stage of model training, the intermediate visual encoder performs visual encoding on the second microscopic view image (i.e., the street image) to extract visual features (i.e., second visual features). These second visual features are the foundation for the Address VLM to understand the image content. The intermediate visual encoder is obtained by fine-tuning the initial visual encoder through cross-view alignment. When processing street scene images, it already has a certain macroscopic view understanding capability and can better locate address information in the image.

[0132] The intermediate visual language adapter described above maps the second visual features into a form that matches the second question language embedding vector representation, resulting in a second visual feature vector representation. This mapping process can be based on the correlation between the second question language embedding vector representation generated from the second question text and the visual features, ensuring that the Address VLM can understand the relationship between the image and the question. The intermediate visual language adapter is obtained by fine-tuning the initial visual language adapter through cross-view alignment, and it also enables greater accuracy when handling address localization tasks.

[0133] Furthermore, the second question language embedding vector representation and the second visual feature vector representation are input into the intermediate language model for address localization training, generating the final target multimodal address localization model. During address localization training, the intermediate language model learns how to comprehensively process visual and textual information to generate a question-related address localization answer. The intermediate language model is obtained by fine-tuning the initial language model through cross-view alignment and exhibits strong performance on the address localization task.

[0134] The aforementioned target multimodal address localization model includes: a target visual encoder, a target visual language adapter, and a target language model. The target visual encoder is obtained by training an intermediate visual encoder for address localization, the target visual language adapter is obtained by training an intermediate visual language adapter for address localization, and the target language model is obtained by training an intermediate language model for address localization.

[0135] It should be noted that the intermediate multimodal address localization model, after cross-view alignment fine-tuning, can more accurately identify and locate fine-grained address information when processing second microscopic view images. Further address localization fine-tuning of the intermediate multimodal address localization model allows the final target multimodal address localization model to better understand the relationship between images and text, improving the model's overall performance and generalization ability. The target multimodal address localization model not only provides accurate address localization but also generates associated address information and explanations based on the input second question text, enhancing the model's interactivity and flexibility, enabling it to handle diverse address-related questions and provide detailed answers.

[0136] Through steps S231 to S233 above, this embodiment of the application not only enhances the model's fine-grained address location capability through address location training, but also provides support for building an easy-to-interact address location model by improving the model's performance in understanding and generating address information.

[0137] In an optional embodiment, the visual question-answering dataset further includes: a second answer text matched with the second question text. In step S233, an intermediate language model is used to train the address localization of the second question language embedding vector representation and the second visual feature vector representation to generate a target multimodal address localization model, including the following method steps:

[0138] Step S2331: The intermediate language model is used to perform address localization prediction processing on the second question language embedding vector representation and the second visual feature vector representation to obtain the predicted answer text, wherein the predicted answer text is used to predict the matching address of the second microscopic view image.

[0139] Step S2332: Based on the predicted response text and the second response text, fine-tune the model parameters of the intermediate visual encoder, the intermediate visual language adapter, and the intermediate language model to generate a target multimodal address localization model.

[0140] The second response text mentioned above refers to the correct answer text that matches the second question text. For example, if the second question text is "What is the house number of my current location?", the corresponding second response text would be "You are currently located at house number n on street c1 in area B of city A."

[0141] The predicted answer text mentioned above can be the address prediction text generated by the intermediate language model based on the input second question text and the second visual feature vector representation. In the second stage of model training, the predicted answer text is compared with the second answer text to fine-tune the model parameters.

[0142] The target multimodal address localization model can be generated by fine-tuning the address localization of an intermediate multimodal address localization model. The target multimodal address localization model has more granular address localization capabilities (such as at the block and street level within a city).

[0143] In an exemplary application scenario, in the Address VLM (i.e., intermediate multimodal address localization model) obtained from the first stage of training, the intermediate language model receives the second question language embedding vector representation generated from the second question text and the second visual feature vector representation obtained through processing by the intermediate visual encoder and intermediate visual language adapter, and performs address localization prediction processing to generate predicted answer text, that is, to predict the address of the scene in the second microscopic view image (i.e., street image).

[0144] Furthermore, the predicted answer text is compared with the second answer text (the correct answer) in the visual question-answering dataset to obtain the prediction discrepancy (e.g., training loss). This discrepancy can be used to evaluate the accuracy of the model's predictions. Based on the prediction discrepancy, the model parameters of the intermediate visual encoder, intermediate visual language adapter, and intermediate language model are fine-tuned for address localization to optimize the performance of Address VLM in address localization at the city block and street levels. For example, if the address predicted by Address VLM differs significantly from the actual address, the fine-tuning process will adjust the model parameters to reduce the prediction discrepancy in future predictions.

[0145] In one specific implementation, such as Figure 4 The diagram shown illustrates the address localization training phase. Figure 4 As shown, based on the second microscopic viewpoint image, the second question text corresponding to the second microscopic viewpoint image, and the second answer text corresponding to the second question text, address localization training is performed on the intermediate visual encoder, intermediate visual language adapter, and intermediate language model in the intermediate multimodal address localization model. Figure 4 The intermediate visual encoder, intermediate visual language adapter, and intermediate language model all contain learnable parameters, which are fine-tuned during the address localization training. After address localization training, the intermediate visual encoder becomes the target visual encoder, the intermediate visual language adapter becomes the target visual language adapter, and the intermediate language model becomes the target language model, thus obtaining the target multimodal address localization model. It should be noted that... Figure 4The intermediate visual encoder, intermediate visual language adapter, and intermediate language model in the process can be Figure 3 The cross-view alignment training process shown is obtained.

[0146] Through steps S2331 to S2332, this embodiment of the application generates predicted response text and compares it with the actual second response text. This allows the model to learn how to more accurately identify and locate address information at the city block and street level, thereby improving the prediction accuracy of address location. Based on the error feedback from address location prediction, the parameters of the intermediate visual encoder, intermediate visual language adapter, and intermediate language model in the model are fine-tuned to help the model better understand and associate visual and textual information, improving the performance of the Address VLM in multimodal spatial localization tasks. The fine-tuned target multimodal address localization model can not only accurately predict addresses but also generate detailed answers related to the second question text (such as street names, house numbers, descriptions of landmarks, etc.), significantly enhancing the flexibility and interactivity of the target multimodal address localization model.

[0147] In one alternative embodiment, in the model training method, both cross-view alignment fine-tuning and address positioning fine-tuning are performed using a low-rank adaptive fine-tuning method.

[0148] The aforementioned Low-Rank Adaptation (LoRA) fine-tuning refers to a technique used to fine-tune parameters in machine learning models, especially large-scale pre-trained models. Specifically, LoRA introduces low-rank matrices into weight updates to achieve model adjustment. These low-rank matrices can capture task-related features, enabling the model to learn task-specific knowledge without having to completely update the parameters of the entire model.

[0149] In the embodiments of this application, both the cross-view alignment fine-tuning and the address location fine-tuning stages of model training adopt the LoRA fine-tuning method to fine-tune the model parameters.

[0150] Specifically, in the cross-view alignment fine-tuning stage, LoRA fine-tuning is used to fine-tune some parameters of the initial visual encoder, initial visual language adapter, and initial language model to enhance the model's understanding of the relationship between images and addresses. Through LoRA fine-tuning, the model adapts to the tasks of fusion of satellite and street images and generation of alignment fine-tuning labels by updating a small number of parameters. This helps the model quickly learn and integrate information from both macro and micro perspectives, improving the model's fine-grained address localization capabilities.

[0151] Specifically, in the address localization fine-tuning stage, LoRA fine-tuning is used to fine-tune some parameters of the intermediate visual encoder, intermediate visual language adapter, and intermediate language model to enhance the model's fine-grained address prediction and question answering performance.

[0152] Specifically, the visual encoder, visual language adapter, and language model in the above model can include a LoRA layer. During cross-view alignment fine-tuning and address localization fine-tuning, the parameters in the LoRA layer can be updated while keeping other model parameters unchanged. This ensures that the model's performance on address localization tasks is enhanced while maintaining its generalization ability on other tasks.

[0153] It should be noted that, during the cross-view alignment fine-tuning and address location fine-tuning processes using the LoRA fine-tuning method, only the low-rank matrix of the model is updated, significantly reducing the computational resources and time required for fine-tuning. In other words, even on large-scale visual language models, the technical solution provided in this application can perform fine-tuning at a lower cost, improving the efficiency and operability of model training.

[0154] It should be noted that, through LoRA fine-tuning, the technical solution provided in this application embodiment can significantly enhance the model's fine-grained address localization capability while retaining the model's general capabilities. The LoRA-tuned model can not only accurately predict addresses, but also generate detailed answers based on input address-related questions, explaining the basis for address localization. This significantly enhances the interactivity between the model and the user, making the technical solution provided in this application embodiment more flexible and practical in real-world applications.

[0155] In the exemplary application scenario, performing cross-view alignment fine-tuning and address location fine-tuning through LoRA fine-tuning can effectively improve the multimodal spatial positioning capability of Address VLM, especially in fine-grained address positioning at the city street level, achieving a balance between performance and resource utilization.

[0156] In one optional embodiment, the question-answering types of the visual question-answering dataset in the model training method include at least one of the following: address generation question-answering type, address judgment question-answering type, and address selection question-answering type.

[0157] In the exemplary application scenario, the question-answering information in the above training dataset mainly focuses on question-answering types such as address generation, address judgment, and address selection.

[0158] The address generation question-and-answer type mentioned above refers to a question-and-answer type that requires the model to generate specific address information based on a given image. For example, the question corresponding to the address generation question-and-answer type is "At what specific address was this photo taken?".

[0159] The address-based question-and-answer type mentioned above refers to a question-and-answer type involving the model judging the authenticity of address information in an image. For example, the question corresponding to the address-based question-and-answer type is "Was this photo taken on street c5?", and the model needs to judge the correctness of this statement in the question based on the image content.

[0160] The address selection question-and-answer type mentioned above refers to a question-and-answer type that requires the model to select the address that best matches the location shown in the image from a set of candidate addresses. For example, the question for the address selection question-and-answer type is "Select the most likely shooting location from the following options: A. Street c6; B. Next to building x; C. Next to lake L." The model needs to analyze clues in the image and make a selection from the options.

[0161] By employing a training dataset containing multiple question-answering types in the model training method, the technical solution of this application embodiment can effectively improve the model's fine-grained spatial localization capability while enhancing the flexibility and breadth of the model's interaction with users.

[0162] It should be noted that, in the embodiments of this application, the visual encoder of the target multimodal address localization model can be based on the CLIP model, or it can be based on different architectures such as deep convolutional neural networks based on residual network architecture, visual Transformers, etc. Furthermore, the input image resolution of the visual encoder can be adjusted to achieve a balance between performance and resources, enhancing the generalization performance of the target multimodal address localization model and its adaptability to different application scenarios.

[0163] It should be noted that, in the embodiments of this application, the language model of the target multimodal address localization model can adopt any implementable large language model. The type of large language model can be flexibly selected according to the needs of the scenario to improve the language understanding and generation capabilities of the target multimodal address localization model, as well as enhance the multimodal fusion effect of the target multimodal address localization model.

[0164] It should be noted that in the embodiments of this application, the generation of alignment tags is not limited to a specific visual language large model. Different types of visual language large models can also be flexibly selected according to the needs of the scenario to reduce the implementation threshold of the automatic generation scheme of alignment tags. On the other hand, alignment tags can also be generated by selecting a model that is suitable for the project needs. According to the characteristics and requirements of specific tasks, the fine-tuning strategy can be customized to further improve the accuracy of address positioning and the generalization ability of the model.

[0165] It should be noted that in this embodiment of the application, the internationally recognized administrative address hierarchy can be used as a reference when identifying addresses, or it can be adjusted according to the characteristics of the administrative divisions of different countries, thereby adjusting the regional applicability of the technical solution provided in this embodiment of the application, enhancing user experience and the acceptability of the solution.

[0166] As can be seen from the above, the technical solutions provided in this application also include other alternative solutions in multiple aspects such as visual encoder, large language model, alignment label generation and text address representation. By implementing these alternative solutions, not only can the fine-grained address positioning performance of the model be improved, but it can also be flexibly adjusted according to different computing resources, costs and target application scenarios to achieve a balance between high performance and practicality.

[0167] Compared with related technologies, the technical solution provided in this application breaks through the limitations of traditional large-scale visual language models in fine-grained address localization. Through cross-view alignment fine-tuning technology, it combines macroscopic and microscopic perspectives, injecting a more fine-grained spatial understanding capability into the large-scale visual language model. The technical solution provided in this application abandons the rigid address classification method based on discriminative models and adopts a generative model for image address localization. That is, the above technical solution can not only accurately predict address information but also generate detailed localization explanations based on visual cues, enabling address question-and-answer interaction with users in natural language, greatly enhancing the interactivity and flexibility of address localization. Furthermore, the technical solution provided in this application, by introducing a dynamic grafting mechanism between macroscopic and microscopic view images and an automatic alignment fine-tuning label generation mechanism, significantly reduces the reliance on manual annotation during model training. This not only makes the solution more adaptable to large-scale visual language models but also significantly reduces model training overhead, improving overall implementation efficiency and economy.

[0168] In summary, the technical solutions provided in this application not only achieve a breakthrough in the address positioning capability of large visual language models, but also provide a more efficient and practical address positioning solution from the perspectives of user interaction experience and resource cost.

[0169] In the aforementioned operating environment, this application also provides, as follows: Figure 5 This illustrates an address location method. Figure 5 This is a flowchart of an address location method according to an embodiment of this application, such as... Figure 5 As shown, the address location method includes:

[0170] Step S51: Obtain the target scene image and address information to be processed;

[0171] Step S52: Use the target multimodal address localization model to perform address localization analysis on the target scene image and address problem to obtain the address answer;

[0172] The target multimodal address localization model is generated by training the intermediate multimodal address localization model with the training dataset. The intermediate multimodal address localization model is generated by training the initial multimodal address localization model with the training dataset for cross-view alignment. The training dataset includes: training scene images and a visual question-and-answer dataset containing the address information corresponding to the training scene images.

[0173] The aforementioned address localization method can be implemented based on the target multimodal address localization model obtained through the aforementioned model training method. After acquiring the target scene image and address problem to be processed, the target multimodal address localization model is used to perform address localization analysis on the target scene image and address problem to obtain the address answer.

[0174] In this embodiment, a target scene image and address question are obtained; a target multimodal address localization model is used to analyze the address localization of the target scene image and address question to obtain an address answer; wherein, the target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset, and the intermediate multimodal address localization model is generated by training an initial multimodal address localization model using a training dataset for cross-view alignment. The training dataset includes a visual question-and-answer dataset of training scene images and corresponding address information of the training scene images. In this embodiment, the target multimodal address localization model is obtained through two stages of training based on a multimodal training dataset including training scene images and address information, and has strong cross-view alignment and address localization performance. The target multimodal address localization model can flexibly generate more accurate address answers. That is, this application achieves the goal of providing more flexible and accurate address answers through the target multimodal address localization model, thereby improving the flexibility and accuracy of image address localization question answering, and solving the technical problems of coarse localization granularity, poor localization accuracy, and poor question-and-answer interaction flexibility of image address localization schemes provided in related technologies.

[0175] It should be noted that the preferred embodiments of steps S51 to S52 described above can be found in the foregoing description, and will not be repeated here.

[0176] In the aforementioned operating environment, this application also provides, as follows: Figure 6 Another address location method is shown. Figure 6 This is a flowchart of another address location method according to an embodiment of this application, such as... Figure 6 As shown, the address location method includes:

[0177] Step S61: Obtain the city street scene image and the city street address problem;

[0178] Step S62: Use the target multimodal address localization model to perform address localization analysis on the urban street scene image and the urban street address problem to obtain the answer to the urban street address;

[0179] The target multimodal address localization model is generated by training the intermediate multimodal address localization model with the training dataset. The intermediate multimodal address localization model is generated by training the initial multimodal address localization model with the training dataset for cross-view alignment. The training dataset includes: training scene images and a visual question-and-answer dataset containing the address information corresponding to the training scene images.

[0180] The address localization method described in this application embodiment can be used to provide flexible and accurate address question-and-answer services in urban street image address localization scenarios. The training dataset used to train the target multimodal address localization model can also be pre-constructed for urban street image address localization scenarios, enabling the target multimodal address localization model to exhibit higher performance in such scenarios.

[0181] In this embodiment, a city street scene image and a city street address question are acquired. A target multimodal address localization model is used to perform address localization analysis on the city street scene image and the city street address question to obtain the city street address answer. The target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset. The intermediate multimodal address localization model is generated by training an initial multimodal address localization model using a training dataset for cross-view alignment. The training dataset includes a visual question-and-answer dataset containing training scene images and corresponding address information. In this embodiment, the target multimodal address localization model is obtained through two stages of training based on a multimodal training dataset including training scene images and address information. It possesses strong cross-view alignment and address localization performance, and can flexibly generate more accurate address answers. In other words, this application achieves the goal of providing address answers more flexibly and accurately through the target multimodal address positioning model, thereby realizing the technical effect of improving the flexibility and accuracy of image address positioning question answering, and thus solving the technical problems of coarse positioning granularity, poor positioning accuracy and poor question answering interaction of image address positioning schemes provided in related technologies.

[0182] It should be noted that the preferred embodiments of steps S61 to S62 described above can be found in the foregoing description, and will not be repeated here.

[0183] In the aforementioned operating environment, this application also provides, as follows: Figure 7 This is another address location method shown. Figure 7 This is a flowchart of another address location method according to an embodiment of this application, such as... Figure 7As shown, the address location method includes:

[0184] Step S71: Obtain an address location request through the first application programming interface, wherein the request data carried in the address location request includes: target scene image and address problem;

[0185] Step S72: Return an address location response through the second application programming interface. The response data carried in the address location response includes: an address answer, which is obtained by performing address location analysis on the target scene image and the address question using the target multimodal address location model. The target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset. The intermediate multimodal address location model is generated by training an initial multimodal address location model using a training dataset for cross-view alignment. The training dataset includes: a training scene image and a visual question-and-answer dataset containing the address information corresponding to the training scene image.

[0186] The address location method described in this application embodiment can run on a cloud server to provide address location cloud services to clients. The client sends an address location request by calling a first application programming interface (API). After obtaining the address location request through the first API, the cloud server generates an address response using a target multimodal address location model and further returns the address location response to the client through a second API.

[0187] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0188] In this embodiment, an address location request is obtained through a first application programming interface (API), wherein the request data carried in the address location request includes: a target scene image and an address question; an address location response is returned through a second API, wherein the response data carried in the address location response includes: an address answer, which is obtained by performing address location analysis on the target scene image and the address question using a target multimodal address location model. The target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset, and the intermediate multimodal address location model is generated by training an initial multimodal address location model using a training dataset for cross-view alignment. The training dataset includes: a training scene image and a visual question-and-answer dataset containing the address information corresponding to the training scene image. In this embodiment, the target multimodal address location model is obtained through two stages of training based on a multimodal training dataset including the training scene image and address information. It possesses strong cross-view alignment and address location performance, and can flexibly generate more accurate address answers. In other words, this application achieves the goal of providing address answers more flexibly and accurately through the target multimodal address positioning model, thereby realizing the technical effect of improving the flexibility and accuracy of image address positioning question answering, and thus solving the technical problems of coarse positioning granularity, poor positioning accuracy and poor question answering interaction of image address positioning schemes provided in related technologies.

[0189] It should be noted that the preferred embodiments of steps S71 to S72 described above can be found in the foregoing description, and will not be repeated here.

[0190] In the aforementioned operating environment, this application also provides, as follows: Figure 8 This is another address location method shown. Figure 8 This is a flowchart of another address location method according to an embodiment of this application, such as... Figure 8 As shown, the address location method includes:

[0191] Step S81: Obtain the currently input address location dialogue request, wherein the request data carried in the address location dialogue request includes: target scene image and address question;

[0192] Step S82, in response to the address location dialogue request, return an address location dialogue response, wherein the information carried in the address location dialogue response includes: address answer, which is obtained by performing address location analysis on the target scene image and address question using the target multimodal address location model; the target multimodal address location model is generated by training the intermediate multimodal address location model using the training dataset; the intermediate multimodal address location model is generated by training the initial multimodal address location model using the training dataset for cross-view alignment; the training dataset includes: training scene image and visual question answering dataset of address information corresponding to the training scene image;

[0193] Step S83: Play the address answer within the graphical user interface.

[0194] The address location method provided in this application can utilize a target multimodal address location model to achieve a visual address location scheme, facilitating human-computer interaction. The user inputs an address location request, triggering the system to generate an address response using the target multimodal address location model, and then plays the response within the graphical user interface.

[0195] Based on the above method steps, a visualization scheme for address location function is provided. The terminal device provides a graphical user interface (GUI), which displays at least one image address location scene. The GUI display content also includes input components (such as text input boxes, voice input controls, etc.) and display components (such as text display windows, image display windows, etc.). The user inputs an address location dialog request through the input components to specify the target scene image and address question in the address location task. After detecting the user's input, the address location process is executed based on the target scene image and address question to obtain the address answer. Furthermore, the address answer is played through the playback component within the GUI.

[0196] In this embodiment, the currently input address location dialogue request is obtained. The request data carried in the address location dialogue request includes: a target scene image and an address question. In response to the address location dialogue request, an address location dialogue reply is returned. The address location dialogue reply carries information including: an address answer, which is obtained by performing address location analysis on the target scene image and the address question using a target multimodal address location model. The target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset. The intermediate multimodal address location model is generated by training an initial multimodal address location model using a training dataset for cross-view alignment. The training dataset includes: a visual question-and-answer dataset of training scene images and corresponding address information of the training scene images. The address answer is played within the graphical user interface. In this embodiment, the target multimodal address location model is obtained through two stages of training based on a multimodal training dataset including training scene images and address information. It has strong cross-view alignment performance and address location performance, and can flexibly generate more accurate address answers. In other words, this application achieves the goal of providing address answers more flexibly and accurately through the target multimodal address positioning model, thereby realizing the technical effect of improving the flexibility and accuracy of image address positioning question answering, and thus solving the technical problems of coarse positioning granularity, poor positioning accuracy and poor question answering interaction of image address positioning schemes provided in related technologies.

[0197] It should be noted that the preferred embodiments of steps S81 to S82 described above can be found in the foregoing description, and will not be repeated here.

[0198] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0199] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0200] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0201] According to embodiments of this application, an apparatus embodiment for implementing the above-described model training method or address location method is also provided.

[0202] Figure 9 This is a structural block diagram of a model training device according to an embodiment of this application, such as... Figure 9 As shown, the device includes:

[0203] The acquisition module 901 is used to acquire the training dataset, which includes: training scene images and a visual question answering dataset containing the address information corresponding to the training scene images;

[0204] The first training module 902 is used to train the initial multimodal address location model across views using the training dataset to generate an intermediate multimodal address location model.

[0205] The second training module 903 is used to train the intermediate multimodal address localization model using the training dataset to generate the target multimodal address localization model. The target multimodal address localization model is used to perform address localization analysis on the target scene image and address problem to obtain the address answer.

[0206] Optionally, the training scene images include: a macroscopic view image and a first microscopic view image; the visual question answering dataset includes at least: a first question text; the initial multimodal address localization model includes: an initial visual encoder, an initial visual language adapter, and an initial language model; the aforementioned first training module 902 is further configured to: use the initial visual encoder to visually encode the target grafting image to obtain a first visual feature, wherein the target grafting image is obtained by grafting the macroscopic view image and the first microscopic view image; use the initial visual language adapter to map the first visual feature to a first visual feature vector representation that matches the first question language embedding vector representation, wherein the first question language embedding vector representation is generated based on the first question text; and use the initial language model to perform cross-view alignment training on the first question language embedding vector representation and the first visual feature vector representation to generate an intermediate multimodal address localization model.

[0207] Optionally, in addition to the above-mentioned multiple modules, the model training device also includes a grafting module (not shown in the figure), which is used to graft the macroscopic view image and the first microscopic view image based on a preset grafting mechanism to obtain a target grafted image. The preset grafting mechanism is used to merge the macroscopic view image and the first microscopic view image into a single input image while determining the master-slave relationship between the macroscopic view image and the first microscopic view image and keeping the aspect ratio of the macroscopic view image and the first microscopic view image unchanged.

[0208] Optionally, the grafting module is further configured to: determine the fill position and deletion position in the macroscopic view image based on a preset grafting mechanism using a binary mask to obtain a first processing result; and determine the fill position and deletion position in the first microscopic view image using a binary mask to obtain a second processing result; and graft the first processing result and the second processing result to obtain a target grafted image.

[0209] Optionally, in the above-mentioned model training device, in the target grafting image, based on the overlap rate of the long side of the image, the first microscopic view image is set in a designated display area of ​​the macroscopic view image.

[0210] Optionally, the visual question answering dataset also includes: the first answer text matched with the first question text. The first training module 902 is further used to: perform cross-view alignment label generation processing on the first question language embedding vector representation, the first answer text corresponding to the first answer language embedding vector representation, and the first visual feature vector representation using the initial language model to obtain fine-tuned labels. The fine-tuned labels are used to explain the matching reason between the macroscopic view image and the first microscopic view image and to predict the matching address of the first microscopic view image. Based on the fine-tuned labels, the model parameters of the initial visual encoder, the model parameters of the initial visual language adapter, and the model parameters of the initial language model are fine-tuned across views to generate an intermediate multimodal address localization model.

[0211] Optionally, the training scene image includes: a second microscopic viewpoint image; the visual question answering dataset includes at least: a second question text; the intermediate multimodal address localization model includes: an intermediate visual encoder, an intermediate visual language adapter, and an intermediate language model; the second training module 903 is further configured to: visually encode the second microscopic viewpoint image using the intermediate visual encoder to obtain second visual features, wherein the intermediate visual encoder is obtained by fine-tuning the model parameters of the initial visual encoder through cross-view alignment; map the second visual features to a second visual feature vector representation matching the second question language embedding vector representation using the intermediate visual language adapter, wherein the second question language embedding vector representation is generated based on the second question text, and the intermediate visual language adapter is obtained by fine-tuning the model parameters of the initial visual language adapter through cross-view alignment; train the second question language embedding vector representation and the second visual feature vector representation for address localization using the intermediate language model to generate a target multimodal address localization model, wherein the intermediate language model is obtained by fine-tuning the model parameters of the initial language model through cross-view alignment.

[0212] Optionally, the visual question answering dataset also includes: a second answer text matched with the second question text. The second training module 903 is further configured to: perform address localization prediction processing on the second question language embedding vector representation and the second visual feature vector representation using an intermediate language model to obtain a predicted answer text, wherein the predicted answer text is used to predict the matching address of the second microscopic viewpoint image; and fine-tune the address localization of the model parameters of the intermediate visual encoder, the model parameters of the intermediate visual language adapter, and the model parameters of the intermediate language model based on the predicted answer text and the second answer text to generate a target multimodal address localization model.

[0213] Optionally, in the above model training device, both cross-view alignment fine-tuning and address positioning fine-tuning are performed in a low-rank adaptive fine-tuning manner.

[0214] Optionally, in the above-mentioned model training device, the question-answering types of the visual question-answering dataset include at least one of the following: address generation question-answering type, address judgment question-answering type, and address selection question-answering type.

[0215] It should be noted that the aforementioned determining module 901, first training module 902, and second training module 903 correspond to steps S201 to S203 in the above embodiments. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0216] Figure 10 This is a structural block diagram of an address positioning device according to an embodiment of this application, such as... Figure 10 As shown, the device includes:

[0217] Module 1001 is used to acquire the target scene image and address information to be processed.

[0218] Analysis module 1002 is used to perform address location analysis on target scene images and address problems using a target multimodal address location model to obtain address answers;

[0219] The target multimodal address localization model is generated by training the intermediate multimodal address localization model with the training dataset. The intermediate multimodal address localization model is generated by training the initial multimodal address localization model with the training dataset for cross-view alignment. The training dataset includes: training scene images and a visual question-and-answer dataset containing the address information corresponding to the training scene images.

[0220] It should be noted that the acquisition module 1001 and the analysis module 1002 mentioned above correspond to steps S51 to S52 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0221] Figure 11 This is a structural block diagram of another address positioning device according to an embodiment of this application, such as... Figure 11 As shown, the device includes:

[0222] Module 1101 is used to acquire images of urban street scenes and urban street addresses.

[0223] Analysis module 1102 is used to perform address location analysis on urban street scene images and urban street address problems using a target multimodal address localization model, and obtain the answer to the urban street address;

[0224] The target multimodal address localization model is generated by training the intermediate multimodal address localization model with the training dataset. The intermediate multimodal address localization model is generated by training the initial multimodal address localization model with the training dataset for cross-view alignment. The training dataset includes: training scene images and a visual question-and-answer dataset containing the address information corresponding to the training scene images.

[0225] It should be noted that the acquisition module 1101 and the analysis module 1102 mentioned above correspond to steps S61 to S62 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0226] Figure 12 This is a structural block diagram of another address positioning device according to an embodiment of this application, such as... Figure 12 As shown, the device includes:

[0227] The acquisition module 1201 is used to acquire an address location request through a first application programming interface, wherein the request data carried in the address location request includes: a target scene image and an address problem;

[0228] The response module 1202 is used to return an address location response through the second application programming interface. The response data carried in the address location response includes: an address answer, which is obtained by performing address location analysis on the target scene image and the address question using a target multimodal address location model. The target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset. The intermediate multimodal address location model is generated by training an initial multimodal address location model using a training dataset for cross-view alignment. The training dataset includes: a training scene image and a visual question-and-answer dataset containing the address information corresponding to the training scene image.

[0229] It should be noted that the above-mentioned acquisition module 1201 and response module 1202 correspond to steps S71 to S72 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0230] Figure 13 This is a structural block diagram of another address positioning device according to an embodiment of this application, such as... Figure 13 As shown, the device includes:

[0231] The acquisition module 1301 is used to acquire the currently input address location dialogue request, wherein the request data carried in the address location dialogue request includes: target scene image and address question;

[0232] Return module 1302 is used to respond to the address location dialogue request and return an address location dialogue response. The address location dialogue response carries information including: the address answer, which is obtained by performing address location analysis on the target scene image and the address question using the target multimodal address location model. The target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset. The intermediate multimodal address location model is generated by training an initial multimodal address location model using a training dataset for cross-view alignment. The training dataset includes: a training scene image and a visual question-and-answer dataset containing the address information corresponding to the training scene image.

[0233] Playback module 1303 is used to play address answers within a graphical user interface.

[0234] It should be noted that the above-mentioned acquisition module 1301, return module 1302 and playback module 1303 correspond to steps S81 to S83 in the above embodiments. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0235] It should be noted that the above-mentioned modules or units may be hardware or software components stored in memory and processed by one or more processors. The above-mentioned modules may also be part of the device and run in the server 10 provided in the above embodiments.

[0236] It should be noted that the preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0237] Embodiments of this application may provide a computing device. Figure 14 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 14 As shown, the computing device 140 may include one or more (only one is shown in the figure) processors 142, memory 144, memory controller, and peripheral interfaces.

[0238] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, personal computers (PCs), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0239] Specifically, this computing device can pre-install various types of models, including but not limited to models in fields such as natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing Application Programming Interface (API) invocation capabilities. Models can be invoked into created applications through the API interface, and application management tools are provided for application management and monitoring.

[0240] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master artificial intelligence (AI) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0241] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0242] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0243] Embodiments of this application may provide an electronic device. Figure 15 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 15 As shown, the electronic device may include: an input / output device 152; a memory 154; and a processor 156, wherein the processor 156 is connected to the input / output device 152 and the memory 154 via a bus 158.

[0244] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0245] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0246] It will be understood by those skilled in the art that the structure shown in the figure is merely illustrative, and the computing device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a mobile internet device (MID), etc. This figure does not limit the structure of the aforementioned computing device. For example, the computing device 100 may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or may have a different configuration than that shown in the figure.

[0247] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0248] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0249] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0250] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0251] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0252] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0253] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0254] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0255] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0256] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0257] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0258] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, ROM, RAM, portable hard drives, magnetic disks, or optical disks.

[0259] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Obtain a training dataset, wherein the training dataset includes: a visual question-answering dataset of training scene images and address information corresponding to the training scene images; The initial multimodal address localization model is trained with cross-view alignment using the training dataset to generate an intermediate multimodal address localization model. The intermediate multimodal address localization model is trained using the training dataset to generate a target multimodal address localization model. The target multimodal address localization model is used to perform address localization analysis on the target scene image and address problem to obtain the address answer.

2. The model training method according to claim 1, characterized in that, The training scene images include: a macroscopic view image and a first microscopic view image; the visual question-answering dataset includes at least: a first question text; the initial multimodal address localization model includes: an initial visual encoder, an initial visual language adapter, and an initial language model; the initial multimodal address localization model is trained using the training dataset to perform cross-view alignment training, generating the intermediate multimodal address localization model, which includes: The target grafting image is visually encoded using the initial visual encoder to obtain a first visual feature, wherein the target grafting image is obtained by grafting the macroscopic view image and the first microscopic view image. The first visual feature is mapped to a first visual feature vector representation that matches the first question language embedding vector representation using the initial visual language adapter, wherein the first question language embedding vector representation is generated based on the first question text; The initial language model is used to train the first problem language embedding vector representation and the first visual feature vector representation to perform cross-view alignment training, thereby generating the intermediate multimodal address localization model.

3. The model training method according to claim 2, characterized in that, The model training method also includes: Based on a preset grafting mechanism, the macroscopic view image and the first microscopic view image are grafted to obtain the target grafted image. The preset grafting mechanism is used to merge the macroscopic view image and the first microscopic view image into a single input image while determining the master-slave relationship between the macroscopic view image and the first microscopic view image and keeping the aspect ratio of the macroscopic view image and the first microscopic view image unchanged.

4. The model training method according to claim 3, characterized in that, Based on the preset grafting mechanism, the macroscopic view image and the first microscopic view image are grafted to obtain the target grafted image, including: Based on the preset grafting mechanism, the fill position and deletion position in the macroscopic view image are determined by binary mask to obtain a first processing result, and the fill position and deletion position in the first microscopic view image are determined by binary mask to obtain a second processing result; The target grafting image is obtained by grafting the first processing result and the second processing result together.

5. The model training method according to claim 3, characterized in that, In the target grafting image, based on the overlap rate of the long side of the image, the first microscopic view image is set in a designated display area of ​​the macroscopic view image.

6. The model training method according to claim 2, characterized in that, The visual question-answering dataset further includes: the first answer text matched with the first question text; and the intermediate multimodal address localization model generated by training the first question language embedding vector representation and the first visual feature vector representation across views using the initial language model, including: The initial language model is used to perform cross-view alignment label generation processing on the first question language embedding vector representation, the first answer language embedding vector representation corresponding to the first answer text, and the first visual feature vector representation to obtain fine-tuned labels. The fine-tuned labels are used to explain the matching reason between the macroscopic view image and the first microscopic view image and to predict the matching address of the first microscopic view image. Based on the fine-tuning tags, the model parameters of the initial visual encoder, the model parameters of the initial visual language adapter, and the model parameters of the initial language model are fine-tuned across views to generate the intermediate multimodal address localization model.

7. The model training method according to claim 2, characterized in that, The training scene images include: a second microscopic viewpoint image; the visual question-answering dataset includes at least: a second question text; the intermediate multimodal address localization model includes: an intermediate visual encoder, an intermediate visual language adapter, and an intermediate language model; the intermediate multimodal address localization model is trained for address localization using the training dataset to generate the target multimodal address localization model, which includes: The intermediate visual encoder is used to visually encode the second microscopic view image to obtain the second visual feature; The intermediate visual language adapter is used to map the second visual features to a second visual feature vector representation that matches the second question language embedding vector representation, wherein the second question language embedding vector representation is generated based on the second question text; The intermediate language model is used to train the second problem language embedding vector representation and the second visual feature vector representation for address localization, thereby generating the target multimodal address localization model.

8. The model training method according to claim 7, characterized in that, The visual question-answering dataset further includes: the second answer text matched with the second question text; and the target multimodal address localization model generated by training the address localization model using the intermediate language model on the language embedding vector representation of the second question and the second visual feature vector representation. The intermediate language model is used to perform address localization prediction processing on the second question language embedding vector representation and the second visual feature vector representation to obtain the predicted answer text, wherein the predicted answer text is used to predict the matching address of the second microscopic view image; Based on the predicted response text and the second response text, the model parameters of the intermediate visual encoder, the model parameters of the intermediate visual language adapter, and the model parameters of the intermediate language model are fine-tuned for address localization, thereby generating the target multimodal address localization model.

9. The model training method according to claim 6 or 8, characterized in that, Both the cross-view alignment fine-tuning and the address positioning fine-tuning are performed using a low-rank adaptive fine-tuning method.

10. The model training method according to claim 1, characterized in that, The question-answering types of the visual question-answering dataset include at least one of the following: address generation question-answering type, address judgment question-answering type, and address selection question-answering type.

11. An address location method, characterized in that, include: The problem of obtaining the target scene image and address to be processed; A target multimodal address localization model is used to perform address localization analysis on the target scene image and the address question to obtain the address answer; The target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset. The intermediate multimodal address localization model is generated by training an initial multimodal address localization model using the training dataset for cross-view alignment. The training dataset includes: training scene images and a visual question-and-answer dataset containing the address information corresponding to the training scene images.

12. An address location method, characterized in that, include: The problem of obtaining images of urban street scenes and their addresses; A target multimodal address localization model is used to perform address localization analysis on the urban street scene image and the urban street address problem to obtain the answer to the urban street address; The target multimodal address localization model is generated by training an intermediate multimodal address localization model using a training dataset. The intermediate multimodal address localization model is generated by training an initial multimodal address localization model using the training dataset for cross-view alignment. The training dataset includes: training scene images and a visual question-and-answer dataset containing the address information corresponding to the training scene images.

13. An address location method, characterized in that, include: An address location request is obtained through a first application programming interface, wherein the request data carried in the address location request includes: a target scene image and an address problem; The address location response is returned through the second application programming interface. The response data carried in the address location response includes: an address answer, which is obtained by performing address location analysis on the target scene image and the address question using a target multimodal address location model. The target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset. The intermediate multimodal address location model is generated by performing cross-view alignment training on an initial multimodal address location model using the training dataset. The training dataset includes: a training scene image and a visual question-and-answer dataset containing the address information corresponding to the training scene image.

14. An address location method, characterized in that, include: Obtain the currently input address location dialogue request, wherein the request data carried in the address location dialogue request includes: target scene image and address question; In response to the address location dialogue request, an address location dialogue response is returned, wherein the information carried in the address location dialogue response includes: an address answer, which is obtained by performing address location analysis on the target scene image and the address question using a target multimodal address location model; the target multimodal address location model is generated by training an intermediate multimodal address location model using a training dataset; the intermediate multimodal address location model is generated by performing cross-view alignment training on an initial multimodal address location model using the training dataset; the training dataset includes: a training scene image and a visual question-and-answer dataset containing the address information corresponding to the training scene image. Play the address response within the graphical user interface.

15. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the model training method according to any one of claims 1 to 10 or the address location method according to any one of claims 11 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the model training method according to any one of claims 1 to 10 or the address location method according to any one of claims 11 to 14.

17. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the model training method according to any one of claims 1 to 10 or the address location method according to any one of claims 11 to 14.

Images