Vehicle question and answer method, device, equipment and storage medium

Abstract

This disclosure provides a vehicle question-and-answer method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. In some embodiments of this disclosure, an original vehicle-mounted image is input into a target positioning system to obtain the positioning of all vehicles within the original vehicle-mounted image, resulting in positioning information for at least one vehicle. Based on the positioning information of at least one vehicle, the original vehicle-mounted image is cropped to obtain at least one vehicle-mounted image, enabling accurate cropping. Vehicle information for each vehicle-mounted image is determined based on its similarity to images in a vehicle image library. The vehicle-mounted images are then combined with images from the vehicle image library to improve image recognition rate and, consequently, vehicle information recognition accuracy. Model prompts and the original vehicle-mounted image are input into a visual language model to obtain a response corresponding to the user's original question, further improving the accuracy of vehicle question-and-answer by combining the visual language model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a vehicle question-and-answer method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, in-vehicle vision systems are playing an increasingly important role in scenarios such as assisted driving, human-vehicle interaction, and environmental perception. Users interacting with the vehicle's infotainment system via natural language, asking in real-time about the brand and model of a specific vehicle in the external environment captured by the in-vehicle cameras, has become a crucial requirement for enhancing the driving experience. Such tasks not only demand high-precision image understanding capabilities from the system but also the ability to accurately locate and identify targets in complex scenes by combining contextual semantics.

[0003] Vehicle brand and model identification primarily relies on deep learning-based image classification models or multimodal question-answering systems that combine large language models. The former typically uses convolutional neural networks (CNNs) or visual Transformers to classify the entire image and output the vehicle brand or model in a predefined category; the latter attempts to encode the image into a feature vector and input it into a large model, combining it with natural language prompts to generate an answer.

[0004] Currently, on the one hand, in-vehicle images are often affected by lens distortion, motion blur, changes in lighting, and occlusion, which leads to a decrease in image recognition rate; on the other hand, due to the rapid iteration of new energy vehicle brands and models, supervised learning methods rely on a large amount of labeled data, which makes it difficult to cover newly launched models in a timely manner, resulting in a low accuracy rate of vehicle information recognition, which in turn affects the accuracy rate of vehicle question answering. Summary of the Invention

[0005] This disclosure provides a vehicle question-and-answer method, apparatus, device, and storage medium to at least solve the problems of reduced image recognition rate, low vehicle information recognition accuracy, and low accuracy of vehicle question-and-answer.

[0006] The technical solution disclosed herein is as follows: This disclosure provides a vehicle question-and-answer method, including: The original vehicle image is input into the target positioning system to obtain the positioning of all vehicles in the original vehicle image, and the positioning information of at least one vehicle is obtained, wherein the original vehicle image corresponds to the user's original question. Based on the location information of the at least one vehicle, the original vehicle image is cropped to obtain at least one vehicle image. The vehicle information of each in-vehicle image is determined based on the similarity between each in-vehicle image and an image in the vehicle image library. The location information of at least one vehicle, the vehicle information of each in-vehicle image, and the user's original question are concatenated to obtain model prompt words; The model prompts and the original vehicle image are input into the visual language model to obtain the response result corresponding to the user's original question.

[0007] Optionally, determining the vehicle information of each in-vehicle image based on its similarity to images in the vehicle image library includes: Select candidate images from all image libraries whose similarity meets the set similarity criteria; For any of the vehicle images, calculate the correlation score between each candidate image in the image library and any of the vehicle images. The target image and the target vehicle image with the highest relevance score are selected from each combination of the candidate image library image and any one of the vehicle images. Based on the correlation score between the target image library image and the target vehicle image, the vehicle information of any one of the vehicle images is determined.

[0008] Optionally, the method further includes: Each of the vehicle images is input into the embedding model to obtain a first vector representation corresponding to each vehicle image; The similarity between each of the first vector representations and the second vector representations of all images in the vehicle image library is calculated, and this similarity is used as the similarity between each of the in-vehicle images and the images in the vehicle image library.

[0009] Optionally, calculating the correlation score between each candidate image in the image library and any one of the in-vehicle images includes: The re-ranking model is used to calculate the correlation score between each candidate image in the image library and any one of the in-vehicle images.

[0010] Optionally, determining the vehicle information of any one of the vehicle images based on the correlation score between the target image library image and the target vehicle image includes: If the correlation score between the target image library image and the target vehicle image is greater than or equal to a set score threshold, then the vehicle information of the target image library image is used as the vehicle information of the target vehicle image. If the correlation score between the target image in the image library and the target vehicle image is less than the set score threshold, then the vehicle information of the target vehicle image is set to unknown vehicle information.

[0011] Optionally, before using the embedding model, the method further includes: Construct the vehicle image library; A first training set is constructed based on the images in the vehicle image library; wherein, the first training set includes: a first positive example data and a first negative example data, the first positive example data refers to two images of vehicles with the same vehicle information, and the first negative example data refers to two images of vehicles with different vehicle information. Based on the first training set, the initial embedding model is trained using contrastive learning loss to obtain the trained embedding model.

[0012] Optionally, before using the reordering model, the method further includes: A second training set is constructed based on the images in the vehicle image library; wherein, the second training set includes: second positive example data and second negative example data, the second positive example data refers to two images of vehicles with the same vehicle information, and the second negative example data refers to two images of vehicles with different vehicle information; Based on the second training set, the initial re-ranking model is trained using binary classification loss to obtain the trained re-ranking model.

[0013] This disclosure also provides a vehicle question-and-answer device, including: The positioning module is used to input the original vehicle image into the target positioning system to obtain the positioning of all vehicles in the original vehicle image and obtain the positioning information of at least one vehicle, wherein the original vehicle image corresponds to the user's original question. The cropping module is used to crop the original vehicle image based on the positioning information of the at least one vehicle to obtain at least one vehicle image. The determination module is used to determine the vehicle information of each in-vehicle image based on the similarity between each in-vehicle image and an image in the vehicle image library. The stitching module is used to stitch together the location information of the at least one vehicle, the vehicle information of each of the in-vehicle images, and the user's original question to obtain model prompt words; The response module is used to input the model prompts and the original vehicle image into the visual language model to obtain a response result corresponding to the user's original question.

[0014] This disclosure also provides an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the steps in the above method.

[0015] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In some embodiments of this disclosure, the original vehicle-mounted image is input into a target positioning system to obtain the positioning of all vehicles within the original vehicle-mounted image, resulting in the positioning information of at least one vehicle. The original vehicle-mounted image corresponds to the user's original question. Based on the positioning information of at least one vehicle, the original vehicle-mounted image is cropped to obtain at least one vehicle-mounted image, ensuring accurate cropping. Vehicle information for each vehicle-mounted image is determined based on its similarity to images in a vehicle image library. The vehicle-mounted images are then combined with images from the vehicle image library to improve image recognition rate and, consequently, vehicle information recognition accuracy. The positioning information of at least one vehicle, the vehicle information of each vehicle-mounted image, and the user's original question are concatenated to obtain model prompts. The model prompts and the original vehicle-mounted image are input into a visual language model to obtain a response corresponding to the user's original question. This combination with the visual language model improves the accuracy of vehicle question-and-answer processing.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 A flowchart illustrating a vehicle question-and-answer method provided as an exemplary embodiment of this disclosure; Figure 2 A schematic diagram of a vehicle question-and-answer process provided for an exemplary embodiment of this disclosure; Figure 3 A schematic diagram of the structure of a vehicle question-and-answer device provided for an exemplary embodiment of this disclosure; Figure 4 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0022] It should be noted that the user information involved in this disclosure includes, but is not limited to, user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] To address the aforementioned technical problems, in some embodiments of this disclosure, the original vehicle-mounted image is input into a target positioning system to obtain the positioning of all vehicles within the original vehicle-mounted image, resulting in the positioning information of at least one vehicle. The original vehicle-mounted image corresponds to the user's original question. Based on the positioning information of at least one vehicle, the original vehicle-mounted image is cropped to obtain at least one vehicle-mounted image, ensuring accurate cropping. Vehicle information for each vehicle-mounted image is determined based on its similarity to images in a vehicle image library. This information is then combined with images from the vehicle image library to identify the vehicle-mounted images, improving the image recognition rate and consequently, the accuracy of vehicle information recognition. The positioning information of at least one vehicle, the vehicle information of each vehicle-mounted image, and the user's original question are concatenated to obtain model prompts. The model prompts and the original vehicle-mounted image are input into a visual language model to obtain a response corresponding to the user's original question, further improving the accuracy of vehicle question-and-answering by combining the visual language model with the original question.

[0024] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a vehicle question-and-answer method provided as an exemplary embodiment of this disclosure. Figure 1 As shown, the method includes: S101: Input the original vehicle image into the target positioning system to obtain the positioning of all vehicles in the original vehicle image, and obtain the positioning information of at least one vehicle, wherein the original vehicle image corresponds to the user's original question. S102: Based on the location information of at least one vehicle, the original vehicle image is cropped to obtain at least one vehicle image; S103: Determine the vehicle information for each in-vehicle image based on the similarity between each in-vehicle image and the images in the vehicle image library. S104: Combine the location information of at least one vehicle, the vehicle information of each in-vehicle image, and the user's original question to obtain the model prompt words; S105: Input the model prompts and the original vehicle image into the visual language model to obtain the response results corresponding to the user's original question.

[0026] In this embodiment, the execution subject of the above method is a terminal device or a server.

[0027] The terminal device includes, but is not limited to, mobile stations (MS), mobile terminals, mobile phones, handsets, and portable equipment. This terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or "cellular" phone), a computer with wireless communication capabilities, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical care, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. The operating systems installed on the terminal device include, but are not limited to, iOS, Android, Windows, Linux, and Mac OS. In different networks, terminals may be called by different names, such as: user equipment, mobile station, user unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop, cordless phone, wireless local loop station, television, etc. For ease of description, this embodiment will simply refer to it as terminal device.

[0028] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly consists of a processor, hard disk, memory, system bus, and other common computer architecture types.

[0029] To provide a clear and detailed explanation of this disclosure, the following explanations are provided for the technical terms used in this disclosure.

[0030] Embedding models represent information (such as images and text) as vectors in a low-dimensional, measurable space, thereby supporting similarity calculation, classification and clustering, and learning of downstream tasks. It is one of the core technologies in modern machine learning and deep learning. Embedding models can obtain a vector representation of an image from a single input image.

[0031] The reranking model is responsible for ranking the initially screened candidate results to a higher quality, and is a key step in improving the accuracy of retrieval or recommendation. The rerank model can take two images as input and obtain a relevance score between the two images.

[0032] Visual language models are a class of artificial intelligence models capable of simultaneously understanding and processing visual information (such as images and videos) and linguistic information (such as text and speech-to-text). Their core objective is to establish semantic alignment between visual and linguistic modalities, thereby enabling cross-modal understanding, generation, or reasoning.

[0033] Vehicle information, including but not limited to: vehicle brand, model, and angle information.

[0034] In some embodiments of this disclosure, the embedding model needs to be trained before it can be used. One possible approach is to construct a vehicle image library; construct a first training set based on the images in the vehicle image library; wherein the first training set includes: first positive example data and first negative example data, the first positive example data refers to two images of vehicles with the same vehicle information, and the first negative example data refers to two images of vehicles with different vehicle information; based on the first training set, the initial embedding model is trained using a contrastive learning loss to obtain a trained embedding model.

[0035] Specifically, training the embedding model includes the following steps: Step 1: Build a vehicle image library. Each image in the vehicle image library is an image of a vehicle from one of the four angles: front, rear, left, and right. The library contains images of various vehicles and records the brand, model, and angle information of each vehicle in each image. Step 2: Use images from the image library to construct a training set. The training set contains positive and negative examples. Positive examples are two images of vehicles with the same brand, model, and angle, while negative examples are two images of vehicles with different brands, models, and angles.

[0036] Step 3: Train the embedding model on the training set using contrastive learning loss to obtain the trained embedding model. Contrastive learning loss methods include CoSENT and InfoNCE.

[0037] In some embodiments of this disclosure, a re-ranking model needs to be trained before it can be used. One possible approach is to construct a second training set based on images from a vehicle image library. This second training set includes two positive examples and two negative examples. The positive examples are two images with identical vehicle information, and the negative examples are two images with different vehicle information. Based on this second training set, a binary classification loss is used to train the initial re-ranking model, resulting in a trained re-ranking model.

[0038] Specifically, training the reordering model involves the following steps: A training set is constructed using images from the image library. The training set contains positive and negative examples. Positive examples are two images of vehicles with the same brand, model, and angle, while negative examples are two images of vehicles with different brand, model, and angle.

[0039] On the training set, a rerank model is trained using binary classification loss (cross-entropy loss, hinge loss, etc.) to obtain a well-trained rerank model.

[0040] The embodiments disclosed herein can quickly increase the ability to recognize new vehicle information: only a small number of images of the new vehicle in the front, rear, left, and right directions need to be added to the image library, without retraining the embedding model and rerank model, to achieve question answering of new vehicle information.

[0041] In some embodiments of this disclosure, the original vehicle-mounted image is input into a target localization system to obtain the localization of all vehicles within the original vehicle-mounted image, thus obtaining the localization information of at least one vehicle. The target localization system can be a system composed of a visual language model or a target detection image model with vehicle localization capabilities. For example, firstly, the original vehicle-mounted image is input into a pre-trained visual language model, and the model is guided to perform target localization using the prompt "Please detect and box all vehicles in the image." The model combines the image features extracted by its visual encoder with the semantics of the language instruction to output the bounding box coordinates (x_min, y_min, x_max, y_max) and their confidence scores for each vehicle, ultimately forming a structured list of vehicle localization results. By acquiring the localization of all vehicles within the image, embodiments of this disclosure can correctly respond to situations where no vehicle is present in the image (e.g., replying "Sorry, the vehicle you mentioned is not found in the image"), thereby improving the accuracy of responses.

[0042] In some embodiments of this disclosure, the original vehicle image is cropped based on the location information of at least one vehicle to obtain at least one vehicle image. For example, for a detected car located at image coordinates (200, 150) to (320, 280), a sub-image is cropped using this bounding box as the region to generate a vehicle image with a size of 120×130 pixels; if three vehicles are detected, they are cropped according to their respective bounding boxes to obtain three independent vehicle images for subsequent vehicle model recognition.

[0043] In some embodiments of this disclosure, each vehicle image is input into an embedding model to obtain a first vector representation corresponding to each vehicle image; the similarity between each first vector representation and the second vector representations of all images in the vehicle image library is calculated, and this similarity is used as the similarity between each vehicle image and the images in the vehicle image library. Specifically, each vehicle image is input into the embedding model to obtain a first vector representation corresponding to each vehicle image, and each image in the vehicle image library is input into the embedding model to obtain a second vector representation corresponding to each image; the similarity between each first vector representation and the second vector representations of all images in the vehicle image library is calculated. It should be noted that this disclosure does not limit the similarity algorithm; it can be cosine similarity, Euclidean distance, etc.

[0044] In some embodiments of this disclosure, vehicle information for each in-vehicle image is determined based on the similarity between each in-vehicle image and images in a vehicle image library. One possible approach is to select candidate image libraries from all image libraries whose similarity meets a set similarity condition; for any in-vehicle image, calculate the relevance score between each candidate image library image and any in-vehicle image; select the target image library image and the target in-vehicle image with the highest relevance score from the combinations of each candidate image library image and any in-vehicle image; and determine the vehicle information for any in-vehicle image based on the relevance score between the target image library image and the target in-vehicle image.

[0045] In the above embodiments, candidate images that meet the set similarity criteria are selected from all image libraries. One possible approach is to select the top K candidate images with the highest similarity from all image libraries.

[0046] In the above embodiments, the relevance score between each candidate image and any vehicle image is calculated, and the target image and target vehicle image with the highest relevance score are selected from the combinations of each candidate image and any vehicle image. One possible approach is to use a reranking model to calculate the relevance score between each candidate image and any vehicle image, and then select the target image and target vehicle image with the highest relevance score from the combinations of each candidate image and any vehicle image. For example, K candidate images are combined with vehicle images, and the rerank model is used to calculate the relevance score for each combination, selecting the combination with the highest relevance score.

[0047] In the above embodiments, vehicle information for any vehicle image is determined based on the correlation score between the target image library image and the target vehicle image. One possible implementation is that if the correlation score between the target image library image and the target vehicle image is greater than or equal to a set threshold, the vehicle information of the target image library image is used as the vehicle information of the target vehicle image; if the correlation score is less than the set threshold, the vehicle information of the target vehicle image is set as unknown vehicle information. It should be noted that this disclosure does not limit the set threshold; the set threshold can be adjusted according to actual conditions, for example, 0.85, 0.9, etc. For example, if the correlation score between the target image library image and the target vehicle image is greater than 0.9, the vehicle information of the target image library image is used as the vehicle information of the target vehicle image; if the correlation score is less than 0.9, the vehicle information of the target vehicle image is set as unknown vehicle information. This embodiment of the disclosure can achieve correct rejection of unknown vehicles (such as replying "Sorry, we don't know the information about that vehicle at the moment") by performing relevance score threshold filtering, thereby improving the accuracy of the response.

[0048] In some embodiments of this disclosure, the location information of at least one vehicle, the vehicle information of each in-vehicle image, and the user's original question are concatenated to obtain model prompts. The model prompts and the original in-vehicle images are then input into a visual language model to obtain a response corresponding to the user's original question. For example, the system structurally concatenates the detected vehicle location information (e.g., "Two vehicles are detected in the image, located in areas (210,140,340,290) and (501,240,704,380) respectively"), the vehicle information of each cropped in-vehicle image (e.g., "Model: Brand, Model"), and the user's original question, "What model is the car to the right of the black car in the left front?", to generate model prompts: "The car located at (210,140,340,290) is a sedan of brand A, model a; the car located at (501,240,704,380) is an SUV of brand B, model b; the user asked, 'What model is the car to the right of the black car in the left front?'" "Please respond to the user's question based on the content of the image." Subsequently, the prompt word and the original in-vehicle image are input into the visual language model. The model integrates the visual content and text context, and outputs the response: "To the right of those two black A cars is a red B brand b model SUV."

[0049] This embodiment of the disclosure can accurately correspond to vehicle information in response to user questions; by identifying information of all vehicles in the vehicle image and using natural language to concatenate with the user's original question as prompt words, the visual language model can accurately answer information of a specific vehicle by combining image information.

[0050] Figure 2 This is a schematic diagram of a vehicle question-and-answer process provided as an exemplary embodiment of this disclosure. Figure 2 As shown, a target localization system is first used to locate all vehicles in the original in-vehicle image provided by the user. Based on the location coordinates, all vehicles in the in-vehicle image are extracted, resulting in N in-vehicle vehicle images. A vehicle image library is constructed, where each image is an image of a vehicle from a certain angle (front, back, left, right). The library contains images of various vehicles and their corresponding brand and model information. An image embedding model is trained using the images from the vehicle image library to achieve image retrieval based on the brand, model, and angle information of the vehicles in the image. Using a trained embedding model, retrieve the top K similar images from the image library for each vehicle image extracted from the vehicle images to be answered. Train an image rerank model using the vehicle image library to calculate the relevance score between two images based on the brand, model, and angle information of the vehicles in the images. For N vehicle images, use the trained embedding model to retrieve the top K similar images from the image library for each. Perform the following operations on each of the N vehicle images to obtain the recognition result: use the trained embedding model to retrieve the top K similar images from the image library for each vehicle image; use the trained rerank model... The ANK model calculates and sorts the relevance scores of in-vehicle images and the corresponding top K near-view images retrieved from the library. It then selects the image with the highest relevance score from the K images. Further, it performs a relevance score threshold filtering: if the relevance score is higher than the threshold, the brand and model information of that image is used as the vehicle information for the in-vehicle image; if the relevance score is lower than the threshold, the in-vehicle image is marked as unrecognizable. The model then uses the location information of N in-vehicle images and the corresponding vehicle brand and model information, combined with the user's original question, to construct a prompt word in natural language. This prompt word and the original in-vehicle image are then input into the visual language model to obtain the response to the user's question.

[0051] Figure 3 This is a schematic diagram of the structure of a vehicle question-and-answer device 30 provided for an exemplary embodiment of this disclosure. (See diagram below.) Figure 3 As shown, the vehicle question-and-answer device 30 includes: a positioning module 31, an interception module 32, a determination module 33, a splicing module 34, and a response module 35.

[0052] The positioning module 31 is used to input the original vehicle image into the target positioning system to obtain the positioning of all vehicles in the original vehicle image and obtain the positioning information of at least one vehicle. The original vehicle image corresponds to the user's original question. The cropping module 32 is used to crop the original vehicle image based on the positioning information of at least one vehicle to obtain at least one vehicle image. The determination module 33 is used to determine the vehicle information of each in-vehicle image based on the similarity between each in-vehicle image and the images in the vehicle image library. The stitching module 34 is used to stitch together the location information of at least one vehicle, the vehicle information of each in-vehicle vehicle image, and the user's original question to obtain model prompt words; The response module 35 is used to input the model prompts and the original vehicle image into the visual language model to obtain the response result corresponding to the user's original question.

[0053] Optionally, when determining the vehicle information of each in-vehicle image based on the similarity between each in-vehicle image and images in the vehicle image library, the determining module 33 is used to: Select candidate images from all image libraries that meet the set similarity criteria; For any in-vehicle image, calculate the correlation score between each candidate image and any in-vehicle image. Select the target image from the image library and the target vehicle image with the highest relevance score from each combination of candidate image library images and any vehicle image. Based on the correlation score between the target image library image and the target vehicle image, determine the vehicle information of any vehicle image.

[0054] Optionally, module 33 can also be used for: Each in-vehicle image is input into the embedding model to obtain the first vector representation corresponding to each in-vehicle image; Calculate the similarity between each first vector representation and the second vector representation of all images in the vehicle image library, and use this as the similarity between each in-vehicle vehicle image and the images in the vehicle image library.

[0055] Optionally, when determining the correlation score between each candidate image in the image library and any on-board vehicle image, the determining module 33 is used to: The re-ranking model is used to calculate the correlation score between each candidate image in the image library and any on-board vehicle image.

[0056] Optionally, when determining vehicle information for any vehicle image based on the correlation score between the target image library image and the target vehicle image, the determining module 33 is used to: If the correlation score between the target image library image and the target vehicle image is greater than or equal to a set score threshold, then the vehicle information of the target image library image will be used as the vehicle information of the target vehicle image. If the correlation score between the target image library image and the target vehicle image is less than a set score threshold, the vehicle information of the target vehicle image will be set to unknown vehicle information.

[0057] Optionally, the determining module 33 can also be used before using the embedded model: Build a vehicle image library; Based on the images in the vehicle image library, a first training set is constructed; wherein, the first training set includes: a first positive example data and a first negative example data, the first positive example data refers to two images of vehicles with the same vehicle information, and the first negative example data refers to two images of vehicles with different vehicle information; Based on the first training set, the initial embedding model is trained using contrastive learning loss to obtain the trained embedding model.

[0058] Optionally, the determination module 33 can also be used before using the reordering model: A second training set is constructed based on the images in the vehicle image library. The second training set includes: second positive examples and second negative examples. The second positive examples refer to two images with the same vehicle information, and the second negative examples refer to two images with different vehicle information. Based on the second training set, the initial re-ranking model is trained using binary classification loss to obtain the trained re-ranking model.

[0059] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0060] Figure 4 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. For example... Figure 4 As shown, the electronic device includes a memory 41 and a processor 42. Additionally, the electronic device also includes a power supply component 43 and a communication component 44.

[0061] Memory 41 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0062] The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0063] Communication component 44 is used for data transmission with other devices.

[0064] The processor 42 is capable of executing computer instructions stored in the memory 41 for: inputting the original vehicle-mounted image into the target positioning system to obtain the positioning of all vehicles in the original vehicle-mounted image, and obtaining the positioning information of at least one vehicle, wherein the original vehicle-mounted image corresponds to the user's original question; cropping the original vehicle-mounted image based on the positioning information of at least one vehicle to obtain at least one vehicle-mounted image; determining the vehicle information of each vehicle-mounted image based on the similarity between each vehicle-mounted image and images in the vehicle image library; concatenating the positioning information of at least one vehicle, the vehicle information of each vehicle-mounted image, and the user's original question to obtain model prompt words; and inputting the model prompt words and the original vehicle-mounted image into a visual language model to obtain a response result corresponding to the user's original question.

[0065] Accordingly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores a computer program, and the computer program is executed by one or more processors, it causes one or more processors to perform... Figure 1 Each step in the method embodiment.

[0066] Accordingly, embodiments of this disclosure also provide a computer program product, which includes a computer program / instructions that are executed by a processor. Figure 1 Each step in the method embodiment.

[0067] The above Figure 4 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 4G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0068] The above Figure 4 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0069] The aforementioned electronic devices also include a display screen and audio components.

[0070] The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also the duration and pressure associated with the touch or swipe operation.

[0071] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0072] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0080] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle question-and-answer method, characterized in that, include: The original vehicle image is input into the target positioning system to obtain the positioning of all vehicles in the original vehicle image, and the positioning information of at least one vehicle is obtained, wherein the original vehicle image corresponds to the user's original question. Based on the location information of the at least one vehicle, the original vehicle image is cropped to obtain at least one vehicle image. The vehicle information of each in-vehicle image is determined based on the similarity between each in-vehicle image and an image in the vehicle image library. The location information of at least one vehicle, the vehicle information of each in-vehicle image, and the user's original question are concatenated to obtain model prompt words; The model prompts and the original vehicle image are input into the visual language model to obtain the response result corresponding to the user's original question.

2. The method according to claim 1, characterized in that, The step of determining the vehicle information for each in-vehicle image based on its similarity to images in the vehicle image library includes: Select candidate images from all image libraries whose similarity meets the set similarity criteria; For any of the vehicle images, calculate the correlation score between each candidate image in the image library and any of the vehicle images. The target image and the target vehicle image with the highest relevance score are selected from each combination of the candidate image library image and any one of the vehicle images. Based on the correlation score between the target image library image and the target vehicle image, the vehicle information of any one of the vehicle images is determined.

3. The method according to claim 2, characterized in that, The method further includes: Each of the vehicle images is input into the embedding model to obtain a first vector representation corresponding to each vehicle image; The similarity between each of the first vector representations and the second vector representations of all images in the vehicle image library is calculated, and this similarity is used as the similarity between each of the in-vehicle images and the images in the vehicle image library.

4. The method according to claim 2, characterized in that, The step of calculating the correlation score between each candidate image in the image library and any one of the in-vehicle images includes: The re-ranking model is used to calculate the correlation score between each candidate image in the image library and any one of the in-vehicle images.

5. The method according to claim 2, characterized in that, The step of determining the vehicle information of any one of the vehicle images based on the correlation score between the target image library image and the target vehicle image includes: If the correlation score between the target image library image and the target vehicle image is greater than or equal to a set score threshold, then the vehicle information of the target image library image is used as the vehicle information of the target vehicle image. If the correlation score between the target image in the image library and the target vehicle image is less than the set score threshold, then the vehicle information of the target vehicle image is set to unknown vehicle information.

6. The method according to claim 1, characterized in that, The method further includes the following steps prior to using the embedding model: Construct the vehicle image library; A first training set is constructed based on the images in the vehicle image library; wherein, the first training set includes: a first positive example data and a first negative example data, the first positive example data refers to two images of vehicles with the same vehicle information, and the first negative example data refers to two images of vehicles with different vehicle information. Based on the first training set, the initial embedding model is trained using contrastive learning loss to obtain the trained embedding model.

7. The method according to claim 1, characterized in that, Prior to using the reordering model, the method further includes: A second training set is constructed based on the images in the vehicle image library; wherein, the second training set includes: second positive example data and second negative example data, the second positive example data refers to two images of vehicles with the same vehicle information, and the second negative example data refers to two images of vehicles with different vehicle information; Based on the second training set, the initial re-ranking model is trained using binary classification loss to obtain the trained re-ranking model.

8. A vehicle question-and-answer device, characterized in that, include: The positioning module is used to input the original vehicle image into the target positioning system to obtain the positioning of all vehicles in the original vehicle image and obtain the positioning information of at least one vehicle, wherein the original vehicle image corresponds to the user's original question. The cropping module is used to crop the original vehicle image based on the positioning information of the at least one vehicle to obtain at least one vehicle image. The determination module is used to determine the vehicle information of each in-vehicle image based on the similarity between each in-vehicle image and an image in the vehicle image library. The stitching module is used to stitch together the location information of the at least one vehicle, the vehicle information of each of the in-vehicle images, and the user's original question to obtain model prompt words; The response module is used to input the model prompts and the original vehicle image into the visual language model to obtain a response result corresponding to the user's original question.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images