Driving exploration method and device
By using routing agents and function calls agents in the intelligent cockpit system combined with the target map tool, accurately searching and identifying points of interest in the target direction, the problem of poor results accuracy when users query complex scenarios in the intelligent cockpit system is solved, and higher query results accuracy is achieved.
Patent Information
- Application Number
- CN202510160127.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-13
- Publication Date
- 2025-05-16
AI Technical Summary
The exploration function of surrounding scenes in the existing smart cockpit system is relatively single, and the query results returned by users when querying complex scenes are poor.
The target workflow for processing user query information is determined through the routing agent, and the function call agent and the target map tool are used to accurately search for the points of interest within the preset range of the target direction based on the user query information, vehicle coordinates, driving direction and other context information, and return the list of interest points, and determine the target interest points based on the similarity between the user query information and the list of interest points.
It improves the accuracy of the query results returned by users' query information during driving exploration, and can more accurately identify and understand complex scenarios.
Smart Images
Figure CN120011657A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of smart cockpits, and in particular to a driving exploration method and device. Background Art
[0002] With the rapid development of the smart cockpit industry and big model technology, many car companies have tried to add the function of exploring surrounding scenes while driving to the smart cockpit system.
[0003] However, the current intelligent cockpit system's exploration function of surrounding scenes is relatively simple. It only uses the capabilities of the visual big language model itself to generate descriptions of images taken by ADAS, AVM and other cameras. It can often only recognize and understand some simple scenes, such as describing the scene in front and identifying road signs.
[0004] For more complex scenarios of user queries, the accuracy of the returned query results is poor. Summary of the invention
[0005] In view of this, the present disclosure provides a driving exploration method and device to solve the problem of poor accuracy of query results returned by user queries.
[0006] In a first aspect, the present disclosure provides a driving exploration method, the method comprising: upon receiving user query information, using a routing agent to determine a target workflow for processing the user query information; upon determining that the target workflow is a point of interest exploration workflow, inputting the user query information, vehicle coordinates and driving direction into a function calling agent, obtaining query parameters and a target map tool returned by the function calling agent; the query parameters include: vehicle coordinates, driving direction, exploration direction, point of interest keywords and point of interest description information; calling the target map tool, searching for points of interest within a preset range of the target direction according to the query parameters, and returning a list of candidate points of interest; the points of interest in the candidate list of points of interest include: point of interest name, point of interest coordinates, point of interest distance, point of interest introduction and point of interest picture; determining the target point of interest based on the similarity between the user query information and each point of interest in the candidate list of points of interest; and returning the query results generated based on the target point of interest to the user.
[0007] The driving exploration method in the above-mentioned embodiment of the present disclosure can adopt a routing agent to dynamically route to the most suitable workflow according to the user query information, and use a function call intelligent agent and a target map tool to accurately search for points of interest within a preset range of the target direction according to contextual information such as user query information, vehicle coordinates, and driving direction, thereby improving the accuracy of the query results returned by the user query information during the driving exploration process.
[0008] In a second aspect, the present disclosure provides a driving exploration device, which includes: a workflow determination module, which is used to use a routing agent to determine a target workflow for processing user query information when receiving user query information; an agent calling module, which is used to input user query information, vehicle coordinates and driving direction into a function calling agent when determining that the target workflow is an interest point exploration workflow, and obtain query parameters and a target map tool returned by the function calling agent; the query parameters include: vehicle coordinates, driving direction, exploration direction, interest point keywords and interest point description information; the map tool calling module is used to call the target map tool, search for interest points within a preset range of the target direction according to the query parameters, and return a list of interest point candidates; the interest points in the interest point candidate list include: interest point name, interest point coordinates, interest point distance, interest point introduction and interest point picture; the interest point determination module is used to determine the target interest point based on the similarity between the user query information and each interest point in the interest point candidate list; the interest point return module is used to return the query results generated based on the target interest point to the user.
[0009] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the driving exploration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0010] In a fourth aspect, the present disclosure provides a smart cockpit, in which computer instructions are stored, and the computer instructions are used to enable the smart cockpit to execute the driving exploration method of the first aspect or any corresponding embodiment thereof.
[0011] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, and the computer instructions are used to enable a computer to execute the driving exploration method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0013] Figure 1 is a flowchart of a driving exploration method according to an embodiment of the present disclosure;
[0014] Figure 2 is a flowchart of another vehicle exploration method according to an embodiment of the present disclosure;
[0015] Figure 3 is a flowchart of an application scenario of the driving exploration method according to an embodiment of the present disclosure;
[0016] Figure 4 is a structural block diagram of a driving exploration device according to an embodiment of the present disclosure;
[0017] Figure 5 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0019] According to an embodiment of the present disclosure, an embodiment of a driving exploration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0020] In this embodiment, a driving exploration method is provided, which can be used in the above-mentioned computer system, such as a vehicle brain, a smart cockpit, etc. Those skilled in the art can understand that in some instances, the execution subject of the embodiments of the present disclosure can exist in the form of a driving exploration agent. Figure 1 is a flow chart of a driving exploration method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:
[0021] Step S101, when receiving user query information, use a routing agent to determine a target workflow for processing the user query information.
[0022] In this embodiment, the driving exploration agent is used as an example for explanation. After receiving the user query information (query), the driving exploration agent first performs a domain based on the routing agent (RouterAgent, also called routing agent) to determine which workflow to distribute to.
[0023] Specifically, the routing agent can be pre-set to be able to land on multiple driving exploration workflows, each of which is responsible for processing a specific type of task. After the routing agent receives the user query information, it can distribute the user query information to the corresponding driving exploration workflow according to predefined rules or dynamic analysis. When a user query information includes multiple requirements, the routing agent can distribute the multiple requirements to different driving exploration workflows respectively.
[0024] In some specific examples, the routing agent may be a locally deployed large model, or may be an application program interface (API) based on an external large model. The driving exploration workflow may be a point of interest exploration workflow, a visual question answering workflow, etc. In some examples, the driving exploration workflow may also be a travel recommendation workflow, a life service workflow, etc.
[0025] Step S102, when it is determined that the target workflow is the POI exploration workflow, the user query information, vehicle coordinates and driving direction are input into the function calling agent, and the query parameters and target map tool returned by the function calling agent are obtained.
[0026] In this embodiment, the query parameters may include: vehicle coordinates, driving direction, exploration direction, interest point keywords and interest point description information.
[0027] Among them, the vehicle coordinates can describe the current position of the user's vehicle, for example, the description method can include longitude and latitude, etc. The driving direction can describe the direction of the vehicle's front, for example, the description content can include east / west / south / north, etc. The exploration direction can describe the direction in the user's query information, for example, the description content can include front / back / left / right, etc. The point of interest keywords can describe the key information of the point of interest included in the user's query information, for example, the description content can include the type of target, such as shopping mall / building / park, etc. The point of interest description can be a description of the point of interest, for example, it can be none / white building / glass building, etc.
[0028] When the routing agent determines that the target workflow is the point of interest exploration workflow, the driving exploration agent can obtain the vehicle coordinates and driving direction of the current vehicle, and input the user query information, vehicle coordinates and driving direction into the function calling agent, so that the function calling agent processes the user query information, vehicle coordinates and driving direction, generates the above query parameters, and returns the target map tool. Specifically, the target map tool can be a map tool in the list of available map tools.
[0029] Step S103, calling the target map tool, searching for points of interest within a preset range of the target direction according to the query parameters, and returning a list of candidate points of interest.
[0030] In this embodiment, the points of interest in the candidate list of points of interest include: point of interest name, point of interest coordinates, point of interest distance, point of interest introduction, and point of interest picture. The target direction may be a query direction determined according to the query parameter. For example, the target direction may be determined according to the driving direction, or according to the exploration direction, or according to the driving direction and the exploration direction that the user wants to explore.
[0031] In some examples, the driving exploration agent may combine the relative direction information of the exploration direction with the absolute direction information of the driving direction to obtain the target direction. In some specific examples, if the driving direction is south and the exploration direction is forward, the target direction may be forward of the south.
[0032] After receiving the target map tool returned by the function calling agent, the driving exploration agent can call the target map tool to search for points of interest within a preset range of the target direction according to the query parameters, and receive a candidate list of points of interest that meet the query parameters returned by the target map tool.
[0033] Step S104: determining a target point of interest based on the similarity between the user query information and each point of interest in the candidate point of interest list.
[0034] In this embodiment, the above-mentioned driving exploration agent can generate user query information features and interest point features that can be used for comparison based on the user query information and each interest point in the interest point candidate list, so as to compare the similarity between the two, and thus determine the interest point with the greatest similarity as the target interest point corresponding to the user query information.
[0035] It is understandable that when generating user query information features and point of interest features, features can be generated separately based on similar or identical contents included in the two types of data, namely, user query information and point of interest, so as to ensure the comparability of features generated by the two types of data. For example, the contents included in the two types of data can be similar or identical, such as the name of the point of interest, the geographical location of the point of interest (absolute latitude and longitude or relative distance, etc.), and the introduction of the point of interest, etc.
[0036] Step S105: Return the query result generated based on the target point of interest to the user.
[0037] In this embodiment, after determining the target point of interest, the driving exploration agent may directly return the target point of interest as a query result to the user, or may generate a query result using a predetermined template for the relevant information of the target point of interest and then return the query result to the user. This disclosure does not limit this.
[0038] The driving exploration method in the above-mentioned embodiment of the present disclosure can adopt a routing agent to dynamically route to the most suitable workflow according to the user query information, and use a function call intelligent agent and a target map tool to accurately search for points of interest within a preset range of the target direction according to contextual information such as user query information, vehicle coordinates, and driving direction, thereby improving the accuracy of the query results returned by the user query information during the driving exploration process.
[0039] In some optional implementations of the above embodiments, the above method may further include: when it is determined that the target workflow is a visual question-answering workflow of driving record images, calling a visual large language model to process user query information and driving record images, and returning target reply information.
[0040] In this implementation, the Visual Large Language Model (VLM), also known as the Joint Visual Language Model or Visual Language Large Model, is a deep learning model that combines computer vision (CV) and natural language processing (NLP) technologies. The Visual Large Language Model can automatically recognize, understand, and generate image content. It can not only understand and analyze image content, but also generate natural language descriptions or instructions related to it, realizing seamless conversion between vision and language.
[0041] When receiving user query information, the above-mentioned driving exploration agent can determine whether the user query information is a question and answer regarding the target object recorded in the driving record image. If it is a question and answer regarding the target object recorded in the driving record image, the visual large language model is called to understand and analyze the user query information and the driving record image, generate a natural language description related thereto, and return the natural language description to the user as the target reply information.
[0042] For example, if a user queries information about a car or tree in front that can be answered using visual recognition, there is no need to call external tools. Instead, the visual large language model can be called to process the user query information and driving record images and return the target answer information.
[0043] The driving exploration method in this implementation can call the visual large language model to process user query information and driving record images when determining that the target workflow is the visual question and answer workflow of driving record images, and return the target response information, thereby realizing seamless conversion between vision and language, realizing effective fusion between visual embedding and language embedding, and improving the ability to understand user query information and driving record images.
[0044] In this embodiment, another driving exploration method is provided, which can be used in the above-mentioned computer systems, such as the vehicle-mounted brain, the smart cockpit, etc. Figure 2 is a flow chart of a driving exploration method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:
[0045] Step S201, when receiving user query information, use a routing agent to determine a target workflow for processing the user query information.
[0046] In this embodiment, the driving exploration agent is still used as an example for explanation. After receiving the user query information (query), the driving exploration agent first performs a domain based on the routing agent (RouterAgent, also called routing agent) to determine which workflow to distribute to.
[0047] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0048] Step S202, when it is determined that the target workflow is the POI exploration workflow, the user query information, vehicle coordinates and driving direction are input into the function calling agent, and the query parameters and target map tool returned by the function calling agent are obtained.
[0049] In this embodiment, the query parameters may include: vehicle coordinates, driving direction, exploration direction, interest point keywords and interest point description information.
[0050] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0051] Step S203, calling the target map tool, searching for points of interest within a preset range of the target direction according to the query parameters, and returning a list of candidate points of interest.
[0052] In this embodiment, the points of interest in the candidate list of points of interest may include: point of interest name, point of interest coordinates, point of interest distance, point of interest introduction, and point of interest picture. The target direction may be a query direction determined according to the query parameter. For example, the target direction may be determined according to the driving direction, or according to the exploration direction, or according to the driving direction and the exploration direction that the user wants to explore.
[0053] In some examples, the driving exploration agent may combine the relative direction information of the exploration direction with the absolute direction information of the driving direction to obtain the target direction. In some specific examples, if the driving direction is south and the exploration direction is forward, the target direction may be forward of the south.
[0054] After receiving the target map tool returned by the function calling agent, the driving exploration agent can call the target map tool to search for points of interest within a preset range of the target direction according to the query parameters, and receive a candidate list of points of interest that meet the query parameters returned by the target map tool.
[0055] Step S204, when the description information of the point of interest is empty, call the visual large language model, identify the coordinates of the point of interest bounding box based on the point of interest keywords from the driving record image located in the exploration direction, and obtain the point of interest record image corresponding to the point of interest bounding box coordinates.
[0056] In this embodiment, the Visual Large Language Model (VLM), also known as the joint visual language model or the visual language large model, is a deep learning model that combines computer vision (CV) and natural language processing (NLP) technology. The visual large language model can realize the automatic recognition, understanding and generation of image content. It can not only understand and analyze the image content, but also generate natural language descriptions or instructions related to it, realizing seamless conversion between vision and language. In some specific examples, the visual large language model can be a comparative language-image pre-training model (CLIP), a Dali model (DALL·E), a visual BERT model (VisualBERT), a language-visual multimodal Transformer (LXMERT), a Flamingo (Flamingo), etc., or it can be a self-developed visual large language model.
[0057] When the description information of the points of interest in the user query information is empty, it means that the user has not described the points of interest in words, and the user query information cannot provide multimodal information to determine the points of interest that the user wants to explore. At this time, the driving exploration agent can call the visual large language model to identify the bounding box coordinates of the points of interest from the driving record images located in the exploration direction based on the keywords of the points of interest, and obtain the recorded images of the points of interest corresponding to the bounding box coordinates of the points of interest.
[0058] Step S205 , calling the contrast language-image pre-training model, generating a first similarity between the interest point record image and the interest point pictures of each interest point in the interest point candidate list, and obtaining a first similarity set.
[0059] In this embodiment, the contrastive language-image pre-training model is a multimodal model that can embed natural language and images into the same semantic space, thereby realizing cross-modal understanding and association between images and images, and between images and texts, so that paired image pairs / image-text pairs are closer in the vector space, and unpaired image pairs / image-text pairs are farther in the vector space. For example, the contrastive language-image pre-training model (CLIP), efficient network (EfficientNet), vision transformer (Vision Transformer), DINO-v2 (DINO-v2), image-language pre-training model (2BLIP-2), unified perceptron (Uni-Perceiver), etc., can also be a self-developed contrastive language-image pre-training model.
[0060] After obtaining the interest point record image, the above-mentioned driving exploration agent can input the interest point record image and the interest point pictures of each interest point in the candidate list into the comparative language-image pre-training model to output the image representation vector and perform similarity calculation to obtain the first similarity corresponding to each interest point, thereby obtaining a first similarity set.
[0061] Step S206: determine the point of interest corresponding to the point of interest picture with the maximum similarity in the first similarity set as the target point of interest.
[0062] In this embodiment, the driving exploration agent can determine the maximum similarity value, that is, the maximum similarity, from the first similarity set, and then determine the point of interest to which the point of interest picture corresponding to the maximum similarity belongs, and determine the point of interest as the target point of interest.
[0063] Step S207, when the interest point description information is not empty, calling the comparative language-image pre-training model, respectively generating second similarities between the interest point description information and the interest point pictures of each interest point in the interest point candidate list, and obtaining a second similarity set.
[0064] In this embodiment, the called contrastive language-graphic pre-training model may be the same as the contrastive language-graphic pre-training model in step S205, or may be different, and this disclosure does not limit this.
[0065] After obtaining the POI description information, the above-mentioned driving exploration agent can input the POI description information and the POI pictures of each POI in the POI candidate list into the comparative language-image pre-training model, thereby outputting a text representation vector and an image representation vector, and calculating the similarity between the two vectors (such as cosine similarity), thereby obtaining a second similarity set.
[0066] Step S208: determining the point of interest corresponding to the point of interest picture with the maximum similarity in the second similarity set as the target point of interest.
[0067] In this embodiment, the driving exploration agent may first determine the maximum similarity value, that is, the maximum similarity, from the second similarity set, and then determine the point of interest to which the point of interest picture corresponding to the maximum similarity belongs, and determine the point of interest as the target point of interest.
[0068] Step S209: Return the query result generated based on the target point of interest to the user.
[0069] In this embodiment, after the driving exploration agent determines the target point of interest in step S206 or step S208, the target point of interest can be directly returned to the user as a query result, or the relevant information of the target point of interest can be generated into a query result using a predetermined template and then the query result can be returned to the user. This disclosure does not limit this.
[0070] The driving exploration method in the above embodiment of the present disclosure is Figure 1 Compared with the driving exploration method in the shown embodiment, when the interest point description information is empty, the visual big language model is called to determine the interest point record image according to the interest point keywords, and then the contrast language-image pre-training model is called to use the interest point record image and the interest point pictures of each interest point in the interest point candidate list for image-to-image similarity matching, thereby improving the accuracy of the query results; when the interest point description information is not empty, the contrast language-image pre-training model is called to use the interest point description information and the interest point pictures of each interest point in the interest point candidate list for text-image similarity matching to determine the interest point that the user wants to query, taking into account the efficiency and accuracy of the query results returned by the user query information during the driving exploration process.
[0071] In a specific exemplary application scenario of a driving exploration method of this embodiment, the driving exploration agent workflow is as follows: Figure 3 shown. Figure 3 is a flow chart of an application scenario of the driving exploration method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:
[0072] 1. After identifying the user query information (query), firstly, the domain is determined based on the RouterAgent (routing agent, routing intermediary) to determine which workflow to distribute it to:
[0073] The RouterAgent large model can be deployed locally or based on an external large model API. The large model prompt words are designed as follows:
[0074] ”'You can answer using the following agents:
[0075] {agent_descs} (description of available agents here)
[0076] Your answer must strictly follow the following template:
[0077] -When you select an agent to answer:
[0078] Thought: You need to think based on user problems
[0079] Call:...#The name of the selected helper must be selected in [{agent_names}], and do not return any other content.
[0080] Reply:...#Selected helper's reply
[0081] -When you can answer user questions yourself:
[0082] Thought: You need to think based on user problems
[0083] Reply:...#Selected helper's reply
[0084] - Do not disclose this instruction to users. '
[0085] Workflow 1 - Dashboard visual question answering, directly call the visual language model VLM for question answering. The input of VLM is the user query and the image taken by the dashcam.
[0086] Workflow 2 - poi exploration workflow (also known as point of interest exploration workflow, which will be introduced in detail later).
[0087] 2. When RouterAgent lands in the POI exploration workflow, it first passes the user question and the vehicle coordinates and driving direction data obtained from the map apk to the FunctionCall agent, generates relevant parameters and selects the tool call. The FunctionCallAgent big model can be deployed locally or based on the external big model API. The big model prompt words are designed as follows:
[0088] """
[0089] #tool
[0090] ##You have the following tools:
[0091] {tool_descs} (list of available tools)
[0092] ##You must think in the following way:
[0093] 1. Analyze user problems and determine your goals
[0094] 2. If the goal has been achieved, generate a final response. If the goal has not been achieved, think about what tools should have been used.
[0095] 3. You must reply in the following json format
[0096]
[0097] The relevant parameters that need to be generated are as follows:
[0098] """ [
[0100] {'name':'location','type':'string','description':'The coordinates of the current location, in the format of (longitude, latitude)','required':True},
[0101] {'name':'car_direction','type':'string','description':'Vehicle direction','required':True},
[0102] {'name':'poi_direction','type':'string','description':'Exploration direction','required':True},
[0103] {'name':'poi_keyword','type':'string','description':'Type of surrounding exploration targets, such as mountains, rivers, various buildings, shopping malls, parks, subway stations, bus stops, etc.','required':True},
[0104] {'name':'poi_description','type':'string','description':'Description of the exploration target, such as color, shape, height, material, pattern and other feature descriptions','required':False}, ]
[0106] """
[0107] 3. Call map API tools, such as Amap or Baidu Map, and search for a candidate list of POIs within the target direction based on the generated POI keywords, current coordinates, and exploration direction parameters. The returned list includes POI name, POI introduction, POI distance, and POI picture.
[0108] 4. If the generated parameters include POI description, the multimodal large model CLIP is called to perform image-text similarity matching. The method is: input the POI image and POI description in the candidate list in the previous step into the CLIP multimodal large model respectively, output the text representation vector and the image representation vector and calculate the cosine similarity between the two vectors. The POI with the smallest cosine similarity is the POI that the user ultimately wants to query, and the answer is returned to the user.
[0109] 5. If the generated parameters do not include POI description, it is necessary to call the visual language model VLM, input the driving record image of the target direction and the POI keyword into the VLM, obtain the POI coordinate frame in the captured image and crop the corresponding part of the POI driving record image; then input the POI driving record image and the POI pictures in the candidate list in Part 3 into the CLIP multimodal large model, output the image representation vector and calculate the cosine similarity. The POI with the smallest cosine similarity is the POI that the user ultimately wants to query, and the answer is returned to the user.
[0110] In this embodiment, a driving exploration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0111] This embodiment provides a driving exploration device, such as Figure 4 As shown, including:
[0112] The workflow determination module 401 is used to determine the target workflow for processing the user query information using the routing agent when receiving the user query information;
[0113] The agent calling module 402 is used to input the user query information, vehicle coordinates and driving direction into the function calling agent when the target workflow is determined to be the POI exploration workflow, and obtain the query parameters and target map tool returned by the function calling agent; the query parameters include: vehicle coordinates, driving direction, exploration direction, POI keywords and POI description information;
[0114] The map tool calling module 403 is used to call the target map tool, search for points of interest within a preset range of the target direction according to the query parameters, and return a list of candidate points of interest; the points of interest in the candidate list of candidate points of interest include: point of interest name, point of interest coordinates, point of interest distance, point of interest introduction and point of interest picture;
[0115] The interest point determination module 404 is used to determine the target interest point based on the similarity between the user query information and each interest point in the interest point candidate list;
[0116] The interest point returning module 405 is used to return the query result generated based on the target interest point to the user.
[0117] In some optional embodiments, the interest point determination module 404 includes (not shown in the figure): an image acquisition submodule, which is used to call the visual large language model when the interest point description information is empty, identify the interest point bounding box coordinates based on the interest point keywords from the driving record image located in the exploration direction, and obtain the interest point record image corresponding to the interest point bounding box coordinates; an image comparison submodule, which is used to call the comparison language-image pre-training model, and respectively generate a first similarity between the interest point record image and the interest point picture of each interest point in the interest point candidate list to obtain a first similarity set; an interest point determination submodule, which is used to determine the interest point corresponding to the interest point picture with the maximum similarity in the first similarity set as the target interest point.
[0118] In some optional embodiments, the interest point determination module 404 includes (not shown in the figure): a text-image comparison sub-module, which is used to call the comparison language-image pre-training model when the interest point description information is not empty, and generate a second similarity between the interest point description information and the interest point pictures of each interest point in the interest point candidate list, to obtain a second similarity set; an interest point determination sub-module, which is used to determine the interest point corresponding to the interest point picture with the maximum similarity in the second similarity set as the target interest point.
[0119] In some optional embodiments, the device further comprises:
[0120] The model calling module 406 is used to call the visual large language model to process the user query information and the driving record image and return the target answer information when it is determined that the target workflow is the visual question answering workflow.
[0121] In some optional implementations, the target direction in the map tool calling module is determined based on the driving direction and the exploration direction.
[0122] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0123] The driving exploration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0124] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present disclosure, and the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).
[0125] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0126] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0127] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0128] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0129] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0130] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.
[0131] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A driving exploration method, characterized in that: The method comprises: Upon receiving user query information, determining a target workflow for processing the user query information using a routing agent; When the target workflow is determined to be a point of interest exploration workflow, the user query information, vehicle coordinates and driving direction are input into a function calling agent, and query parameters and a target map tool returned by the function calling agent are obtained; the query parameters include: vehicle coordinates, driving direction, exploration direction, point of interest keywords and point of interest description information; Calling the target map tool, searching for points of interest within a preset range of the target direction according to the query parameters, and returning a list of candidate points of interest; the points of interest in the candidate list of points of interest include: point of interest name, point of interest coordinates, point of interest distance, point of interest introduction and point of interest picture; Determining a target point of interest based on the similarity between the user query information and each point of interest in the candidate list of points of interest; Return the query result generated based on the target point of interest to the user.
2. The method according to claim 1, characterized in that The determining the target point of interest based on the similarity between the user query information and each point of interest in the candidate list of points of interest includes: When the description information of the point of interest is empty, calling the visual large language model, identifying the coordinates of the bounding box of the point of interest based on the keyword of the point of interest from the driving record image located in the exploration direction, and obtaining the recorded image of the point of interest corresponding to the coordinates of the bounding box of the point of interest; Calling a comparative language-image pre-training model to respectively generate first similarities between the interest point record image and interest point pictures of each interest point in the interest point candidate list to obtain a first similarity set; The point of interest corresponding to the point of interest picture with the greatest similarity in the first similarity set is determined as the target point of interest.
3. The method according to any one of claims 1 or 2, characterized in that: The determining the target point of interest based on the similarity between the user query information and each point of interest in the candidate list of points of interest includes: When the interest point description information is not empty, calling a comparative language-image pre-training model to respectively generate second similarities between the interest point description information and interest point images of each interest point in the interest point candidate list to obtain a second similarity set; The point of interest corresponding to the point of interest picture with the maximum similarity in the second similarity set is determined as the target point of interest.
4. The method according to claim 1, characterized in that: The method further comprises: When it is determined that the target workflow is a visual question-answering workflow, a visual large language model is called to process the user query information and the driving record image, and target answer information is returned.
5. The method according to claim 1, characterized in that The target direction is determined based on the driving direction and the exploration direction.
6. A driving exploration device, characterized in that: The device includes: A workflow determination module, configured to, upon receiving user query information, use a routing agent to determine a target workflow for processing the user query information; An agent calling module, for inputting the user query information, vehicle coordinates and driving direction into a function calling agent when determining that the target workflow is a point of interest exploration workflow, and obtaining query parameters and a target map tool returned by the function calling agent; The query parameters include: vehicle coordinates, driving direction, exploration direction, POI keywords and POI description information; A map tool calling module is used to call a target map tool, search for points of interest within a preset range of a target direction according to the query parameters, and return a list of candidate points of interest; the points of interest in the candidate list of points of interest include: point of interest name, point of interest coordinates, point of interest distance, point of interest introduction and point of interest picture; An interest point determination module, configured to determine a target interest point based on the similarity between the user query information and each interest point in the interest point candidate list; The interest point returning module is used to return the query result generated based on the target interest point to the user.
7. The device according to claim 6, characterized in that The point of interest determination module includes: An image acquisition submodule, for calling a visual large language model when the description information of the interest point is empty, identifying the coordinates of the bounding box of the interest point based on the interest point keyword from the driving record image located in the exploration direction, and acquiring the interest point record image corresponding to the coordinates of the bounding box of the interest point; An image comparison submodule, used for calling a comparison language-image pre-training model, respectively generating first similarities between the interest point record image and the interest point pictures of each interest point in the interest point candidate list, and obtaining a first similarity set; The interest point determination submodule is used to determine the interest point corresponding to the interest point picture with the maximum similarity in the first similarity set as the target interest point.
8. The device according to claim 7, characterized in that The point of interest determination module includes: A text-image comparison submodule is used to call a comparison language-image pre-training model when the interest point description information is not empty, and respectively generate a second similarity between the interest point description information and the interest point pictures of each interest point in the interest point candidate list to obtain a second similarity set; The interest point determination submodule is used to determine the interest point corresponding to the interest point picture with the maximum similarity in the second similarity set as the target interest point.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the driving exploration method of any one of claims 1 to 5 by executing the computer instructions.
10. A smart cockpit, characterized in that: The smart cockpit stores computer instructions, which are used to enable the smart cockpit to execute the driving exploration method of any one of claims 1 to 5.