End-to-end map engine evaluation method, apparatus and electronic device

Abstract

This disclosure provides an end-to-end map engine evaluation method, apparatus, and electronic device, relating to the field of data processing technology, specifically artificial intelligence, map navigation, and intelligent agent technology. The specific implementation scheme includes: obtaining evaluation instructions; simulating the execution of an evaluation task in the map engine according to the evaluation instructions using an intelligent agent; collecting page information of the target service area displayed by the map engine during the task simulation; obtaining second POI information of the target service area from a map database; evaluating the map engine based on the first and second POI information to obtain evaluation results, thereby improving the evaluation accuracy and reliability of map navigation POI display quality.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to the fields of artificial intelligence, map navigation and intelligent agents, and particularly to an end-to-end map engine evaluation method, apparatus and electronic device. Background Technology

[0002] In map navigation applications, the Point of Interest (POI) database is the core supporting all upper-level functions. Due to the complexity of multiple stages such as data production, cross-departmental data transmission, and data application, there are often inconsistencies between the original information in the database and the information finally displayed on the user's end, which are difficult to detect and correct in a timely manner. Summary of the Invention

[0003] This disclosure provides an end-to-end map engine evaluation method, apparatus, and electronic device.

[0004] According to one aspect of this disclosure, an end-to-end map engine evaluation method is provided, comprising:

[0005] Obtain assessment instructions, wherein the assessment instructions are used to instruct the assessment task and the target service area to be assessed on the map; The intelligent agent simulates the execution of the evaluation task in the map engine according to the evaluation instructions; Collect page information of the target service area displayed by the map engine during the task simulation, wherein the page information includes at least the first point of interest (POI) information of the target service area; Obtain the second POI information of the target service area from the map database; The map engine is evaluated based on the first POI information and the second POI information to obtain the evaluation result.

[0006] According to another aspect of this disclosure, an end-to-end map engine evaluation apparatus is provided, comprising: The first acquisition module is used to acquire assessment instructions, wherein the assessment instructions are used to indicate the assessment task and the target service area to be assessed in the map. An execution module is used to simulate the execution of the evaluation task in the map engine by an intelligent agent according to the evaluation instructions; The data acquisition module is used to acquire page information of the target service area displayed by the map engine during the task simulation process, wherein the page information includes at least the first point of interest (POI) information of the target service area. The second acquisition module is used to acquire the second POI information of the target service area from the map database; The evaluation module is used to evaluate the map engine based on the first POI information and the second POI information to obtain the evaluation result.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect embodiment.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect embodiment.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect embodiment.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of an end-to-end map engine evaluation method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of an end-to-end map engine evaluation method provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of an end-to-end map engine evaluation method provided in an embodiment of this disclosure; Figure 4 A logical schematic diagram of an end-to-end map engine evaluation method provided in an embodiment of this disclosure; Figure 5 A structural block diagram of an end-to-end map engine evaluation device provided in an embodiment of this disclosure; Figure 6 A schematic block diagram of an electronic device used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] In existing map navigation processes, when a user approaches a service area, the application interface typically pops up an information panel displaying the facilities within that service area, such as gas stations, charging stations, restrooms, and restaurants. The display logic of this information panel is constrained by a series of complex business rules and spatial relationships, including: the geofence of the service area (parent point), the coordinates of the internal facilities (child points), the operating status of the child points, the tag attributes of the child points, and the parent-child relationship established between the child points and the parent point. The geofence of the service area is a continuous string of coordinates, the operating status of the child points is such as open or temporarily closed, and the attribute tags of the child points are such as transportation facilities or charging stations.

[0014] The business rules stipulate that a child point icon will be displayed normally on the navigation panel only if the child point is in a normal state, the child point label meets the display conditions, the child point and the parent point have established a parent-child relationship, and the child point coordinates are within the parent point's geofence.

[0015] Therefore, any abnormal data processing, transmission, or application, or any inconsistency in logical judgment, can lead to the problem that "the data in the database is correct, but the user cannot see it."

[0016] Existing technologies for improving data display accuracy include two methods: full-field database quality inspection and manual on-site road testing. Full-field database quality inspection involves developing a series of quality inspection strategies during the data production stage to perform logical and consistency checks on all relevant data fields. This only ensures the correctness of the data production level and cannot cover the consistency of data transmission to the application, as well as subsequent stages such as rendering strategies, client compatibility, and network loading. Therefore, it cannot guarantee the end-to-end user display effect. Manual on-site road testing involves dispatching testers to navigate on actual roads, manually observing and recording the display of service area panels. This requires extremely high manpower, time, and financial costs, has limited coverage, and is difficult to execute frequently and on a large scale, making it unsuitable for the rapid iteration pace of development.

[0017] Data processing is the collection, storage, retrieval, processing, transformation, and transmission of data. Its basic purpose is to extract and derive valuable and meaningful data from large amounts of potentially messy and difficult-to-understand data.

[0018] Artificial intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. It attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0019] Map navigation is a software system that provides users with route planning and real-time guidance services based on positioning technology and electronic maps. It achieves route planning services by integrating positioning technology with real-time traffic data.

[0020] An intelligent agent is a proxy that can perceive the environment and take actions to achieve specific goals. It can be software, hardware or a system, and has autonomy, adaptability and interactivity. By perceiving changes in the environment, it makes judgments and decisions based on its learned knowledge and algorithms, and then performs actions to influence the environment or achieve predetermined goals.

[0021] Figure 1 This is a schematic diagram illustrating an end-to-end map engine evaluation method provided in an embodiment of this disclosure. Figure 1 As shown, the method includes the following steps: S101, Obtain evaluation instructions.

[0022] In this embodiment, the evaluation instruction is used to indicate the evaluation task and the target service area to be evaluated in the map, wherein the target service area to be evaluated can be one or more.

[0023] Optionally, the evaluation tasks may include navigation page evaluation tasks and retrieval page evaluation tasks; the navigation page evaluation task analyzes the panels during the map engine's navigation process, and the retrieval page evaluation task analyzes the panels during the map engine's content retrieval.

[0024] In some embodiments, service areas can be randomly selected from the map POI database, or newly created / modified service areas can be prioritized as target service areas to be evaluated.

[0025] Understandably, service areas include corresponding associated sub-points of interest (POIs), such as charging stations, gas stations, restrooms, and restaurants within the service area.

[0026] S102, through an intelligent agent, simulates the execution of the evaluation task in the map engine according to the evaluation instructions.

[0027] Optionally, the intelligent agent in this embodiment can at least simulate the execution of the navigation page evaluation task and the retrieval page evaluation task.

[0028] In some embodiments, the agent can simulate map navigation using a map engine, thereby analyzing the navigation page during the map navigation simulation process to execute the evaluation task in the evaluation instruction.

[0029] In some embodiments, the agent can also simulate map retrieval using a map engine, thereby analyzing the retrieval pages during the map retrieval simulation process to execute the evaluation task in the evaluation instruction.

[0030] Optionally, in this embodiment, the agent can use an open-source visual understanding large language model as a base and perform supervised fine-tuning on the base model. The purpose of fine-tuning is to enable the model to simulate the execution of the evaluation task, thereby obtaining a user interface (UI) agent specifically for map navigation evaluation.

[0031] S103 collects page information of the target service area displayed by the map engine during the task simulation.

[0032] Optionally, the output of the agent can be obtained. Since the agent analyzes the page panel during the task simulation, the page information of the target service area can be parsed from the output of the agent.

[0033] The page information includes at least the first point of interest (POI) information of the target service area. In this embodiment, the POI is a facility in the service area.

[0034] In some embodiments, the first POI information of the target service area may include the current target service area's identification (ID), target service area name, and border coordinates.

[0035] In some embodiments, the first POI information of the target service area may further include facility POI information of the current target service area, such as information related to charging stations, gas stations, toilets, and restaurants; optionally, the facility POI information in this embodiment includes, but is not limited to, facility ID, facility type, facility coordinates, facility status, facility tag, and relationship with the target service area.

[0036] S104. Obtain the second POI information of the target service area from the map database.

[0037] It is understandable that the second POI information of the target service area is the standard information corresponding to the target service area. The second POI information includes, but is not limited to, the target service area ID, the target service area name, the target service area border coordinates, the target service area guide link, and the facility ID information belonging to the target service area, such as the facility ID, facility type, facility coordinates, facility status, facility tag, and relationship with the target service area.

[0038] S105. The map engine is evaluated based on the first POI information and the second POI information to obtain the evaluation result.

[0039] It is understandable that the first POI information is the page information corresponding to the target service area obtained by the map engine through simulation, while the second POI information is the standard information in the map database. Therefore, the first POI information and the second POI information can be compared to evaluate the map engine.

[0040] Optionally, matching first POI information and second POI information can be determined based on the target service area ID. The facility information leading to the target service area involved in the first POI information and second POI information is compared to determine whether the identified first POI information is consistent with the standard second POI information. If the first POI information contains POIs that are not present in the second POI information, then more POIs are displayed in the current map engine; if the first POI information does not contain POIs that are present in the second POI information, then fewer POIs are displayed in the current map engine.

[0041] In this embodiment, a trained agent simulates the evaluation tasks in the evaluation instructions, replacing manual inspection and significantly reducing labor costs. The agent collects page information of the target service area displayed during the simulation, compares the first POI information in the page information with the standard second POI information in the map database, and determines the evaluation result of the map engine. This improves the evaluation accuracy and reliability of map navigation POI display quality. The use of an agent to simulate task execution greatly improves the overall efficiency and automation of map service area information evaluation.

[0042] Based on the above embodiments, Figure 2 This is a schematic diagram illustrating an end-to-end map engine evaluation method provided in an embodiment of this disclosure. Figure 2 As shown, the method includes the following steps: S201, Obtain evaluation instructions.

[0043] In this embodiment of the disclosure, the method for implementing step S201 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0044] S202, in response to the evaluation task being a navigation page evaluation task, the agent performs map navigation simulation in the map engine according to the evaluation instructions to execute the navigation page evaluation task.

[0045] In some embodiments, a navigation path corresponding to the map navigation simulation can be determined based on the target service area; optionally, a guide link for the target service area can be obtained from a map database, the guide link being a road segment that instructs the vehicle to enter the target service area; the area is extended based on the guide link to obtain a first extended area and a second extended area, wherein each extended area includes at least one POI; feature POIs are determined in the first extended area and the second extended area respectively as the start and end points of the navigation path to improve the realism of the simulated navigation process.

[0046] Optionally, the guide link can be used as an anchor point to extend forward and backward by a preset distance to obtain a first extension point and a second extension point; regions can be constructed with the first extension point and the second extension point as the center to obtain a first extended region and a second extended region, which is more suitable for real road driving scenarios.

[0047] For example, a guide link for the target service area is determined, denoted as guide_link. Using guide_link as the anchor point, the guide link extends forward and backward by a length L to obtain a first extension point and a second extension point. Using the coordinates of the first extension point and the second extension point as the center, an extension area with a radius of R is calculated to obtain a first extension area and a second extension area. A point of interest (POI) is selected from each of the first and second extension areas. The location of the POI is used as the start point and the end point, thereby obtaining a navigation path based on the start point and the end point. Here, L and R are preset values, and the settings of L and R must at least satisfy that each extension area includes at least one POI.

[0048] Furthermore, map navigation simulation can be performed based on the navigation path, and screenshots of the navigation page output by the map engine at different vehicle locations can be taken during the map navigation simulation.

[0049] Optionally, a simulated navigation service can be invoked to start a map navigation simulation based on the start and end points of the navigation path, and generate a navigation session handle for the current map navigation simulation. The current simulated navigation process can be bound to the navigation session handle to take screenshots of the navigation pages output by the map engine at different vehicle locations.

[0050] Understandably, the navigation session handle is the unique identifier of the current map navigation simulation task, preventing interference between different routes, vehicle locations, and interface data.

[0051] Furthermore, the intelligent agent analyzes the page screenshots until the navigation page evaluation task is completed, in order to obtain the page information of the target service area.

[0052] Optionally, when training the agent, a high-quality training dataset is first constructed. In this embodiment, the agent is mainly used to implement the navigation page evaluation task and the map retrieval page evaluation task. The training dataset is manually labeled in the form of multi-turn text and image dialogue.

[0053] Optionally, for the navigation page evaluation task, the manually labeled user instruction can be: "My starting point is [location A], my destination is [location B], and I will pass through [XX service area]. Please check the charging stations, gas stations, restrooms, and restaurants in this service area and output a formatted conclusion.", denoted as Task1_Input_i; The agent training objective is: the agent needs to perform a series of operations such as "start the navigation application (APP) → set the starting point and destination → start simulated navigation → during the navigation process, periodically (e.g., every 1 kilometer or every 30 seconds) check the current interface to detect whether the information panel of the target service area appears → if it appears, parse all the sub-point icons and text information in the panel → record and format the output → end the navigation".

[0054] Each round of dialogue data in the dataset includes: Input: Screenshot of the current page, user command, and summary information of the current execution status, where the summary information of the current execution status refers to the summary information of the results after execution in previous rounds.

[0055] Output: The next atomic operation that the model should execute. The set of atomic operations can be predefined in the agent's system instructions, including but not limited to the following operations: do(action="Launch", app="xxx"): Launch the application.

[0056] do(action="Tap", element=[x,y]): Clicks the screen at coordinates (x, y).

[0057] do(action="Input", text="xxx"): Input text.

[0058] do(action="Wait", duration="x seconds"): Wait for loading.

[0059] finish(message="xxx"): The task is completed, and a conclusion is attached.

[0060] In this embodiment, the agent connects to the navigation session handle to obtain vehicle location information and captures screenshots of different locations. The task instruction loop process of the agent during training is as follows: System instruction input: The system prompts, user task instructions, and summary information of the current execution status are used as the initial text input for the agent. The initial image input for the agent is the current page screenshot. The system prompts are used to indicate the predefined set of operations that the agent can use.

[0061] Model Inference: Based on the input page screenshot and user task instructions, the agent outputs the next operation instruction At. The internal decision-making process can be decomposed into thought chain reasoning and formatted action generation. Thought chain reasoning: Thought_t = LLM_Reasoning(“Based on the summary information and the current page screenshot, my next step should be…”), and formatted action generation: Action_t = LLM_Format(Thought_t, allowed action formats). In this embodiment, the output format of the model corresponding to the agent is forced to be: <think> Thought_t< / think> \n <action> A_t< / action> The newline character (\n) separates the thought process and the action instructions into two lines. <think> Thought_t< / think> The "think" section contains the reasoning text within the "think" tag. <action> A_t< / action> This is the action area, where the final determined atomic operations are written into the label.

[0062] Action execution: The automation framework parses At. If it is a do-type operation, it controls the simulator to perform the corresponding click, swipe, or other operation and proceed to the next step; if it is a finish, it outputs the task completion instruction.

[0063] The state capture and loop process works as follows: after executing At, wait for T seconds for the interface to refresh, then capture a new page screenshot, update the current execution state's summary information with the new page screenshot and the At summary, and use the updated summary information and the new page screenshot as input for the next round of reasoning and action execution until finish is obtained.

[0064] Optionally, in this embodiment, T can be set to 2, that is, after executing At, wait 2 seconds for the interface to refresh.

[0065] Furthermore, during the training phase of the agent, the agent model is trained using a supervised fine-tuning method on a constructed training dataset. Each training sample includes a page screenshot, summary information, and the next action to be executed. The loss function adopts the standard cross-entropy loss based on the prediction of the next token. Through fine-tuning training, the agent learns to predict the correct next action given a task instruction and screen context.

[0066] In other words, in this embodiment, the trained agent outputs the next round of atomic operation instructions based on the evaluation instructions, page screenshots, and historical operation records; executes the atomic operation instructions until the task completion instruction is output; the historical operation records are the atomic operation records of the agent in the historical dialogue rounds, and the atomic operation instructions are any one instruction in the atomic operation set. All operations are completed based on the agent, saving human resources and costs.

[0067] In some embodiments, after the agent training is completed, the agent performance can be evaluated based on a reserved test set. Evaluation metrics may include, but are not limited to, task completion rate, operation sequence accuracy, and single-step action accuracy. The test set will only be deployed and used if the evaluation metrics on the agent meet the accuracy requirements, so as to improve the accuracy of the map engine evaluation.

[0068] S203, in response to the evaluation task being a retrieval page evaluation task, the agent performs map retrieval simulation in the map engine according to the evaluation instructions to execute the retrieval page evaluation task.

[0069] Optionally, for the navigation page evaluation task, the manually labeled user instruction can be: "Search for [XX service area] on the map, and view the service area's name, address information, and its internal charging stations, gas stations, toilets, and restaurant information, and output a formatted conclusion." This is denoted as Task2_Input_i. The agent's training objective is: the agent needs to perform a series of operations such as "launch the map app → enter the service area name in the search box → enter the search results page → click to enter the service area details page → browse the 'internal facilities' or similar tabs on the details page → view and record the information of each sub-point → output in a formatted manner."

[0070] Understandably, the agent can be trained on a manually labeled training dataset. After training, when the agent receives an evaluation task such as a page retrieval evaluation task, it can search for the target service area in the map engine, obtain the search page output by the map engine, and take a screenshot of the search page. The screenshot is then analyzed until the page retrieval evaluation task is completed, thereby obtaining the page information of the target service area and improving the efficiency and accuracy of page information acquisition.

[0071] It is understandable that the agent can output the next round of atomic operation instructions based on the evaluation instructions, page screenshots and historical operation records; execute the atomic operation instructions until the task completion instruction is output. The specific training method can be implemented using the same method as in step S202, and will not be repeated here.

[0072] S204 collects page information of the target service area displayed by the map engine during the task simulation.

[0073] In this embodiment of the disclosure, the method for implementing step S204 can be implemented in any of the various embodiments of this disclosure, and no limitation is made here, nor will it be described in detail.

[0074] S205: Obtain the second POI information of the target service area from the map database.

[0075] In this embodiment of the disclosure, the method for implementing step S205 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0076] S206. The map engine is evaluated based on the first POI information and the second POI information to obtain the evaluation result.

[0077] In this embodiment of the disclosure, the method for implementing step S206 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0078] In this embodiment, the navigation page evaluation task and the retrieval page evaluation task are determined in the evaluation instructions. The intelligent agent simulates the execution of different evaluation tasks. In the simulated execution of the navigation page evaluation task, the navigation path is constructed with the guidance link of the target service area as the anchor point, which can adapt to real road driving scenarios. In the retrieval page evaluation task, different retrieval tasks are simulated according to user instructions, replacing manual inspection and greatly reducing labor costs. The page screenshots during the simulation process are analyzed to obtain the corresponding page information more accurately. The evaluation result of the map engine is determined by comparing the first POI information in the page information with the standard second POI information in the map database, thereby improving the evaluation accuracy and reliability of the map navigation POI display quality.

[0079] Based on the above embodiments, Figure 3 This is a schematic diagram illustrating an end-to-end map engine evaluation method provided in an embodiment of this disclosure. Figure 3 As shown, the method includes the following steps: S301, Obtain evaluation instructions.

[0080] In this embodiment of the disclosure, the method for implementing step S301 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0081] S302, in response to the evaluation task being a navigation page evaluation task, the agent performs map navigation simulation in the map engine according to the evaluation instructions to execute the navigation page evaluation task.

[0082] In this embodiment of the disclosure, the method for implementing step S302 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0083] S303, in response to the evaluation task being a retrieval page evaluation task, the agent performs map retrieval simulation in the map engine according to the evaluation instructions to execute the retrieval page evaluation task.

[0084] In this embodiment of the disclosure, the method for implementing step S303 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0085] S304, in response to the task completion instruction of the intelligent agent, obtains the information set corresponding to the task completion instruction.

[0086] Optionally, the agent may output a set of information along with the task completion instruction. This set of information may include all the information parsed from the page by the agent when simulating the evaluation task.

[0087] S305, parse the information set to obtain the page information of the target service area.

[0088] Furthermore, the information set is parsed to obtain the page information of the target service area. For example, the output page information is: {"Service Area Name": "XX", "Panel Appears": true, "Facilities": {"Charging Station": 1, "Gas Station": 1, "Toilet": 0}}.

[0089] It is understandable that the service area information and related facilities information in the service area are obtained from the first POI information of the target service area from the page screenshot in order to obtain accurate page information.

[0090] S306: Obtain the second POI information of the target service area from the map database.

[0091] In this embodiment of the disclosure, the method for implementing step S306 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0092] S307. The map engine is evaluated based on the first POI information and the second POI information to obtain the evaluation result.

[0093] Optionally, the POIs in the first POI information and the second POI information can be compared; it is understood that the first POI information and the second POI information are POI information corresponding to the same target service area; in some embodiments, when multiple target service areas to be evaluated appear at one time in the evaluation task, the first POI information and the second POI information corresponding to the target service area are obtained according to the name or ID of the target service area, so as to make a precise comparison to determine whether the POIs in the first POI information and the second POI information are completely consistent, that is, whether the facilities appearing in the first POI information and the second POI information are consistent.

[0094] In response to the inconsistency between the POIs in the first POI information and the second POI information, the inconsistent target POI is obtained; in response to the target POI belonging to the first POI information, the evaluation result is determined to be an anomaly of multiple POI displays; in response to the target POI belonging to the second POI information, the evaluation result is determined to be an anomaly of missing POI displays, thereby improving the evaluation accuracy and reliability of the POI display quality of map navigation.

[0095] For example, each sub-point that is in a normal state in the second POI information, whose label meets the display conditions, whose coordinates are within the parent point's border, and which has a relationship with the parent point can be analyzed. The parent point is the target service area. If the count of a certain type of facility in the first POI information is greater than the number of sub-points that meet the conditions in the second POI information, it is marked as a sub-point over-display anomaly, which is also known as a POI over-display anomaly. This may be caused by data errors or cache errors. If the count of a certain type of facility is not found to increase in the first POI information, it is marked as a sub-point under-display anomaly, which is also known as a POI under-display anomaly.

[0096] Furthermore, an evaluation report can be generated based on the evaluation results. This report includes all detected inconsistencies, such as "a 'toilet' sub-point exists in the map database and is in normal condition, but is not displayed in the navigation panel." Issues in the evaluation report are marked and automatically transferred to a manual processing platform for manual verification and repair, ensuring a good user experience for the map engine.

[0097] In this embodiment, navigation page evaluation tasks and retrieval page evaluation tasks are determined in the evaluation instructions. These tasks are simulated and executed by an intelligent agent. In the simulated execution of the navigation page evaluation task, a navigation path is constructed using the guidance link of the target service area as an anchor point, adapting to real-world road driving scenarios. In the retrieval page evaluation task, different retrieval tasks are simulated according to user instructions, replacing manual inspection and significantly reducing labor costs. Analysis of page screenshots during the simulation process allows for more accurate acquisition of corresponding page information. A comparison is made between the first POI information in the page information and the standard second POI information in the map database to identify inconsistent POIs. Based on the attribution of these inconsistent POIs, it is determined whether the current issue is an anomaly of too many or too few POIs. An evaluation report is generated based on the evaluation results for verification and repair, improving the evaluation accuracy and reliability of map navigation POI display quality.

[0098] Figure 4 This is a logical schematic diagram of an end-to-end map engine evaluation method provided in this embodiment. The method involves determining the second POI information of the target service area from the map database, invoking a pre-trained agent to simulate the evaluation task, which includes a navigation page evaluation task and a retrieval page evaluation task. The navigation page evaluation task analyzes the navigation panel, while the retrieval page evaluation task analyzes the map retrieval details. The execution flow of the navigation page evaluation task is: launch the APP → set the route → start navigation → periodically take screenshots of the panel → parse the panel information → end. The execution flow of the retrieval page evaluation task is: launch the APP → search for the service area → enter the details page → browse sub-point information → end. The results output by the agent are analyzed and summarized to obtain the first POI information. The first POI information and the second POI information are compared, and an evaluation report is output for manual verification and correction. This embodiment can directly simulate the user's operation path, with fully automated processing without manual intervention, greatly saving the manpower and time costs of traditional road testing. It can also be extended to other complex UI interaction and experience evaluation scenarios, such as traffic light countdown recognition and along-the-way search recommendations. Only different task datasets need to be constructed and fine-tuned during agent training, making it more flexible.

[0099] Figure 5 This is a structural block diagram of an end-to-end map engine evaluation device provided in an embodiment of this disclosure. Figure 5 As shown, the end-to-end map engine evaluation device 500 includes: The first acquisition module 501 is used to acquire assessment instructions, wherein the assessment instructions are used to indicate the assessment task and the target service area to be assessed in the map. The execution module 502 is used to simulate the execution of evaluation tasks in the map engine by an intelligent agent according to evaluation instructions; The acquisition module 503 is used to acquire page information of the target service area displayed by the map engine during the task simulation process, wherein the page information includes at least the first point of interest (POI) information of the target service area. The second acquisition module 504 is used to acquire the second POI information of the target service area from the map database; The evaluation module 505 is used to evaluate the map engine based on the first POI information and the second POI information to obtain the evaluation result.

[0100] In some implementations, the evaluation task includes at least one of a navigation page evaluation task and a retrieval page evaluation task, and execution module 502 is used for: In response to the evaluation task being a navigation page evaluation task, the agent performs map navigation simulation in the map engine according to the evaluation instructions to execute the navigation page evaluation task; In response to the evaluation task being a page retrieval evaluation task, the agent performs map retrieval simulation in the map engine according to the evaluation instructions to execute the page retrieval evaluation task.

[0101] In some implementations, module 502 is used to: Based on the target service area, determine the corresponding navigation path for map navigation simulation; The map navigation simulation is performed based on the navigation route, and screenshots of the navigation page output by the map engine at different vehicle locations are taken during the map navigation simulation. The intelligent agent analyzes the page screenshots until the navigation page evaluation task is completed, so as to obtain the page information of the target service area.

[0102] In some implementations, module 502 is used to: Obtain the guidance links for the target service area from the map database. The guidance links are road segments that direct vehicles to the target service area. Based on the boot link, the region is extended to obtain a first extended region and a second extended region, wherein each extended region includes at least one POI; Featured Points of Interest (POIs) are identified in the first and second extended regions, respectively, as the start and end points of the navigation path.

[0103] In some implementations, module 502 is used to: Using the guide link as an anchor point, extend it forward and backward by a preset distance to obtain the first extension point and the second extension point. The regions are constructed with the first extension point and the second extension point as the center, respectively, to obtain the first extended region and the second extended region.

[0104] In some implementations, module 502 is used to: Invoke the simulated navigation service to start a map navigation simulation based on the start and end points of the navigation path, and generate a navigation session handle for the current map navigation simulation; The navigation session handle is bound to the current simulated navigation process to capture screenshots of the navigation pages output by the map engine at different vehicle locations.

[0105] In some implementations, module 502 is used to: The intelligent agent searches for the target service area in the map engine, obtains the search page output by the map engine, and takes a screenshot of the search page. Analyze the page screenshots until the page retrieval evaluation task is completed to obtain the page information of the target service area.

[0106] In some implementations, module 502 is used to: The agent outputs the next round of atomic operation instructions based on the evaluation instructions, page screenshots, and historical operation records; Execute atomic operation instructions until a task completion instruction is output; The historical operation record is a record of the atomic operations performed by the agent in each round of the historical dialogue.

[0107] In some implementations, the acquisition module 503 is used for: In response to the task completion instruction from the intelligent agent, obtain the information set corresponding to the task completion instruction; The information set is parsed to obtain the page information of the target service area.

[0108] In some implementations, the evaluation module 505 is used for: Compare the POIs in the first POI information and the second POI information; In response to the fact that the POIs in the first POI information and the second POI information are not completely consistent, the inconsistent target POIs are obtained; In response to the target POI belonging to the first POI information, the evaluation result is determined to be an anomaly in POI display; In response to the target POI belonging to the second POI information, the evaluation result is determined to be an anomaly of POI missing display.

[0109] In this embodiment, navigation page evaluation tasks and retrieval page evaluation tasks are determined in the evaluation instructions. These tasks are simulated and executed by an intelligent agent. In the simulated execution of the navigation page evaluation task, a navigation path is constructed using the guidance link of the target service area as an anchor point, adapting to real-world road driving scenarios. In the retrieval page evaluation task, different retrieval tasks are simulated according to user instructions, replacing manual inspection and significantly reducing labor costs. Analysis of page screenshots during the simulation process allows for more accurate acquisition of corresponding page information. A comparison is made between the first POI information in the page information and the standard second POI information in the map database to identify inconsistent POIs. Based on the attribution of these inconsistent POIs, it is determined whether the current issue is an anomaly of too many or too few POIs. An evaluation report is generated based on the evaluation results for verification and repair, improving the evaluation accuracy and reliability of map navigation POI display quality.

[0110] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0112] Figure 6 A schematic block diagram of an electronic device for implementing embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0113] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0114] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as an end-to-end map engine evaluation method. For example, in some embodiments, the end-to-end map engine evaluation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the end-to-end map engine evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform an end-to-end map engine evaluation method by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0121] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0122] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An end-to-end map engine evaluation method, wherein, The method includes: Obtain assessment instructions, wherein the assessment instructions are used to instruct the assessment task and the target service area to be assessed on the map; The intelligent agent simulates the execution of the evaluation task in the map engine according to the evaluation instructions; Collect page information of the target service area displayed by the map engine during the task simulation, wherein the page information includes at least the first point of interest (POI) information of the target service area; Obtain the second POI information of the target service area from the map database; The map engine is evaluated based on the first POI information and the second POI information to obtain the evaluation result.

2. The method according to claim 1, wherein, The evaluation task includes at least one of a navigation page evaluation task and a retrieval page evaluation task. The step of simulating the execution of the evaluation task in the map engine by an agent according to the evaluation instructions includes: In response to the evaluation task being a navigation page evaluation task, the agent performs map navigation simulation in the map engine according to the evaluation instructions to execute the navigation page evaluation task; In response to the evaluation task being a page retrieval evaluation task, the agent performs a map retrieval simulation in the map engine according to the evaluation instructions to execute the page retrieval evaluation task.

3. The method according to claim 2, wherein, The step of performing map navigation simulation in the map engine by an intelligent agent according to the evaluation instructions to execute the navigation page evaluation task includes: Based on the target service area, determine the navigation path corresponding to the map navigation simulation; Map navigation simulation is performed based on the navigation path, and screenshots of the navigation page output by the map engine at different vehicle locations are taken during the map navigation simulation. The intelligent agent analyzes the page screenshots until the navigation page evaluation task is completed, so as to obtain the page information of the target service area.

4. The method according to claim 3, wherein, The step of determining the navigation path corresponding to the map navigation simulation based on the target service area includes: Obtain the guidance link of the target service area from the map database. The guidance link is a road segment that instructs vehicles to enter the target service area. Based on the guiding link, the region is extended to obtain a first extended region and a second extended region, wherein each extended region includes at least one POI; Featured Points of Interest (POIs) are determined in the first and second extended regions, respectively, as the start and end points of the navigation path.

5. The method according to claim 4, wherein, The process of extending the region based on the guiding link to obtain a first extended region and a second extended region includes: Using the guide link as an anchor point, extend it forward and backward by a preset distance to obtain a first extension point and a second extension point. Regions are constructed with the first extension point and the second extension point as centers, respectively, to obtain the first extended region and the second extended region.

6. The method according to any one of claims 3-5, wherein, The step of taking screenshots of the navigation page output by the map engine at different vehicle locations during the map navigation simulation includes: Invoke the simulated navigation service, start the map navigation simulation based on the start and end points of the navigation path, and generate the navigation session handle for the current map navigation simulation; The navigation session handle is bound to the current simulated navigation process to take screenshots of the navigation pages output by the map engine at different vehicle locations.

7. The method according to claim 2, wherein, The step of simulating map retrieval in the map engine by an intelligent agent according to the evaluation instructions to execute the retrieval page evaluation task includes: The intelligent agent searches for the target service area in the map engine, obtains the search page output by the map engine, and takes a screenshot of the search page. The page screenshots are analyzed until the page evaluation task is completed, so as to obtain the page information of the target service area.

8. The method according to claim 3 or 7, wherein, The page screenshots are analyzed until the page evaluation task is completed to obtain the page information of the target service area, including: The intelligent agent outputs the next round of atomic operation instructions based on the evaluation instructions, the page screenshot, and the historical operation records; Execute atomic operation instructions until a task completion instruction is output; The historical operation record refers to the atomic operation record of the agent in the historical dialogue rounds.

9. The method according to claim 1, wherein, The process of collecting page information of the target service area displayed by the map engine during the task simulation includes: In response to the task completion instruction of the intelligent agent, obtain the information set corresponding to the task completion instruction; The information set is parsed to obtain the page information of the target service area.

10. The method according to claim 1, wherein, The step of evaluating the map engine based on the first POI information and the second POI information to obtain the evaluation result includes: Compare the POIs in the first POI information and the second POI information; In response to the fact that the POIs in the first POI information and the second POI information are not completely consistent, the inconsistent target POIs are obtained; In response to the fact that the target POI belongs to the first POI information, the evaluation result is determined to be an anomaly of multiple POI displays; In response to the target POI belonging to the second POI information, the evaluation result is determined to be an anomaly in POI display.

11. An end-to-end map engine evaluation apparatus, comprising: The first acquisition module is used to acquire assessment instructions, wherein the assessment instructions are used to indicate the assessment task and the target service area to be assessed in the map. An execution module is used to simulate the execution of the evaluation task in the map engine by an intelligent agent according to the evaluation instructions; The data acquisition module is used to acquire page information of the target service area displayed by the map engine during the task simulation process, wherein the page information includes at least the first point of interest (POI) information of the target service area. The second acquisition module is used to acquire the second POI information of the target service area from the map database; The evaluation module is used to evaluate the map engine based on the first POI information and the second POI information to obtain the evaluation result.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

Images