Map data validity verification method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a map data validity verification method and device, electronic equipment and storage medium, which relates to the technical field of artificial intelligence such as generative large language model, map data update, validity verification, supervised training and reinforcement learning. The method comprises: obtaining task description information of a validity verification task initiated for map data; processing the task description information by using a preset validity verification model to obtain a returned validity verification result, the validity verification model being a model constructed based on automatic machine learning and trained by taking a generative large language model as an underlying core, and a training process of the validity verification model comprising: fine-tuning training based on historical manual verification job logs collected from a map application as supervised data; and determining target map data that passes the validity verification according to the validity verification result. The method can realize large-scale automatic production of map data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of task processing, and to the fields of artificial intelligence such as generative large language models, map data updating, validity verification, supervised training, reinforcement learning, and the like, and in particular to a map data validity verification method and device, electronic equipment, computer readable storage medium, and computer program product. BACKGROUND

[0002] Large language models (LLM, Large Language Model, which is essentially a generative model, also referred to as generative large language model) have shown strong NLP (Natural Language Processing) understanding and generation capabilities.

[0003] Map data, as a core element in the fields of intelligent transportation, urban planning, route navigation, etc., its accuracy and real-time performance play a crucial role in improving the performance and user experience of various applications. However, traditional map data production mainly relies on manual work, including data collection, editing, auditing, etc. This approach has many shortcomings: 1) low efficiency, difficult to meet the requirements of large-scale data updating and real-time performance; 2) high staff turnover, labeling quality difficult to control, prone to errors, affecting data quality; 3) high cost, especially for more complex data production tasks, such as which information is valid and how to verify its validity. SUMMARY

[0004] The present disclosure provides a map data validity verification method, device, electronic equipment, computer readable storage medium, and computer program product.

[0005] In a first aspect, the present disclosure provides a map data validity verification method, comprising: obtaining task description information of a validity verification task initiated for map data; processing the task description information using a preset validity verification model to obtain a returned validity verification result; wherein the validity verification model is a model constructed based on automatic machine learning and trained using a generative large language model as the underlying core; the training process for obtaining the validity verification model includes fine-tuning based on historical manual verification job logs collected from map applications as supervised data; and determining target map data that passes the validity verification based on the validity verification result.

[0006] In a second aspect, the embodiments of the present disclosure provide a map data validity verification device, which comprises: a task description information acquisition unit configured to acquire task description information of a validity verification task initiated for map data; a validity verification model processing unit configured to process the task description information by using a preset validity verification model to obtain a returned validity verification result; wherein the validity verification model is a model trained by a framework constructed based on automatic machine learning with a generative large language model as a bottom core; a training process of the validity verification model comprises fine-tuning training based on historical manual verification job logs collected from a map application as supervised data; and a target map data determination unit configured to determine target map data that passes the validity verification according to the validity verification result.

[0007] In a third aspect, the embodiments of the present disclosure provide an electronic device, which comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the map data validity verification method as described in the first aspect when executed.

[0008] In a fourth aspect, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the map data validity verification method as described in the first aspect when executed.

[0009] In a fifth aspect, the embodiments of the present disclosure provide a computer program product comprising a computer program, which is used to enable a processor to implement the steps of the map data validity verification method as described in the first aspect when executed.

[0010] The map data validity verification scheme provided by the present disclosure creatively introduces a validity verification model trained by a framework constructed based on automatic machine learning with a generative large language model as a core to process task description information containing corresponding problem contents, so as to enable the model to accurately understand and determine a validity verification result corresponding to the task description information by virtue of its core and framework, thereby automatically completing a validity verification task for map data, liberating manual verification from tedious manual verification operations, improving validity verification efficiency and accuracy, and further improving the use experience of map applications.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments thereof as taken in conjunction with the accompanying drawings:

[0013] Figure 1 is an exemplary system architecture to which the present disclosure can be applied;

[0014] Figure 2 a flowchart of a map data validity verification method provided by an embodiment of the present disclosure;

[0015] Figure 3 a flowchart of a training process for training an effectiveness verification model provided by an embodiment of the present disclosure;

[0016] Figure 4 a schematic diagram of various mechanisms included in a framework constructed based on an automatic machine learning method provided by an embodiment of the present disclosure;

[0017] Figure 5 a structural block diagram of a map data validity verification apparatus provided by an embodiment of the present disclosure;

[0018] Figure 6 a structural schematic diagram of an electronic device suitable for executing a map data validity verification method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only and not limiting. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0020] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0021] Figure 1 An exemplary system architecture 100 to which embodiments of the map data validity verification method, apparatus, electronic device and computer readable storage medium of the present disclosure can be applied is shown.

[0022] As Figure 1As shown, the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0023] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 and the server 105 can be installed with various applications for realizing information communication between them, such as map data validity verification applications, model training applications, data transmission applications, etc.

[0024] The terminal devices 101, 102, 103 and the server 105 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices, and can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here.

[0025] The server 105 can provide various services through various built-in applications. Taking a map data validity verification application that can provide a validity verification service based on map data as an example, the server 105 can achieve the following effects when running the map data validity verification application: First, the server 105 receives, through the network 104, task description information of a validity verification task initiated by the terminal devices 101, 102, 103 for map data; then, the server 105 processes the task description information using a preset validity verification model to obtain a returned validity verification result, the validity verification model is a model constructed based on an automatic machine learning method and trained based on a generative large language model as a bottom core; the training process of the validity verification model includes fine-tuning based on historical manual verification job logs collected from map applications as supervised data; finally, the server 105 determines target map data that passes the validity verification based on the validity verification result.

[0026] Further, the server 105 can also use the determined target map data to update the corresponding part of the content in the map application, so as to improve the accuracy of the content in the map application.

[0027] It should be noted that the task description information can be pre-stored in the server 105 locally in various ways in addition to being obtained from the terminal devices 101, 102, and 103 through the network 104. Therefore, when the server 105 detects that the local has already stored these data (for example, before starting to process the remaining to-be-processed verification tasks), the server 105 can choose to obtain these data directly from the local, and in this case, the example system architecture 100 can also not include the terminal devices 101, 102, and 103 and the network 104.

[0028] Since the corresponding validity verification based on the task description information needs to occupy more computing resources and stronger computing capability, the map data validity verification method provided in each of the subsequent embodiments of the present disclosure is generally executed by the server 105 which has stronger computing capability and more computing resources, and accordingly, the map data validity verification apparatus is generally also arranged in the server 105. However, it should also be noted that when the terminal devices 101, 102, and 103 also have computing capability and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations by the map data validity verification application installed thereon, and then output the same result as the server 105. Especially in the case where there are multiple terminal devices with different computing capabilities, but the map data validity verification application judges that the terminal device where the map data validity verification application is located has stronger computing capability and more remaining computing resources, the terminal device can be allowed to execute the above-mentioned operations, so as to appropriately reduce the computing pressure of the server 105, and accordingly, the map data validity verification apparatus can also be arranged in the terminal devices 101, 102, and 103. In this case, the example system architecture 100 can also not include the server 105 and the network 104.

[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the example system architecture 100 is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0030] Please refer to Figure 2 , Figure 2 A flowchart of a map data validity verification method provided by an embodiment of the present disclosure is shown in FIG. 2, where the flow 200 includes the following steps:

[0031] Step 201: Obtain task description information of a validity verification task initiated for map data;

[0032] The step is designed to obtain task description information of a validity verification task initiated for map data by an execution subject of the map data validity verification method (for example, the server 105 shown in Figure 1 The task description information is input by various terminal devices, such as the terminal devices 101, 102, and 103 shown in Figure 1 Since the task description information should be able to know how to perform the validity verification task, the task description information should at least include, in general, a target point of interest to be verified (for example, which scenic spot, which institution, which commercial center, or which store), a verification question for the target point of interest (for example, whether the relevant information recorded at present, including business hours, business scope, and opening objects, is correct), and of course, in addition to the target point of interest and the verification question, the task description information can also contain other additional task requirements, such as a limited search, a number of information channels for verification, a type, a required accuracy, a time limit for completing verification, and the like.

[0033] It should be noted that the task description information can originally be in a text form, or can be converted from an image, voice, or other information form into a text form that is convenient for model recognition and processing.

[0034] Step 202: processing the task description information by using a preset validity verification model to obtain returned validity verification results;

[0035] On the basis of step 201, the step is designed to process the task description information by using a preset validity verification model by the above-mentioned execution subject to obtain returned validity verification results. The validity verification model is a model trained by using a generative large language model as a bottom core and constructing a framework based on automatic machine learning (for example, AutoGPT, which is an English full name: Auto Generative Pre-trained Transformer, and the corresponding Chinese name is an automatic generation pre-training transformer). The training process of the validity verification model includes fine-tuning training based on historical manual verification job logs collected from a map application as supervised data.

[0036] Therefore, the present step is actually to input the task description information as input data into the validity verification model, so as to have the ability to understand the task requirements in the task description information and implement the task requirements through the validity verification model after at least supervised fine-tuning (SFT). Further, the core generative large language model can be selected as ChatGPT or similar models, and the framework constructed based on automatic machine learning can set various related strategies or mechanisms according to actual needs, such as task understanding, splitting, reasoning mechanism based on chain-of-thought (COT) or prompt chain, calling mechanism based on external tools, and the like.

[0037] Step 203: determining target map data passing the validity verification according to the validity verification result.

[0038] On the basis of step 202, the present step aims to determine the target map data passing the validity verification according to the validity verification result by the above-mentioned execution subject. That is, the validity verification result should be divided into two categories in general, i.e., passing the validity verification and not passing the validity verification, so the present step is to determine the map data belonging to the category passing the validity verification as the target map data. For example, the opening time of a scenic spot in summer is subjected to validity verification, then the opening time period obtained after passing the validity verification should be the target map data. Whether the suspected summer opening time is carried in the original task description information is not important, because even if the suspected summer opening time is carried, it can only provide a reference for subsequent validity verification.

[0039] Further, the above-mentioned execution subject can further update the corresponding content in the map application by using the target map data, so as to replace the suspected map data with the accurate map data obtained after validity verification, and further make the information in the updated map application more accurate, and further improve the use experience.

[0040] The map data validity verification method provided by the embodiments of the present disclosure creatively introduces the validity verification model trained by the framework constructed based on automatic machine learning and taking the generative large language model as the core, to the task description information containing the corresponding problem content, so as to have the ability to accurately understand and determine the validity verification result corresponding to the task description information through the core and framework, thereby automatically completing the validity verification task of the map data, liberating the manual verification operation from the complex manual verification operation, improving the validity verification efficiency and accuracy, and further improving the use experience of the map application.

[0041] To deepen the understanding of how to train the effectiveness verification model through specific training methods, the present embodiment trains the effectiveness verification model through the following steps Figure 3 A training process scheme for training an effectiveness verification model is specifically provided, wherein the process 300 comprises the following steps:

[0042] Step 301: Fine-tuning training based on historical manual verification job logs collected from map applications as supervised data;

[0043] This step is the training scheme mentioned in the previous embodiment step 202, and a specific implementation can be:

[0044] 1) Collect and record the specific production process of human workers;

[0045] 2) Convert the collected data into a training format suitable for LLM, including task understanding and reasoning, task decomposition and planning, action generation, etc.;

[0046] 3) Perform supervised fine-tuning training.

[0047] The purpose of the fine-tuning training provided in this step is to mine the operation steps of human workers when producing specific data (such as which data sources are used, specific decision logic, steps, etc.) and use them as SFT data to enhance the automated data production capability of LLM.

[0048] Step 302: Use the information of high-frequency points of interest with online access times exceeding a preset number of times as standard answers to optimize the model after fine-tuning training in a reinforcement learning manner;

[0049] This step is based on the supervised fine-tuning training of step 301 and attempts to fully utilize the online standard answers of map applications for optimization training. This is because the POI attribute information of high-frequency POIs on the line is relatively accurate, and therefore can be used as a standard answer (also known as Golden answer, yellow gold answer) for optimization training. Specifically, the reward function under reinforcement learning (Reinforcement Learning Reward, also referred to as RL Reward) can be used to guide the optimization training, that is, the difference between the attribute information of the high-frequency POIs collected online and the attribute information automatically produced by the model is used to calculate the reward in the reinforcement learning process.

[0050] That is, for a given POI production task, first, the SFT model is used for automatic data production, and the corresponding attribute information is obtained; then, the reward is calculated according to the RL Reward calculation formula, and the strategy is updated; next, for the case (Case) that the automatic production result and the online attribute information are consistent, the model can be collected for secondary SFT (PseudoSFT) to further strengthen this part of knowledge.

[0051] Step 303: The use feedback information of the user on the model obtained after one-time optimization training is acquired, and the model after one-time optimization training is optimized for secondary optimization training in a reinforcement learning manner.

[0052] This step is based on the one-time optimization training completed in step 302, and attempts to further optimize the model by using the use feedback given by the deployed model in the subsequent user use stage to further enhance the robustness and generalization of the model.

[0053] After the model is deployed for offline production of map data, the online system collects the real feedback of the online user on the point of interest attribute information presented to the user by the map application through the set feedback button (such as the like button and the dislike button), so that the user can evaluate the data accuracy, and also include some content in the comment area that is not satisfied with the corresponding attribute information. Therefore, the model can be further optimized in a reinforcement learning manner by collecting these real feedback data.

[0054] The embodiment provides a model training scheme of sequentially performing three training and optimization by steps 301-303. In some embodiments, the secondary optimization training provided in step 303 can not be included, and the embodiment only exists as a preferred embodiment scheme of simultaneously including two different optimization training.

[0055] In order to better understand the various strategies or mechanisms that the framework constructed based on the automatic machine learning method can include, and how these strategies or mechanisms should be combined with the core generative large language model to finally complete the effectiveness verification, the embodiment further shows Figure 4 a schematic diagram of various mechanisms included in a framework constructed based on an automatic machine learning method, as Figure 4 shown, the framework constructed based on the automatic machine learning method can include:

[0056] 1) An auxiliary decision mechanism based on memory data stored in vector form, which recalls map-related information stored in vector form corresponding to the task description information, and makes decisions corresponding to the task description information based on the map-related information generated by the generative large language model. The map-related information includes images, texts or audio and video data related to points of interest. Further, when recalling the map-related information stored in vector form corresponding to the task description information, the auxiliary decision mechanism is based on at least one of the following recall strategies: a time-sensitive priority recall strategy (i.e. the closer to the current time, the stronger the time sensitivity, the higher the recall priority or the weight at the time of recall), a priority recall strategy based on importance (i.e. the more important, the higher the recall priority or the weight at the time of recall), a priority recall strategy based on the similarity of features with the task description information (i.e. the higher the similarity of the information task description information, the higher the recall priority or the weight at the time of recall). When multiple different recall strategies are included at the same time, the multiple recall strategies can be fused or added in the recall priority or the recall weight.

[0057] 2) A reflection mechanism based on the successful decision cases and failed decision cases in the historical decision results. The reflection mechanism sends the decision process and related historical information corresponding to the successful decision cases and failed decision cases to the generative large language model, so that the generative large language model extracts experience information for assisting subsequent decision making. For example: for the verification of scenic spot business hours, it is assumed that the content provided on website A is often more accurate, and a conclusion can be drawn that website A has high reliability for verifying scenic spot business hours. Then in the future, this conclusion may provide important information for the task of verifying scenic spot business hours.

[0058] That is, the experience information obtained based on the reflection mechanism includes at least one of the following: a success rate sorting of obtaining effective information from different information channels, a time consumption sorting of completing verification in different verification ways, and a proportion sorting of recording error information in different websites. These sorts can actually be summarized as preference information for priority selection among multiple options.

[0059] 3) A task understanding, reasoning, disassembly and planning mechanism based on thought chains.

[0060] 4) A calling mechanism for external tools, which refers to the generative large language model calling corresponding external tools for corresponding queries based on the understanding of the task description information, and continuing to understand or generate validity verification results based on the received query results.

[0061] It should be understood that the embodiments of the present application can be used in the following scenarios: Figure 4The four core mechanisms shown above do not necessarily all appear in the same specific application scenario. The specific mechanisms required should be selected flexibly according to the actual needs in the actual application scenario.

[0062] Although LLMs exhibit strong NLP understanding and generation capabilities, they provide the possibility for automated data production. However, conventional LLMs still have the following defects in use:

[0063] 1) LLMs support a limited number of tokens, making it difficult for them to handle complex map data production tasks;

[0064] 2) The experiences generated (including successful and failed cases) cannot be recorded and summarized in a timely manner, and model updates can only be performed through lagging fine-tuning;

[0065] 3) LLMs are limited by the timeliness of their training sets, making it difficult for them to make timely and accurate decisions when faced with real-time updates and dynamically changing map data;

[0066] 4) Even if LLMs use a Chain-of-Thought (CoT) based approach to task decomposition, it is difficult to ensure the accuracy of the decomposition results. In particular, for some complex scenarios with non-fixed decision-making processes, it is difficult to manually set up a complete and accurate thought chain in advance. This is because LLMs require human-provided large language model prompts (Prompts) to gradually build and improve text prompts to enable LLMs to "understand", and require human feedback and intervention, which is not friendly to complex production tasks.

[0067] To enable LLMs to be applied to map applications for large-scale automatic production of map data, the present embodiment proposes a solution based on the AutoGPT framework - DataGenerationGPT. To enable the model to obtain real-time POI information, handle complex data production tasks, and continuously improve data production capabilities, the following problems and challenges need to be addressed:

[0068] 1) How to solve the problem of limited number of tokens supported by LLMs and the inability to record and summarize experiences (including successful and failed cases) in a timely manner;

[0069] Solution: Build a memory mechanism to allow the model to remember historical decision-making processes.

[0070] 2) How to solve the timeliness problem of LLMs;

[0071] Solution: Give the ability to call various tools.

[0072] 3) How to overcome the limitations of the CoT approach and enable the model to handle complex data production tasks;

[0073] Solution: Let the model perform multi-step autonomous thinking, decision-making, and establish a reflection mechanism.

[0074] Specifically, DataGenerationGPT will combine different production tasks, plan and disassemble the tasks, and call tools (Tools) to obtain real-time information needed to better complete the tasks.

[0075] Specifically, it mainly executes the following steps:

[0076] 1) Production personnel define production tasks: including task objectives, task constraints and limitations, tools that can be called (including Tool name, description, calling parameters, etc.), etc.

[0077] 2) For a given POI, DataGenerationGPT will combine task definitions, call historical information (through the memory mechanism) and the results returned after calling the tools in the previous step, and perform planning (Planning), which includes task understanding and reasoning (Reasoning), task disassembly and planning (Planning), reflection (Reflection), Action generation (whether to call Tool, which Tool to call, Tool calling parameters), and memory update;

[0078] 3) If Tool needs to be called, Tool calling is performed, and the Tool return result is sent to DataGenerationGPT, and steps 2) and 3) are repeated;

[0079] 4) If the Action part shows that the task is completed, or the number of loops exceeds the limit, the task is ended.

[0080] Specific scheme design and implementation steps

[0081] 1. Infrastructure capability building

[0082] Memory construction:

[0083] Memory definition: Use vector database to build memory. Specifically, it will represent each piece of content (pictures, text fragments, etc.) that needs to be saved as a fixed-dimensional vector and store it in the vector database. Each time a decision is made, the most useful information will be recalled to assist the LLM in making decisions.

[0084] Memory retrieval algorithms: When retrieving data from memory, a combination of three methods can be used:

[0085] 1) Memory items are weighted according to their update time; the newer the item, the more valuable it is.

[0086] 2) Weighting is applied based on the importance of memory items; the more important the item, the more valuable it is. Importance is assigned directly when an item is written into memory.

[0087] 3) The weights are calculated based on the cosine similarity between the POI production task (query) and the item embedding features in the memory. The more similar the features, the greater the weight.

[0088] Reflection mechanism:

[0089] Reflection definition: In actual data production, a large number of success and failure cases will be generated. The purpose of the reflection mechanism is to summarize and aggregate these experiences and write them into the memory, so as to provide reference for subsequent data production.

[0090] Reflection algorithm: For each successful or unsuccessful case, its decision-making process and historical information can be sent to the LLM, allowing the LLM to extract valuable information and write it into memory, thus using it as production experience to help with future decisions.

[0091] 2. Model Training Process

[0092] The main training steps are as follows:

[0093] 1) Collect manual work logs and perform SFT on the LLM;

[0094] 2) Utilize reinforcement learning algorithms to optimize LLM;

[0095] 3) After the automated production data is uploaded to the map application, the model is continuously optimized using information such as user feedback.

[0096] To further deepen the understanding of the effects that the above solutions can achieve in specific applications, this disclosure also provides a specific example for illustration:

[0097] Assume the input task description is: verify the opening hours of the Summer Palace (i.e., the verification object is: the scenic spot—the Summer Palace, and the verification question is: the accurate opening hours of the scenic spot).

[0098] The model execution process for processing this task description information can be as follows:

[0099] task definition: verify POI business hours;

[0100] tool set: 1: baidu search engine 2: bing search engine 3, browse website 4, write file write_to_file;

[0101] constraint: 1, only search Chinese web pages; 2, at least get business hours from two web pages to ensure consistency and accuracy 3, prefer baidu search engine, followed by bing search engine;

[0102] specific input: verify the business hours of the Summer Palace;

[0103] step1 LLM output:

[0104] Thoughts: start searching for the Summer Palace business hours;

[0105] PLAN: 1, use baidu search engine 2, search on at least two web pages 3, search with Chinese query;

[0106] ACTION: Command = baidu Arguments = {'input': 'Summer Palace business hours'};

[0107] tools return results: [url1, url2,... url10];

[0108] step2 LLM output:

[0109] Thoughts: select two websites to get the business hours of the Summer Palace;

[0110] PLAN: 1, browse the first reliable website 2, browse the second reliable website;

[0111] ACTION: Command = browse_website Arguments = {'url': 'http: / / www.url1', 'question': 'Summer Palace business hours'};

[0112] tools return results: The business hours of the Summer Palace are as follows: 6:00 open, 19:00 close during peak season (April 1 - October 31). 6:30 open, 18:00 close during off-season (November 1 - March 31).

[0113] step3 LLM output:

[0114] Thoughts: Browse another website to find out the opening hours of the Summer Palace.

[0115] PLAN: Browse a second reliable website;

[0116] ACTION: Command = browse_website Arguments = {'url':'http: / / www.url2','question':'Summer Palace Opening Hours'};

[0117] The tools returned the following results: The Summer Palace's opening hours are as follows: The Summer Palace's opening hours are divided into peak season and off-season. Peak season hours are from 6:00 AM to 7:00 PM, and off-season hours are from 6:30 AM to 6:00 PM.

[0118] Step 4 LLM output:

[0119] Thoughts: I have already verified the Summer Palace's opening hours on another website.

[0120] PLAN: Write the business hours to a file;

[0121] ACTION: Command = write_to_file Arguments = {'file':'Summer Palace Opening Hours.txt','text':'Summer Palace Opening Hours: During peak season (April 1st - October 31st), the park opens at 6:00 AM and closes at 7:00 PM. During off-season (November 1st - March 31st), the park opens at 6:30 AM and closes at 6:00 PM'};

[0122] The tools returned the following result: Input content has been saved.

[0123] Step 5 LLM output:

[0124] Thoughts: The opening hours of the Summer Palace have been recorded in the file.

[0125] PLAN: Task complete, close;

[0126] ACTION: Command=task_complete;

[0127] Task completed!

[0128] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a map data validity verification device, which is similar to... Figure 2The method embodiment shown corresponds to the device, which can be applied to various electronic devices.

[0129] As Figure 5 As shown in the embodiment, the map data validity verification device 500 can include a task description information acquisition unit 501, a validity verification model processing unit 502, and a target map data determination unit 503. The task description information acquisition unit 501 is configured to acquire task description information of a validity verification task initiated for map data. The validity verification model processing unit 502 is configured to process the task description information using a preset validity verification model to obtain a returned validity verification result. The validity verification model is a model trained based on a framework constructed based on an automatic machine learning method, with a generative large language model as the underlying core. The training process for obtaining the validity verification model includes fine-tuning based on historical manual verification job logs collected from map applications as supervised data. The target map data determination unit 503 is configured to determine target map data that passes the validity verification based on the validity verification result.

[0130] In the embodiment, the task description information acquisition unit 501, the validity verification model processing unit 502, and the target map data determination unit 503 in the map data validity verification device 500 can perform specific processing and achieve corresponding technical effects, which can be referred to the descriptions of the task description information acquisition unit 501, the validity verification model processing unit 502, and the target map data determination unit 503 in the method embodiment, respectively. Figure 2 The related descriptions of steps 201-203 in the corresponding embodiment are not repeated here.

[0131] In some optional implementations of the embodiment, the training process further includes: taking the interest point information of a high-frequency interest point with an online access frequency exceeding a preset number of times as a standard answer, and performing one-time optimization training on the model after fine-tuning in a reinforcement learning manner.

[0132] In some optional implementations of the embodiment, the training process further includes: taking the use feedback information of the model obtained after one-time optimization training as a standard answer, and performing two-time optimization training on the model after one-time optimization training in a reinforcement learning manner.

[0133] In some optional implementations of the embodiment, the framework constructed based on the automatic machine learning method includes an auxiliary decision mechanism based on memory data stored in a vector form. The auxiliary decision mechanism is used to recall map-related information stored in a vector form corresponding to the task description information, and assist the generative large language model in making a decision corresponding to the task description information based on the map-related information. The map-related information includes image, text, or audio and video data related to an interest point.

[0134] In some optional implementations of the embodiment, the auxiliary decision mechanism recalls the map-related information stored in the form of vectors corresponding to the task description information based on at least one of the following recall strategies:

[0135] a time-sensitive priority recall strategy, an importance-based priority recall strategy, and a priority recall strategy based on the similarity of features to the task description information.

[0136] In some optional implementations of the embodiment, the framework constructed based on the automatic machine learning method includes a reflection mechanism established based on successful decision cases and failed decision cases in historical decision results, and the reflection mechanism sends the decision process and related historical information corresponding to the successful decision cases and the failed decision cases to the generative large language model, so that the generative large language model extracts experience information for assisting subsequent decision-making from the successful decision cases and the failed decision cases.

[0137] In some optional implementations of the embodiment, the experience information obtained based on the reflection mechanism includes at least one of the following:

[0138] a success rate ranking of obtaining effective information from different information channels, a time consumption ranking of completing verification in different verification manners, and a proportion ranking of recording error information in different websites.

[0139] In some optional implementations of the embodiment, the framework constructed based on the automatic machine learning method includes a task understanding, reasoning, disassembly, and planning mechanism based on a thinking chain.

[0140] In some optional implementations of the embodiment, the framework constructed based on the automatic machine learning method includes a calling mechanism for external tools, and the calling mechanism indicates that the generative large language model calls corresponding external tools based on the understanding of the task description information to perform corresponding queries, and continues to understand or generate validity verification results based on the received query results.

[0141] In some optional implementations of the embodiment, the task description information at least includes a target interest point as a verification object and a verification question for the target interest point.

[0142] In some optional implementations of the embodiment, the map data validity verification apparatus 500 further includes:

[0143] A map application content updating unit configured to update corresponding content in a map application by using the target map data.

[0144] The embodiment provides a map data validity verification device, and is a device embodiment corresponding to the method embodiment. The map data validity verification device provided in the embodiment creatively introduces an effectiveness verification model constructed based on automatic machine learning and trained based on a framework with a generative large language model as a core, to task description information containing corresponding problem content, so that the core and the framework have the ability to accurately understand and determine the effectiveness verification result corresponding to the task description information, thereby automatically completing the effectiveness verification task of the map data, freeing people from tedious manual verification operations, improving the effectiveness verification efficiency and accuracy, and further improving the use experience of the map application.

[0145] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the map data validity verification method described in any of the above embodiments when executed.

[0146] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium storing computer instructions for enabling a computer to implement the map data validity verification method described in any of the above embodiments when executed.

[0147] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which can implement the map data validity verification method described in any of the above embodiments when executed by a processor.

[0148] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0149] As Figure 6As shown, the device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from the storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0150] Various components in the device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc., an output unit 607, such as various types of displays, speakers, etc., a storage unit 608, such as a magnetic disk, an optical disk, etc., and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0151] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the map data validity verification method. For example, in some embodiments, the map data validity verification method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the map data validity verification method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the map data validity verification method by any other appropriate means, such as by means of firmware.

[0152] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0153] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0156] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0158] For the complex map related problems in the field of map that need to be verified for effectiveness, the technical scheme provided by the embodiment of the disclosure creatively introduces an effectiveness verification model trained by a framework constructed based on automatic machine learning, with a generative large language model as the core, to the task description information containing the corresponding problem content, so as to have the ability to accurately understand and determine the effectiveness verification result corresponding to the task description information by means of the core and the framework, thereby automatically completing the effectiveness verification task of the map data and freeing people from tedious manual verification operations, improving the effectiveness verification efficiency and accuracy, and further improving the use experience of map application.

[0159] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0160] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A map data validity verification method, comprising: obtaining task description information of a validity verification task initiated for map data; processing the task description information using a preset validity verification model to obtain a returned validity verification result; wherein the validity verification model is a model constructed based on an automatic machine learning method and trained based on a framework with a generative large language model as a bottom core, a training process of the validity verification model comprises fine-tuning training based on historical manual verification job logs collected from a map application as supervised data, the framework constructed based on the automatic machine learning method comprises an auxiliary decision mechanism based on memory data stored in a vector form, the auxiliary decision mechanism is used to recall map related information stored in a vector form corresponding to the task description information, and assist the generative large language model in making a decision corresponding to the task description information based on the map related information, the map related information comprises image, text or audio and video data related to a point of interest; determining target map data that passes the validity verification according to the validity verification result.

2. The method of claim 1, wherein, The training process further comprises using point of interest information of a high-frequency point of interest with an online access frequency exceeding a preset number of times as a standard answer to perform one optimization training on the model after the fine-tuning training in a reinforcement learning manner.

3. The method of claim 2, wherein, The training process further comprises using obtained user feedback information on the model after the one optimization training to perform two optimization training on the model after the one optimization training in a reinforcement learning manner.

4. The method of claim 1, wherein, The auxiliary decision mechanism performs recall based on at least one of the following recall strategies when recalling the map related information stored in a vector form corresponding to the task description information: a time-sensitive priority recall strategy, an importance-based priority recall strategy, and a priority recall strategy based on a feature similarity with the task description information.

5. The method of claim 1, wherein, The framework constructed based on the automatic machine learning method comprises a reflection mechanism established based on successful decision cases and failed decision cases in historical decision results, the reflection mechanism sends decision processes and related historical information corresponding to the successful decision cases and the failed decision cases to the generative large language model, so that the generative large language model extracts experience information used to assist subsequent decision making therefrom.

6. The method of claim 5, wherein, The experience information obtained based on the reflection mechanism comprises at least one of the following: a success rate ranking of obtaining valid information from different information channels, a time consumption ranking of completing verification in different verification modes, and a proportion ranking of recording error information in different websites.

7. The method of claim 1, wherein, The framework constructed based on the automatic machine learning method comprises a task understanding, reasoning, disassembly and planning mechanism based on a thinking chain.

8. The method of claim 1, wherein, The framework constructed based on the automatic machine learning method comprises a calling mechanism of external tools, the calling mechanism indicates that the generative large language model calls corresponding external tools for corresponding queries based on understanding of the task description information, and continues to understand or generate the validity verification result based on the received query result.

9. The method of claim 1, wherein, The task description information at least includes: a target interest point as a verification object, and a verification question for the target interest point.

10. The method of any one of claims 1-9, further comprising: updating corresponding content in the map application with the target map data.

11. A map data validity verification apparatus, comprising: a task description information acquisition unit configured to acquire task description information of a validity verification task initiated for map data; a validity verification model processing unit configured to process the task description information using a preset validity verification model to obtain a returned validity verification result; wherein the validity verification model is a model constructed based on an automatic machine learning manner and trained based on a framework with a generative large language model as a bottom core, a training process of the validity verification model comprises fine-tuning training based on historical manual verification job logs collected from a map application as supervised data, the framework constructed based on the automatic machine learning manner comprises an auxiliary decision mechanism based on memory data stored in a vector form, the auxiliary decision mechanism is used to recall map related information corresponding to the task description information and stored in a vector form, and assist the generative large language model in making a decision corresponding to the task description information based on the map related information, the map related information comprises image, text or audio and video data related to an interest point; a target map data determination unit configured to determine target map data that passes the validity verification according to the validity verification result.

12. The apparatus of claim 11, wherein, The training process further comprises: taking interest point information of a high-frequency interest point with an online access frequency exceeding a preset number of times as a standard answer, and performing one-time optimization training on the model after the fine-tuning training in a reinforcement learning manner.

13. The apparatus of claim 12, wherein, The training process further comprises: taking obtained user feedback information on the model after the one-time optimization training as a standard answer, and performing two-time optimization training on the model after the one-time optimization training in a reinforcement learning manner.

14. The apparatus of claim 11, wherein, The auxiliary decision mechanism performs recall based on at least one of the following recall strategies when recalling the map related information corresponding to the task description information and stored in a vector form: a time-sensitive priority recall strategy, an importance-based priority recall strategy, and a priority recall strategy based on a feature similarity with the task description information.

15. The apparatus of claim 11, wherein, The framework constructed based on the automatic machine learning manner comprises a reflection mechanism established based on successful decision cases and failed decision cases in historical decision results, the reflection mechanism sends a decision process and related historical information corresponding to the successful decision cases and the failed decision cases to the generative large language model, so that the generative large language model extracts experience information used to assist subsequent decision making from the successful decision cases and the failed decision cases.

16. The apparatus of claim 15, wherein, The experience information obtained based on the reflection mechanism comprises at least one of the following: a success rate sorting of valid information obtained from different information channels, a time consumption sorting of different verification methods all completing verification, and a proportion sorting of error information recorded in different websites.

17. The apparatus of claim 11, wherein, The framework constructed based on the automatic machine learning manner comprises a task understanding, reasoning, disassembly and planning mechanism based on a thinking chain.

18. The apparatus of claim 11, wherein, The framework constructed based on the automatic machine learning manner comprises a calling mechanism of external tools, which indicates that the generative large language model calls corresponding external tools for corresponding queries based on the understanding of the task description information, and continues to understand or generate the validity verification result based on the received query result.

19. The apparatus of claim 11, wherein, The task description information at least comprises a target interest point as a verification object and a verification question for the target interest point.

20. The apparatus according to any one of claims 11-19, further comprising: a map application content updating unit configured to update corresponding content in the map application by using the target map data.

21. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the map data validity verification method according to any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the map data validity verification method according to any one of claims 1-10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the map data validity verification method according to any one of claims 1-10.

Images