A natural language based search quality monitoring method and system

Abstract

A method and system for monitoring search quality based on natural language processing, relating to the field of internet technology. The method includes: obtaining test requests for online travel search services; sequentially initiating search requests to the online travel search service interface and receiving response data; parsing actual search results, extracting the actual resource identifier set, and determining the execution status of test cases; if a test fails, generating a test report including discrepancy information; and displaying the test report in a visual interface, showing the real-time status information of the corresponding resources. By adopting this application, automated inspection of natural language search quality in the online travel field is achieved. Accurate verification of search results is completed based on standardized resource identifiers, while simultaneously enabling visual display of test results and verification of associated resources. This reduces the manual cost of search quality monitoring and improves the accuracy and efficiency of verification.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to a search quality monitoring method and system based on natural language. Background Technology

[0002] With the deepening development of the online travel industry, natural language search services have become a core traffic entry point for online travel platforms, and their search quality directly determines user experience and platform transaction conversion. Online travel search differs fundamentally from general search engines and e-commerce platform search. Search results are strongly correlated with dynamic factors such as user geolocation, request time, and business scenario. The same keywords need to be adapted to different scenarios to output differentiated results. This characteristic poses a specific industry-specific need for automated monitoring of search quality.

[0003] Existing search quality monitoring primarily employs two approaches: automated testing of general APIs and testing of general search engines. Both methods rely on manual input or static scripts to generate test cases, followed by manually writing assertion scripts for result comparison and verification. These approaches suffer from several key drawbacks when applied to online travel scenarios: First, they cannot adapt to the massive test case maintenance demands of dynamic scenarios. Updating test cases and assertions requires manual coding, resulting in high technical barriers, a heavy maintenance workload, and a high risk of errors. Second, they lack unified capabilities for identifying and verifying travel resources in a one-stop manner. Resource validity verification requires cross-system operations, leading to cumbersome processes and extremely low verification efficiency. Third, the general architecture cannot adapt to the differentiated testing needs of multiple interfaces and business lines in online travel, resulting in insufficient business scenario adaptability. Summary of the Invention

[0004] This application provides a natural language-based search quality monitoring method and system to address the problem of low efficiency in test case maintenance and resource verification during online travel search testing.

[0005] Firstly, this application provides a natural language-based search quality monitoring method, the method comprising: Obtain a test request for an online travel search service. The test request includes a test case dataset, which contains multiple test cases. Each test case corresponds to a search keyword and has a preset expected output result. The expected output result includes a resource identifier generated according to a preset coding standard. The resource identifier is used to uniquely identify the business type and associated resources in the search results. Based on the test case dataset, search requests are sent to the online travel search service interface in sequence, and response data returned by the online travel search service interface is received. The response data includes the actual search results, and each result in the actual search results is marked with a resource identifier that conforms to the coding standard. The actual search results in the response data are parsed to extract the actual resource identifier set. The actual resource identifier set is then compared with the expected resource identifier set preset in the test case to determine the execution status of the test case, which includes pass or fail. If the execution status of the test case is "failed", a test report including difference information is generated. The difference information is used to indicate the missing or added resources between the actual set of resource identifiers and the expected set of resource identifiers. The test report is displayed in a visual interface, and based on the resource identifier of each result in the actual search results, the real-time status information of the corresponding resource is queried from the preset resource database and displayed.

[0006] By adopting the above technical solutions, the quality inspection of natural language search in the online tourism field can be automated. Based on a unified and standardized resource identifier, the search results can be accurately verified. At the same time, the test results can be visualized and the related resources can be verified, reducing the manual cost of search quality monitoring and improving the accuracy and efficiency of verification.

[0007] In a specific feasible implementation, after comparing the actual set of resource identifiers with the preset expected set of resource identifiers in the test case to determine the execution status of the test case, the method further includes: If the execution status of a test case is "failed", an assertion update entry is provided in the visual interface; In response to the trigger operation of the assertion update entry, retrieve the top N results in the actual search results that meet the preset filtering conditions, where N is a positive integer; Extract the actual resource identifiers and actual titles corresponding to the first N results, and assemble the actual resource identifiers and actual titles into an updated set of expected resource identifiers and an updated set of expected titles according to the preset encoding specifications; The preset expected output results in the test cases are replaced with the updated expected resource identifier set and the updated expected title set to complete the automatic update of the test cases.

[0008] By adopting the above technical solutions, the maintenance threshold and workload of test cases are significantly reduced, while the risk of errors from manual editing is also reduced, making it suitable for the maintenance needs of massive test cases in the online travel industry.

[0009] In one specific feasible implementation, prior to the step of obtaining the top N results from the actual search results that meet the preset filtering criteria, the method further includes: Obtain the preset data replacement script. The data replacement script is used to define the logical rules for extracting and filtering the actual search results from the response data. The logical rules include resource type filtering rules and sorting rules. Based on the data replacement script, the actual search results are filtered and sorted to obtain the top N results that meet the preset filtering conditions.

[0010] By adopting the above technical solutions, flexible filtering and standardized processing of search results can be achieved, which can adapt to the testing needs of different business scenarios, ensure the accuracy and business adaptability of the expected results after the update, and improve the flexibility of assertion updates and the versatility of the system.

[0011] In a specific feasible implementation, after providing the assertion update entry in the visual interface, the method further includes: Provide an entry point for AI-assisted judgment within the visual interface; In response to the trigger operation of the AI-assisted judgment entry, the actual search results and the preset expected output results in the test cases are input into the preset AI analysis model. The AI ​​analysis model outputs suggested judgment results based on historical data and business logic. The suggested judgment results are used to assist operators in deciding whether to use the actual search results to update the test cases.

[0012] By adopting the above technical solutions, auxiliary judgments can be made for test case update decisions, providing operators with professional advice based on business logic and historical data, reducing the professional threshold for manual decision-making, and improving the accuracy of test case maintenance in complex scenarios.

[0013] In a specific feasible implementation, the actual search results in the response data are parsed to extract the actual resource identifier set, specifically including: Iterate through each result in the actual search results and extract the corresponding resource identifier from the preset fields of each result; Based on the business type code carried in the resource identifier, the actual search results are classified to obtain a subset of actual resource identifiers corresponding to different business types.

[0014] By adopting the above technical solutions, search results can be automatically categorized by business type, supporting accurate verification and statistical analysis by business line, adapting to the testing needs of multiple business resources on online travel platforms, and improving the precision of result comparison.

[0015] In a specific feasible implementation, the visualization interface queries and displays the real-time status information of the corresponding resource from a pre-defined resource database based on the resource identifier of each result in the actual search results. This includes: For each result in the actual search results, its resource identifier is parsed to obtain the business type and resource number; Based on the business type, query the real-time status information of the resource corresponding to the resource number from the corresponding resource database. The real-time status information of the resource includes at least one of the following: whether the resource exists, whether the resource is available for booking, and whether the resource price has changed. The visualization interface displays real-time resource status information in conjunction with corresponding search results.

[0016] By adopting the above technical solutions, a one-stop query and visualization of the real-time status of search results related to resources can be achieved, simplifying the resource verification process and improving the efficiency and accuracy of problem investigation.

[0017] In one specific implementation, prior to the step of obtaining a test request for the online travel search service, the method further includes: It provides a project configuration interface, which is used to create independent test projects for different online travel search services; Under the test project, a scenario configuration interface is provided. The scenario configuration interface is used to create at least one test scenario for the test project and configure the corresponding interface address and assertion script for each test scenario. In the test scenario, a dataset configuration interface is provided. The dataset configuration interface is used to create at least one test dataset for the test scenario. The test dataset includes multiple test cases. In response to a triggering operation on the test dataset, a test request is generated.

[0018] By adopting the above technical solutions, a hierarchical test case management architecture can be constructed, which can adapt to the differentiated testing needs of online travel platforms with multiple interfaces, multiple business lines, and multiple scenarios, realize the refined management and flexible reuse of test resources, and improve the business adaptability and scalability of the automated inspection system.

[0019] A second aspect of this application provides a natural language-based search quality monitoring system, the system comprising: The test request acquisition unit is used to acquire test requests for online travel search services. The test request includes a test case dataset, which contains multiple test cases. Each test case corresponds to a search keyword and has a preset expected output result. The expected output result contains a resource identifier generated according to a preset coding standard. The resource identifier is used to uniquely identify the business type and associated resources in the search results. The search request initiation unit is used to sequentially initiate search requests to the online travel search service interface based on the test case dataset, and receive the response data returned by the online travel search service interface. The response data contains the actual search results, and each result in the actual search results is marked with a resource identifier that conforms to the coding standard. The response parsing unit is used to parse the actual search results in the response data and extract the actual resource identifier set. The comparison unit is used to compare the actual set of resource identifiers with the preset expected set of resource identifiers in the test case to determine the execution status of the test case, which includes pass or fail.

[0020] A third aspect of this application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0021] A fourth aspect of this application provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a natural language-based search quality monitoring method provided in an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0024] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0025] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Please refer to Figure 1This paper presents a flowchart illustrating a natural language-based search quality monitoring method. This method can be implemented using a computer program, a microcontroller, or run on a natural language-based search quality monitoring system. The computer program can be integrated into a computer device or run as a standalone application. Specifically, the method includes steps S100 to S500, as follows: S100. Obtain a test request for the online travel search service. The test request includes a test case dataset, which contains multiple test cases. Each test case corresponds to a search keyword and has a preset expected output result. The expected output result includes a resource identifier generated according to a preset coding standard. The resource identifier is used to uniquely identify the business type and associated resources in the search results. In some embodiments, the test requests received by the system include two types: timed test requests and manually triggered test requests. Timed test requests are automatically generated by the system according to preset timed task rules, while manually triggered test requests are generated after the operator initiates a triggering operation through a visual interface.

[0027] The preset coding standard in this application embodiment refers to the structured coding rules for the customization of online tourism resources of all categories. The coding rules include at least three core fields: business type attribute, regional attribute and unique entity identifier of the resource. Each field is concatenated with a fixed delimiter to form a complete resource identifier.

[0028] In some embodiments, each test case further includes a user geolocation parameter, a request time parameter, and a business scenario identifier parameter. The user geolocation parameter specifies the city or geographic coordinates of the user corresponding to the search request, the request time parameter specifies the date information corresponding to the search request, and the business scenario identifier parameter specifies the business segment corresponding to the search request, thereby adapting to the business characteristics of online travel search services that output different search results as the user's location, time, and business scenario change.

[0029] After receiving a test request, the system performs format validation on the test case dataset in the test request to confirm that the required parameters of each test case are complete and the resource identifiers conform to the preset coding specifications. If the validation passes, the subsequent test process is executed; if the validation fails, a parameter exception message is generated and pushed to the operator.

[0030] Based on the above embodiments, as another optional embodiment, before the step of obtaining a test request for the online travel search service, the method further includes: S101. Provides a project configuration interface, which is used to create independent test projects for different online travel search services. The system assigns a unique project identifier to each independent test project, and each test project corresponds to an independent online tourism search service interface. The online tourism search service interface includes any one of the following: global drop-down search interface, scenic spot-specific search interface, hotel list search interface, vacation product search interface, and transportation product search interface.

[0031] The project configuration interface allows operators to edit, enable, disable, and delete test projects. It also displays core statistics for each test project, including the number of executions, success rate, and number of failed test cases for the day. Operators can access the detailed configuration page of the corresponding test project through the project configuration interface.

[0032] S102. Under the test project, a scenario configuration interface is provided. The scenario configuration interface is used to create at least one test scenario for the test project and configure the corresponding interface address and assertion script for each test scenario. Each test scenario corresponds to a subdivided business test dimension under the test project. The scenario configuration interface allows operators to set independent maintenance personnel permissions for different test scenarios, enabling operators from different business lines to independently maintain the content of the corresponding test scenarios based on their own areas of interest.

[0033] S103. In the test scenario, a dataset configuration interface is provided. The dataset configuration interface is used to create at least one test dataset for the test scenario. The test dataset includes multiple test cases. The dataset configuration interface allows operators to create, edit, and delete test datasets. It also allows operators to import test cases into the test dataset in batches using templates, and allows operators to manually add or edit test cases one by one.

[0034] The dataset configuration interface displays the total number of test cases, execution success rate, number of failed test cases, and the most recent execution time for each test dataset. Operators can directly trigger a single execution operation for the corresponding test dataset through the dataset configuration interface, and can also view the historical execution statistics and historical test reports for the corresponding test dataset.

[0035] S104. In response to the trigger operation on the test dataset, generate a test request.

[0036] Triggering operations include manual execution triggering operations initiated by operators through the dataset configuration interface, and timed triggering operations automatically initiated by the system based on the timed execution rules configured in the test scenario.

[0037] When the system generates a test request, it will encapsulate all test cases, interface addresses configured in the test scenario, request rules, and assertion script information in the corresponding test dataset into the test request. At the same time, it will generate a globally unique execution identifier for this test request. The execution identifier is used to associate all data and log information in the entire test process.

[0038] S200. Based on the test case dataset, sequentially initiate search requests to the online travel search service interface and receive the response data returned by the online travel search service interface. The response data includes the actual search results, and each result in the actual search results is marked with a resource identifier that conforms to the coding standard. Before initiating a search request, the system will first construct the test context, creating an independent execution context for each test case. The execution context stores all parameter information of the corresponding test case and temporary data during the execution process.

[0039] When the system initiates a search request, it assembles the search keywords, user geolocation parameters, request time parameters, and business scenario identifier parameters from the test case into the request parameters according to the parameter specifications of the online travel search service interface, and sends the request according to the preset request method and interface address.

[0040] The system supports both serial and parallel request initiation modes. In serial mode, requests are initiated sequentially according to the order of test cases. In parallel mode, multiple search requests for test cases are initiated simultaneously according to a preset concurrency level. Operators can choose the appropriate request initiation mode based on their testing needs.

[0041] The response data received by the system is encapsulated in JSON format. The response data includes the response status code, response header information, and response body content. The actual search results are stored in preset fields in the response body content. The actual search results are a structured array, and each element in the array corresponds to a search result. Each search result stores a resource identifier that conforms to the encoding standard in a preset fixed field.

[0042] The system performs anomaly checks on the received response data. If an abnormal response status code, an empty response body, or a response body format that does not conform to the specifications is detected, the execution status of the corresponding test case will be marked as abnormal, and the anomaly details and the time of occurrence will be recorded to generate an anomaly log.

[0043] S300. Parse the actual search results in the response data, extract the actual resource identifier set, and compare the actual resource identifier set with the preset expected resource identifier set in the test case to determine the execution status of the test case. The execution status includes pass or fail. When the system parses the response data, it first parses the response body content, locates the array field storing the actual search results, and then extracts the actual search results.

[0044] When the system extracts the actual resource identifier set, it iterates through each result in the actual search results array, extracts resource identifiers from the preset fixed fields of each result, and arranges all the extracted resource identifiers in order according to the sorting order of the actual search results to form the actual resource identifier set.

[0045] When the system performs the comparison operation, it first performs format standardization processing on the actual resource identifier set and the expected resource identifier set, removes invalid whitespace characters, unifies the character encoding format, and then completes the item-by-item comparison according to the preset comparison rules.

[0046] The comparison rules include two dimensions: existence comparison and sorting position comparison. Existence comparison is used to verify whether all elements in the expected resource identifier set appear in the actual resource identifier set, and whether there are any new elements in the actual resource identifier set that are not included in the expected resource identifier set. Sorting position comparison is used to verify whether the sorting position of each resource identifier in the actual resource identifier set is consistent with the preset sorting position in the expected resource identifier set.

[0047] After the system completes the comparison, if the actual set of resource identifiers matches the expected set of resource identifiers perfectly and conforms to the preset comparison rules, the execution status of the test case is determined to be passed. If any of the following situations exist: missing resources, added resources, or incorrect sorting positions, the execution status of the test case is determined to be failed.

[0048] The system will temporarily cache all detailed data and comparison results during the comparison process and record them in the execution log of this test.

[0049] In some embodiments, during the sorting position comparison process, a first sorting weight can be set for the primary core resource and a second sorting weight can be set for the secondary resource. The first sorting weight is greater than the second sorting weight, and the sorting position deviation tolerance of the core resource is less than the sorting position deviation tolerance of the secondary resource.

[0050] Based on the above embodiments, as another optional embodiment, after comparing the actual set of resource identifiers with the preset expected set of resource identifiers in the test case to determine the execution status of the test case, the method further includes: S301. If the execution status of a test case is "failed", provide an assertion update entry in the visual interface. The system displays complete execution information for test cases in a visual interface. This complete execution information includes the test case's basic parameters, the original expected output, the actual search results, the differences in resource identifier comparison, execution time, and exception details.

[0051] The assertion update entry is displayed as an interactive button in a fixed area of ​​the visual interface. Operators can trigger the assertion update process of test cases by clicking the assertion update entry.

[0052] Based on the above embodiments, as another optional embodiment, after providing the assertion update entry in the visual interface, the method further includes: S3011. Provide an entry point for AI-assisted judgment in the visual interface; The AI-assisted judgment entry is displayed in the visual interface as an interactive operation button, in the same display area as the assertion update entry. Operators can trigger the AI-assisted judgment operation at any time before or after triggering the assertion update process.

[0053] S3012. In response to the trigger operation of the AI-assisted judgment entry, the actual search results and the preset expected output results in the test cases are input into the preset AI analysis model. The AI ​​analysis model outputs suggested judgment results based on historical data and business logic. The suggested judgment results are used to assist the operator in deciding whether to use the actual search results to update the test cases.

[0054] When the system inputs data into the artificial intelligence analysis model, it also simultaneously inputs the real-time status information of the corresponding resources, the business scenario rules corresponding to the test cases, and the historical processing results of similar test cases, thereby improving the accuracy of the output results of the artificial intelligence analysis model.

[0055] The AI ​​analysis model outputs three types of recommendations: suggesting test case updates, suggesting interface issues, and suggesting in-depth manual review. Each recommendation is matched with a corresponding confidence level and a description of the judgment criteria, which includes analysis related to changes in resource status, product strategy adjustments, and changes in user search needs.

[0056] S302. In response to the trigger operation of the assertion update entry, obtain the first N results in the actual search results that meet the preset filtering conditions, where N is a positive integer; After receiving the trigger operation, the system can choose to directly use the actual search results obtained in this test, or it can resend a real-time search request to the online travel search service interface to obtain the latest response data and actual search results, thereby avoiding update errors caused by data delays.

[0057] After filtering the actual search results according to the preset filtering conditions, the system extracts the first N results in sorted order for subsequent assertion update operations.

[0058] In some embodiments, N equals 7.

[0059] Based on the above embodiments, as another optional embodiment, before obtaining the first N results in the actual search results that meet the preset filtering conditions, the method further includes: S3021. Obtain a preset data replacement script. The data replacement script is used to define the logical rules for extracting and filtering actual search results from the response data. The logical rules include resource type filtering rules and sorting rules. Data replacement scripts are stored in the system's script management module. Operators can create, edit, copy, and delete data replacement scripts through a visual interface, or directly use the system's pre-built standardized script templates.

[0060] Resource type filtering rules are used to define the types of business resources that need to be retained and the types of business resources that need to be filtered. The selectable business resource types include three major categories: accommodation, transportation, and tourism. Each category is further subdivided into several specific business types. Accommodation includes domestic hotels, international hotels, homestays, and hourly rooms. Transportation includes air tickets, train tickets, bus tickets, ship tickets, and car rental services. Tourism includes scenic spot tickets, group tours, vacation products, and cruise products.

[0061] The sorting rules are used to define the sorting logic of the actual search results. You can choose to use the native sorting returned by the interface, or you can choose to re-sort according to business priority, resource popularity, and resource validity.

[0062] S3022. Based on the data replacement script, filter and sort the actual search results to obtain the top N results that meet the preset filtering conditions.

[0063] The system calls the script execution engine of the corresponding language, loads the preset data replacement script, and inputs the complete response data and the actual search results into the data replacement script.

[0064] The script execution engine executes invalid result filtering, resource type filtering, sorting, and result count truncation operations in sequence according to the logical rules defined in the data replacement script, and finally outputs the first N results that meet the preset filtering conditions.

[0065] S303. Extract the actual resource identifiers and actual titles corresponding to the first N results, and assemble the actual resource identifiers and actual titles into an updated set of expected resource identifiers and an updated set of expected titles according to the preset coding standard. The system iterates through each of the first N results, extracts the corresponding actual resource identifier from the preset fixed fields of each result, extracts the corresponding actual title from the title field of each result, and records the sorting position number of each result.

[0066] The system concatenates the actual resource identifier of each result with its corresponding sorting position number according to the preset assembly rules, forming a single element of the updated expected resource identifier set. All elements are then concatenated according to the sorting order using a preset uniform delimiter to form a complete updated expected resource identifier set.

[0067] The system uses the same assembly rules to concatenate the actual title of each result with its corresponding sorting position number to form a single element of the updated expected title set. Then, it uses the same unified delimiter to complete the concatenation and form a complete updated expected title set.

[0068] S304. Replace the preset expected output results in the test cases with the updated expected resource identifier set and the updated expected title set to complete the automatic update of the test cases.

[0069] The system writes the updated set of expected resource identifiers and the updated set of expected titles into the expected output storage field of the corresponding test case, overwriting the original preset expected output in the test case.

[0070] After the system completes the test case update, it will display a notification message on the visual interface indicating that the update is complete. At the same time, the test case content displayed on the interface will be automatically refreshed. Operators can directly trigger a single verification execution of the updated test cases on the current interface to confirm that the updated test cases can be executed normally.

[0071] Based on the above embodiments, as another optional embodiment, parsing the actual search results in the response data and extracting the actual resource identifier set specifically includes: S305. Traverse each result in the actual search results and extract the corresponding resource identifier from the preset fields of each result; The preset field is a fixed and unique identifier field within each search result in the response data. The field name and data format of this field are kept consistent across all online travel search service interfaces to ensure the universality of the resource identifier extraction logic.

[0072] The system reads the JSON data of each result one by one according to the original sorting order of the actual search results, locates the preset unique identifier field, and reads the resource identifier string stored in the field.

[0073] The system performs format compliance checks on each extracted resource identifier to determine whether the resource identifier conforms to the preset coding specifications and whether it contains complete business type attributes, regional attributes, and unique entity identifier core fields.

[0074] For resource identifiers with compliant formats, the system includes them in the actual resource identifier set; for resource identifiers with non-compliant formats, the system marks them as invalid results, records the corresponding exception information, and does not include them in the actual resource identifier set.

[0075] S306. Based on the business type code carried in the resource identifier, classify the actual search results to obtain a subset of actual resource identifiers corresponding to different business types.

[0076] The business type code is a fixed field in a preset position in the resource identifier, used to uniquely identify the online travel business type corresponding to the resource. Different business type codes correspond to different business lines such as domestic hotels, homestays, air tickets, train tickets, bus tickets, scenic spot tickets, and group tours.

[0077] The system parses each resource identifier in segments, extracts the business type code field, and groups resource identifiers with the same business type code into the same data set, forming a subset of actual resource identifiers corresponding to that business type.

[0078] S400. If the execution status of the test case is "failed", generate a test report including difference information. The difference information is used to indicate the missing or added resources between the actual resource identifier set and the expected resource identifier set. The system summarizes all data from the entire execution process of this test case, including the basic parameters of the test case, request information, response data, comparison process data, differences in resource identifier sets, execution status, and exception logs, and generates a test report.

[0079] The discrepancy information includes three parts: missing resource details, newly added resource details, and resource details with mismatched sorting positions. Missing resource details record resource identifiers that exist in the expected resource identifier set but do not appear in the actual resource identifier set, along with their corresponding titles and business types. Newly added resource details record resource identifiers that exist in the actual resource identifier set but are not included in the expected resource identifier set, along with their corresponding titles and business types. Resource details with mismatched sorting positions record resource identifiers whose sorting positions are inconsistent with the preset positions, along with their corresponding expected and actual positions.

[0080] S500 displays the test report in a visual interface, and in the visual interface, based on the resource identifier of each result in the actual search results, queries and displays the real-time status information of the corresponding resource from the preset resource database.

[0081] The system displays all the contents of the test report in a visual interface, divided into modules, including basic information on test execution, execution statistics, test case execution details, cross-column comparison of difference information, and execution logs.

[0082] Based on the above embodiments, as another optional embodiment, in the visualization interface, based on the resource identifier of each result in the actual search results, the real-time status information of the corresponding resource is queried from a preset resource database and displayed, specifically including: S501. For each result in the actual search results, parse its resource identifier to obtain the business type and resource number; The system parses the resource identifier string in segments according to the preset encoding standard, splits the resource identifier by preset delimiters, and extracts the business type attribute field, the regional attribute field, and the unique entity identifier field respectively.

[0083] The system extracts the business type code from the business type attribute field, maps it to the corresponding business type, and extracts the unique resource number of the corresponding resource under the business type from the unique entity identifier field and the regional attribute field.

[0084] S502. Based on the business type, query the real-time status information of the resource corresponding to the resource number from the corresponding resource database. The real-time status information of the resource includes at least one of the following: whether the resource exists, whether the resource is available for booking, and whether the resource price has changed. The system pre-configures corresponding resource databases and resource query interfaces for each type of online travel business. The resource databases include hotel resource databases, air ticket resource databases, train ticket resource databases, scenic spot resource databases, bus ticket resource databases, and vacation product resource databases. Each resource database is synchronized with the underlying business database of the corresponding business line of the online travel platform.

[0085] Based on the parsed business type, the system calls the standardized resource query interface of the corresponding business line, passes the resource number as a request parameter to the interface, and obtains the real-time status information of the resource returned by the interface.

[0086] In some embodiments, for hotel business type resources, the real-time status information includes whether the hotel is in operation, whether there are rooms available for booking on the current date, and whether the room price has changed; for air ticket business type resources, the real-time status information includes whether the corresponding route exists, whether there are flights available for sale within the next preset number of days, and whether the flight price has changed; for scenic spot business type resources, the real-time status information includes whether the scenic spot is open, whether there are tickets available for sale on the current date, and whether the ticket price has changed.

[0087] For resources related to train tickets, real-time status information includes whether the corresponding route exists and whether there are available trains for sale within a preset number of days. The system can also query real-time status information such as resource availability time, compliance status, inventory status, and any expansion of sales restrictions, based on business needs.

[0088] S503. In the visualization interface, the real-time status information of resources is displayed in association with the corresponding search results.

[0089] In some embodiments, the system marks abnormal resource status information with a prominent display style. For non-existent resources, offline resources, and unbookable resources, corresponding abnormal identifiers and status descriptions are added.

[0090] The system provides an interactive button for re-verifying the resource status in the corresponding area of ​​each search result. After the operator clicks the button, the system will call the resource query interface again to obtain the latest real-time status information of the resource and refresh the content displayed on the interface.

[0091] For resources missing from the expected resource identifier set, the system also provides a corresponding resource query entry. Operators can directly query the real-time status information of the missing resources in the visual interface to determine whether the resource has been taken offline or whether the resource number has been changed, thereby assisting operators in making decisions to update test cases or troubleshoot interface problems.

[0092] Based on the above embodiments, as another optional embodiment, this application also provides a natural language-based search quality monitoring system, including: The test request acquisition unit is used to acquire test requests for online travel search services. The test request includes a test case dataset, which contains multiple test cases. Each test case corresponds to a search keyword and has a preset expected output result. The expected output result contains a resource identifier generated according to a preset coding standard. The resource identifier is used to uniquely identify the business type and associated resources in the search results. The search request initiation unit is used to sequentially initiate search requests to the online travel search service interface based on the test case dataset, and receive the response data returned by the online travel search service interface. The response data contains the actual search results, and each result in the actual search results is marked with a resource identifier that conforms to the coding standard. The response parsing unit is used to parse the actual search results in the response data and extract the actual resource identifier set. The comparison unit is used to compare the actual set of resource identifiers with the preset expected set of resource identifiers in the test case to determine the execution status of the test case, which includes pass or fail.

[0093] In some embodiments, the system further includes: a report generation unit, configured to generate a test report containing difference information if the execution status of a test case is "failed", the difference information being used to indicate missing or added resources between the actual resource identifier set and the expected resource identifier set; and a visualization unit, configured to display the test report in a visualization interface, and in the visualization interface, query and display the real-time status information of the corresponding resource from a preset resource database based on the resource identifier of each result in the actual search results.

[0094] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0095] Based on the above embodiments, as another optional embodiment, the present application embodiment may further include a computer storage medium, which may store multiple instructions adapted for loading by a processor and executing a method of the above embodiments. For the specific execution process, please refer to the detailed description of the above embodiments, which will not be repeated here.

[0096] Based on the above embodiments, as another optional embodiment, this application embodiment may further include an electronic device. The electronic device may include: at least one processor, at least one communication bus, a user interface, at least one network interface, and a memory.

[0097] The communication bus is used to enable communication between these components.

[0098] The user interface may include a display screen and a camera. Optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0099] The network interface may include standard wired interfaces and wireless interfaces (such as Wi-Fi interfaces).

[0100] The processor may include one or more processing cores. It connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0101] The memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. As a computer storage medium, the memory may include an operating system, a network communication module, a user interface module, and an application program of one method.

[0102] In electronic devices, the user interface is primarily used to provide an input interface for users and to acquire user input data; while the processor can be used to call an application program stored in memory that represents a method. When executed by one or more processors, this causes the electronic device to perform one or more methods as described in the above embodiments. It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0104] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0108] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0109] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for natural language based search quality monitoring, characterized in that, The method includes: Obtain a test request for an online travel search service. The test request includes a test case dataset, which contains multiple test cases. Each test case corresponds to a search keyword and has a preset expected output result. The expected output result includes a resource identifier generated according to a preset coding standard. The resource identifier is used to uniquely identify the business type and associated resources in the search results. Based on the test case dataset, search requests are sequentially sent to the online travel search service interface, and response data returned by the online travel search service interface is received. The response data includes actual search results, and each result in the actual search results is marked with a resource identifier that conforms to the coding specification. The actual search results in the response data are parsed to extract the actual resource identifier set, and the actual resource identifier set is compared with the preset expected resource identifier set in the test case to determine the execution status of the test case, the execution status including pass or fail; If the execution status of the test case is "failed", a test report including difference information is generated. The difference information is used to indicate the missing or added resources between the actual resource identifier set and the expected resource identifier set. The test report is displayed in a visual interface, and in the visual interface, based on the resource identifier of each result in the actual search results, the real-time status information of the corresponding resource is queried from the preset resource database and displayed.

2. The natural language based search quality monitoring method of claim 1, wherein, After the step of comparing the actual set of resource identifiers with the preset expected set of resource identifiers in the test case to determine the execution status of the test case, the method further includes: If the execution status of the test case is "failed", an assertion update entry is provided in the visualization interface; In response to the trigger operation of the assertion update entry, the top N results that meet the preset filtering conditions in the actual search results are obtained, where N is a positive integer; Extract the actual resource identifiers and actual titles corresponding to the first N results, and assemble the actual resource identifiers and actual titles into an updated set of expected resource identifiers and an updated set of expected titles according to the preset encoding standard; The preset expected output results in the test cases are replaced with the updated expected resource identifier set and the updated expected title set to complete the automatic update of the test cases.

3. The natural language based search quality monitoring method of claim 2, wherein, Before the step of obtaining the top N results that meet the preset filtering conditions from the actual search results, the method further includes: Obtain a preset data replacement script, which is used to define logical rules for extracting and filtering actual search results from the response data. The logical rules include resource type filtering rules and sorting rules. Based on the data replacement script, the actual search results are filtered and sorted to obtain the top N results that meet the preset filtering conditions.

4. The natural language based search quality monitoring method of claim 2, wherein, After providing the assertion update entry in the visualization interface, the method further includes: An AI-assisted judgment entry point is provided in the visual interface; In response to the triggering operation of the AI-assisted judgment entry point, the actual search results and the preset expected output results in the test case are input into the preset AI analysis model. The AI ​​analysis model outputs a suggested judgment result based on historical data and business logic. The suggested judgment result is used to assist the operator in deciding whether to use the actual search results to update the test case.

5. The natural language based search quality monitoring method of claim 1, wherein, The step of parsing the actual search results in the response data and extracting the actual resource identifier set specifically includes: Iterate through each result in the actual search results and extract the corresponding resource identifier from the preset fields of each result; Based on the business type code carried in the resource identifier, the actual search results are classified to obtain a subset of actual resource identifiers corresponding to different business types.

6. The natural language-based search quality monitoring method according to claim 5, characterized in that, In the visualization interface, based on the resource identifier of each result in the actual search results, the system queries and displays the real-time status information of the corresponding resource from a preset resource database, specifically including: For each result in the actual search results, its resource identifier is parsed to obtain the business type and resource number; Based on the business type, query the real-time status information of the resource corresponding to the resource number from the corresponding resource database. The real-time status information of the resource includes at least one of the following: whether the resource exists, whether the resource is available for booking, and whether the resource price has changed. The visualization interface displays the real-time status information of the resources in association with the corresponding search results.

7. The natural language-based search quality monitoring method according to claim 1, characterized in that, Prior to the step of obtaining a test request for the online travel search service, the method further includes: A project configuration interface is provided, which is used to create independent test projects for different online travel search services. Under the test project, a scenario configuration interface is provided. The scenario configuration interface is used to create at least one test scenario for the test project and configure the corresponding interface address and assertion script for each test scenario. In the test scenario, a dataset configuration interface is provided, which is used to create at least one test dataset for the test scenario, and the test dataset includes multiple test cases; The test request is generated in response to a trigger operation on the test dataset.

8. A search quality monitoring system based on natural language, characterized in that, The system includes: The test request acquisition unit is used to acquire test requests for online travel search services. The test request includes a test case dataset, which includes multiple test cases. Each test case corresponds to a search keyword and has a preset expected output result. The expected output result includes a resource identifier generated according to a preset coding standard. The resource identifier is used to uniquely identify the business type and associated resources in the search results. The search request initiation unit is used to sequentially initiate search requests to the online travel search service interface according to the test case dataset, and receive response data returned by the online travel search service interface. The response data includes actual search results, and each result in the actual search results is marked with a resource identifier that conforms to the coding specification. The response parsing unit is used to parse the actual search results in the response data and extract the actual resource identifier set. The comparison unit is used to compare the actual set of resource identifiers with the preset expected set of resource identifiers in the test case to determine the execution status of the test case, wherein the execution status includes pass or fail.

9. An electronic device, characterized in that, It includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted to be loaded by a processor and executed as described in any one of claims 1-7.

Images