Testing method and system based on logistics field large model context

Abstract

The invention relates to the technical field of logistics large models, and discloses a testing method and system based on the context of a large model in the logistics field, and the method comprises the steps: 1, preparing a single-round successful testing record of an entity and an intention which need to be generated, guaranteeing that the corresponding entity or intention is inherited and switched for assembly when a plurality of rounds of use cases are generated, the prepared single-round successful test records form a basic data set; step 2, determining parameters required for generating multiple rounds of use cases; 3, filtering and classifying single-round test records; step 4, generating a multi-round use case; 5, converting the multi-round use cases into executable tasks, and calculating a model success rate; and step 6, counting failure distribution and feeding back. The test method and system can accurately capture the performance of the model in complex interactions such as intention inheritance and switching, assist in rapidly improving the capture and understanding ability of the logistics large model for context information, and reduce intelligent dialogue errors caused by context faults.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of logistics large model, in particular to a test method and system based on logistics field large model context. BACKGROUND

[0002] In the context of the continuous acceleration of the digital transformation of the logistics industry, intelligent dialogue technology has become a key force to break through the situation. With the rapid development of e-commerce, the logistics business volume is growing explosively, and the traditional communication and interaction mode is difficult to cope with the massive and complex information processing demand. Customer inquiries are high-frequency and repetitive, and problems such as "where is my package" and "how to calculate the shipping cost" continue to emerge, and manual response is inefficient and costly; at the same time, the internal knowledge of logistics enterprises is scattered in different departments and systems, and employees find it inconvenient to query, which greatly affects work coordination.

[0003] Intelligent dialogue is based on advanced technologies such as natural language processing, machine learning, and knowledge graph. At the customer service level, it can automatically identify customer problems, accurately and quickly answer questions, and provide 7x24-hour online service, significantly improving response speed and customer satisfaction. For example, Zhaobao brand intelligent agent zone of 100 Express, users can communicate smoothly with intelligent agents through text or voice, conveniently check express, compare prices, and understand prohibited goods regulations, etc. Within the enterprise, intelligent dialogue can integrate various logistics knowledge, and when employees inquire about transportation policies, customs clearance processes, etc., they can quickly obtain accurate information, break down departmental knowledge barriers, and optimize work processes. For example, Jingdong Logistics Agent, as an intelligent hub, can develop accurate marketing strategies, act as an intelligent customer service to answer customer questions, and optimize various aspects of operation, helping logistics enterprises gain an advantage in the complex market competition. In short, intelligent dialogue is reshaping the way the logistics industry communicates and interacts, and is driving the industry towards a new journey of intelligent and efficient operation.

[0004] The existing logistics field large model has poor coherence in understanding the context in multi-turn dialogue, often with incorrect intent inheritance or switching, such as the user first asking "Shanghai's fake signing volume" and then asking "Beijing's", the model may not inherit the intent of "fake signing volume", and the entity recognition and inheritance ability is weak, when facing unorganized use cases or scenarios that need to reset entity intent, it is easy to confuse or lose key information; lack of adaptability to complex dialogue scenarios, the model is prone to context faults, and cannot accurately respond, etc., resulting in ambiguous optimization direction. SUMMARY

[0005] (I) Technical problems solved In view of the deficiencies of the prior art, the present application provides a test method and system based on the context of a logistics field large model, which has the advantages of being able to quickly construct test samples that fit the actual logistics scene, greatly reducing the cost and time of manually designing use cases, being able to accurately capture the performance of the model in complex interactions such as intent inheritance and switching, helping to quickly improve the ability of the logistics large model to capture and understand context information, and reducing intelligent conversation errors caused by context faults.

[0006] (II) Technical solutions To achieve the above object, the present application provides the following technical solutions: a test method based on the context of a logistics field large model, comprising the following steps: Step one, prepare single-round successful test records of entities and intents that need to be generated, ensure that when generating multi-round use cases, there are corresponding entities or intents for assembly in inheritance and switching, and form a basic data set from the prepared single-round successful test records; Step two, determine the parameters required for generating multi-round use cases; Step three, filter and classify single-round test records; Step four, generate multi-round use cases; Step five, convert the multi-round use cases into executable tasks and calculate the success rate of the model; Step six, count the failure distribution for feedback.

[0007] Preferably, the basic data set is composed of a plurality of single-round successful test record data, and the numbering expression of the basic data set is: wherein, represents the first single-round successful test record data in the basic data set, represents the single-round successful test record data in the basic data set, and represents the total number of single-round successful test record data obtained in the basic data set.

[0008] Preferably, the parameters required for determining the generation of multi-round use cases include the number of rounds , intelligent agents , time range , intent , use case type , and use case number , and the parameters are specifically described as: (1) Number of rounds : the number of rounds of multi-round use cases; (2) Intelligent agent : test records run by a specified intelligent agent are selected; (3) Time range : test records in a specified time period are selected; (4) Intention : Get single round test records of specified intention, for generating context use cases; (5) Use case type : Intention inheritance, intention switching, inheritance & switching; (6) Number of use cases : Generate the number of use cases of the specified use case type.

[0009] Preferably, the filtering and classifying single round test records include entity classification and intention classification, the entity classification is grouped by organizational entities, such as ({"Shanghai": ["Shanghai cost", "Shanghai diagnosis"], "Beijing": ["Beijing signing"]}); The intention classification is grouped by scenarios, such as ({"diagnosis": ["Shanghai diagnosis", "Beijing diagnosis"], "signing": ["Shanghai signing"]}); The string in the above "[]" is a use case object, which is a short form. The use case object contains the attributes: organizational entity, time entity, and intention.

[0010] Preferably, the generating multi-round use cases includes generating by use case type, special rule generation, and randomization generation.

[0011] Preferably, the generating by use case type includes: (1) Intention inheritance: that is, obtaining use cases of the same intention but different organizations. For example: ["Shanghai comprehensive diagnosis", "Beijing", "Wuhan"], which generates a multi-round use case of intention inheritance with 3 rounds; (2) Intention switching: that is, obtaining use cases of the same organizational entity but different intentions. For example: ["Shanghai single ticket capacity cost", "comprehensive diagnosis", "CCR"], which generates a multi-round use case of intention switching with 3 rounds; (3) Inheritance & switching: it needs to include both inheritance and switching types, such as: ["Shanghai comprehensive diagnosis", "Beijing", "How is the weather?"] which generates a multi-round use case of inheritance & switching with 3 rounds; The special rule generation includes: (1) No organization use case: when generating multi-round use cases, some use cases are without organization. At this time, the use case can be added to the multi-round use case as intention switching data. The following use case inherits the intention and the organization is the same as before; (2) Entity intention reset: in the use case type of inheritance & switching, when the length of the use case is greater than 7, the middle round will be a full use case. A sample use case with relatively more entities and intentions will be selected to reset the entities and intentions of the previous use cases, verifying the reset ability of entities and intentions; The random generation includes: according to the parameters, the single round use case that meets the condition is randomly combined to generate a specified number of multi-round use cases.

[0012] Preferably, in the generation according to the use case type, the first round of use cases must be full of entities and intentions, and the following may appear full of entities and intentions, and the following entities and intentions are not full, and the corresponding organization or intention needs to be removed according to the corresponding use case type, and the corresponding number of use cases is generated according to this rule.

[0013] Preferably, the model success rate The calculation formula is: ; In the calculation formula, represents the model success rate; represents the total number of multi-round use cases; represents an indicator function, indicating whether the first task is successful, and the value is 1 when all rounds of the task pass the expected verification, otherwise the value is 0; represents the first task; indicates the cumulative value of the indicators from the first task to the first task.

[0014] Preferably, the specific steps of the statistical failure distribution feedback are: S1, extract the failed multi-round task list; S2, obtain detailed information of each failed task, including round sequence, failure round position, model error response, and expected correct result; S3, statistics the distribution characteristics of the failure cases, including intention understanding error, entity recognition error, context inheritance failure, and irrelevant response; S4, classify by round position, and count the number of failures in the first round; S5, classify by use case type, including intention inheritance, intention switching, inheritance & switching; S6, generate a feedback report according to the above statistical data and the model success rate.

[0015] A test system based on the context of a logistics field large model is applied to a test method based on the context of a logistics field large model, which includes a data acquisition module, a parameter configuration module, a processing module, a generation module, a calculation module, and a feedback module. The data acquisition module is used to acquire single-round successful test records and form a basic data set. The parameter configuration module is used to determine the parameters for generating multi-round use cases, and supports user-defined test scenarios. The processing module is used for classifying single-round test records by entity and intention, and providing structured data for multi-round use case generation; The generation module is used for generating multi-round use cases according to parameters and rules; The calculation module is used for converting the multi-round use cases into executable tasks, verifying model responses round by round, calculating overall success rate and counting failure distribution; The feedback module is used for generating feedback reports for statistics data and model success rate.

[0016] Compared with the prior art, the present application provides a test method and system based on the context of a logistics field large model, which has the following beneficial effects: 1、The present application generates multi-round use cases based on single-round successful test records, filters and classifies clear parameters, can quickly build test samples that fit the actual logistics scene, greatly reduces the cost and time of manual design of use cases, and the classification method and generation rules ensure the pertinence of the test, which can accurately capture the performance of the model in complex interactions such as intention inheritance and switching.

[0017] 2、The present application quantifies the model performance through a clear success rate calculation formula, and further through detailed failure distribution statistics, locates the problem from the dimensions of failure round, use case type, error type, etc., provides accurate direction for model iteration, helps to quickly improve the ability of the logistics large model to capture and understand context information, and reduces intelligent conversation failures caused by context faults.

[0018] 3、The present application can more accurately understand the real needs of users in multi-round dialogue, whether it is information continuation in customer consultation or instruction connection in enterprise internal cooperation, it can maintain coherent response, thereby improving the reliability of the logistics intelligent dialogue system, reducing communication cost, and ultimately promoting the improvement of logistics service efficiency and quality. The collaborative work of the system modules also makes the test process more standardized and reusable, and adapts to the changes in test requirements of different logistics scenes. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The present application is a method step diagram; Figure 2 The present application is a system flowchart; Figure 3 The present application is a multi-round use case generation diagram; Figure 4 The present application is a multi-round use case execution diagram. DETAILED DESCRIPTION

[0020] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0021] Please refer to Figures 1-4 A test method based on a logistics field large model context, comprising the following steps: Step one, prepare the single round success test record of the entity and intent to be generated, ensure that when generating multiple round use cases, do inheritance, switching with corresponding entity or intent to assemble, form a basic data set with the prepared single round success test record; The basic data set is composed of multiple single round success test record data, and the numbering expression of the basic data set is: Wherein, represents the first single round success test record data in the basic data set, represents the single round success test record data in the basic data set, represents the total number of single round success test record data obtained in the basic data set; By systemically preparing single round success test records, a high-quality data base for multiple round dialogue test is constructed, the context logic completeness is guaranteed, the entity and intent are ensured to be traceable and combinable when generating multiple round use cases, the test reliability is strengthened, the multiple round scene is generated based on the verified single round data, the defects of the model in complex interaction are accurately positioned, the scale expansion is supported, the standardized numbering system provides a basis for data management, case tracing and strategy optimization, and helps the continuous iteration of the test system; Step two, determine the parameters required for generating multiple round use cases, including the number of rounds , agent , time range , intent , use case type , and use case number , and the parameters are specifically described as: (1) the number of rounds : the number of rounds of multiple round use cases; (2) agent : the test record of the selected specified agent; (3) time range : select the test record in the specified time period; (4) intent : obtain the single round test record of the specified intent, which is used to generate a context use case; (5) Use case type : intent inheritance, intent switching, inheritance & switching; (6) Number of use cases : generate the number of use cases of the specified use case type; Precise control is provided for multi-round use case generation through parameterized configuration. By setting parameters such as the number of rounds, the time range, and the intent, different complexity of dialogue processes can be simulated. The performance of the model in a specific business scenario or interaction mode can be verified by specifying the agent, intent type, and use case type. The number of use cases parameter is used to generate a batch of use cases that meet the rules, balancing the test breadth and efficiency, and avoiding the redundant labor of manually constructing cases. The single-round records are filtered according to the time range, agent, and other conditions to ensure that the generated multi-round use cases are based on high-quality and up-to-date data, and to improve the effectiveness and representativeness of the test results; Step three, filter and classify single-round test records, including classification by entity and classification by intent; Classification by entity is grouping by organizational entity, such as ({"Shanghai": ["Shanghai cost", "Shanghai diagnosis"], "Beijing": ["Beijing signing"]}). Classification by intent is grouping by scenario, such as ({"diagnosis": ["Shanghai diagnosis", "Beijing diagnosis"], "signing": ["Shanghai signing"]}). The strings in the above "[]" are a use case object, which is a short form. The use case object contains the attributes: organizational entity, time entity, and intent. Structured classification of single-round test records provides efficient data support for multi-round use case generation. After grouping by entity and intent, use case fragments that meet the inheritance and switching rules can be quickly located, improving the efficiency of multi-round scenario construction. The association between entities and intents is clear after classification, avoiding semantic conflicts or entity jumps in multi-round testing. Specific types of use cases can be generated based on the classification results, accurately verifying the model's contextual understanding ability for local scenarios. By filtering irrelevant records, the generated multi-round use cases focus on the core test objectives, improving resource utilization and test accuracy; Step four, generate multi-round use cases, including generation by use case type, special rule generation, and randomization generation. The generation by use case type includes: (1) Intent inheritance: obtaining use cases with the same intent but different organizations. For example: ["Shanghai comprehensive diagnosis", "Beijing", "Wuhan"], which generates a multi-round use case with intent inheritance and a round number of 3; (2) Intent switching: obtaining use cases with the same organizational entity but different intents. For example: ["Shanghai single ticket capacity cost", "comprehensive diagnosis", "CCR"], which generates a multi-round use case with intent switching and a round number of 3; (3) Inheritance & Switch: It needs to include both inheritance and switching types, such as: ["Shanghai comprehensive diagnosis", "Beijing", "What is the weather like"], which generates a multi-round use case with 3 rounds of inheritance & switching; In the generation by use case type, the first round of use case must be full entity and intent, and the following can be full entity and intent, and the following entity and intent are not full, which needs to remove the corresponding organization or intent according to the corresponding use case type, and generate the corresponding number of use cases according to this rule; Special rule generation includes: (1) No organization use case: When generating multi-round use cases, some use cases have no organization, which can be added to the multi-round use case as intent switching data. The following use case inherits the intent and the organization is the previous organization; (2) Entity intent reset: In the use case type of inheritance & switching, when the length of the use case is greater than 7, the middle round will be a full use case, which will select a sample use case with relatively more entities and intents, reset the previous entities and intents, and verify the reset ability of entities and intents; Randomization generation includes: randomly combining single-round use cases that meet the conditions according to parameters to generate a specified number of multi-round use cases; Through multi-dimensional and rule-based generation of multi-round use cases, a test system covering complex dialogue scenarios is constructed, simulating real business complexity, verifying the ability of the model to handle long context, entity jumping and intent conversion in the logistics scene through intent inheritance, intent switching and mixed mode. The first round requires complete entities and intents, and the subsequent rounds are simplified according to the rules, which not only ensures basic logic verification, but also exposes the reasoning defects of the model when information is missing. Through no organization use case, the response ability of the model to generalized intent is tested. Through entity intent reset, the scene of users supplementing key information in the middle is simulated to ensure the robustness of the model in long-term interaction. Randomization generation combined with parameter constraints can not only construct use cases that meet business rules in batches, but also avoid the bias of manually designed cases, improving the comprehensiveness and resource utilization of testing; Step five, convert multi-round use cases into executable tasks, calculate model success rate, and model success rate The calculation formula is: ; In the calculation formula, represents the success rate of the model; represents the total number of multi-round use cases; represents the indicator function, which indicates whether the th task is successful. When all rounds of task pass the expected verification, the value is 1, otherwise the value is 0; represents the th task; indicates the cumulative value of the index from the first task to the task; The multi-round use case is converted into an executable task and the success rate is calculated, realizing the key closed loop from scenario simulation to model performance quantification, and the success rate formula converts the fuzzy model performance into a comparable percentage index, providing an objective basis for comparing the effects of different versions of models or optimization strategies. Step six, feedback by statistics of failure distribution, the specific steps are: S1, extract the multi-round task list of failure; S2, obtain detailed information of each failed task, including round sequence, failure round position, model error response, and expected correct result; S3, statistics of distribution characteristics of failure cases, including intent understanding error, entity recognition error, context inheritance failure, and irrelevant response; S4, classify by round position, and statistics of failure concentrated in which round; S5, classify by use case type, including intent inheritance, intent switching, inheritance & switching; S6, generate feedback report based on the above statistical data and model success rate for feedback; Through systematic analysis of failure cases and generation of feedback report, a deep closed loop from test results to optimization decision is realized. By extracting the round sequence, error response and other details of the failed task, the collapse point of the model in the long context is determined. The error type, round position and use case type are classified and counted to reveal the vulnerability of the model in a specific scenario or interaction stage. Based on the failure distribution feedback, the training data can be supplemented or the model architecture can be adjusted. Finally, through the generation of structured feedback report, an operable improvement direction is provided to avoid repeated trial and error and accelerate the improvement of model maturity; The application system of the above test method based on the context of the logistics field large model includes a data acquisition module, a parameter configuration module, a processing module, a generation module, a calculation module, and a feedback module. The data acquisition module is used to acquire single-round successful test records and form a basic data set; The parameter configuration module is used to determine the parameters for generating multi-round use cases, supporting user-defined test scenarios; The processing module is used to classify single-round test records by entity and intent to provide structured data for multi-round use case generation; The generation module is used to generate multi-round use cases according to parameters and rules; The calculation module is used to convert multi-round use cases into executable tasks, verify model responses round by round, calculate the overall success rate, and count the failure distribution; The feedback module is used to generate feedback reports based on the statistical data and model success rate for feedback.

[0022] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for testing based on a context of a large model in a logistics field, characterized in that, It comprises the following steps: Step one, prepare the entity, intent single round success test record that needs to be generated, ensure that when generating multi-round use cases, inheritance, switching has corresponding entity or intent to assemble, form the basic data set of the prepared single round success test record; Step two, determine the parameters required to generate multi-round use cases; Step three, filter and classify single round test records; Step four, generate multi-round use cases; Step five, convert multi-round use cases into executable tasks, and calculate the success rate of the model; Step six, statistics failure distribution feedback.

2. The logistics field large model context-based test method according to claim 1, characterized in that: The basic data set is composed of a plurality of single round success test record data, and the numbering expression of the basic data set is: Wherein, represents the first single round success test record data in the basic data set, represents the first single round success test record data in the basic data set, represents the first single round success test record data in the basic data set, represents the total number of single round success test record data obtained in the basic data set.

3. The method of claim 2, wherein the method is based on a context of a large model in a logistics field. The determining generates parameters required for a multi-round use case, including the number of rounds , an agent , a time range , an intent , a use case type , and a use case number ; specifically, number of rounds : number of rounds for multi-round use case; Agent : i.e. the test record run by the specified agent; Time Range : Selects test records in a specified time period; Intention : Acquire single-turn test records for the specified intention for generating context use cases; Use case type : intent inheritance, intent switching, inheritance & switching; number of use cases : generates the number of use cases of a specified use case type.

4. The logistics field large model context-based test method according to claim 3, characterized in that: The filtering and classification of single round test records includes classification by entity and classification by intent, the classification by entity is grouped by organizational entity, { "shanghai": ["shanghai cost", "shanghai diagnosis"], "beijing": ["beijing sign"]}; The classification by intent is grouped by scene, { "diagnosis": ["shanghai diagnosis", "beijing diagnosis"], "sign": ["shanghai sign"]}; The string in the above "[]" is a use case object, which is a short form. The use case object contains the following attributes: organizational entity, time entity, and intent.

5. The logistics field large model context-based test method according to claim 4, characterized in that: The generation of multi-round use cases includes generation by use case type, special rule generation, and randomization generation.

6. The logistics field large model context-based test method according to claim 5, characterized in that: The generation by use case type includes: Intent inheritance: that is, obtaining use cases with the same intent but different organizations; ["shanghai comprehensive diagnosis", "beijing", "wuhan"], thus generating a multi-round use case with an intent inheritance of 3 rounds; Intent switching: that is, obtaining use cases with the same organizational entity but different intents; ["shanghai single ticket capacity cost", "comprehensive diagnosis", "CCR"], thus generating a multi-round use case with an intent switching of 3 rounds; Inheritance & switching: it needs to include both inheritance and switching types, ["shanghai comprehensive diagnosis", "beijing", "how is the weather"], thus generating a multi-round use case with an inheritance & switching of 3 rounds; The special rule generation includes: No organization use case: when generating multi-round use cases, a part of the use cases is without organization. At this time, the use case can be used as intent switching data to join the multi-round use case. The following use case inherits the intent and the organization is the previous organization; Entity intent reset: in the use case type of inheritance & switching, when the length of the use case is greater than 7, the middle round will be a full use case. A sample use case with relatively more entities and intents will be selected to reset the previous entities and intents, verifying the reset ability of entities and intents; The randomization generation includes: randomly combining single round use cases that meet the conditions according to parameters to generate a specified number of multi-round use cases.

7. The logistics field large model context-based test method according to claim 6, characterized in that: In the generation by use case type, the first round of use cases must be full entities and intents. The following can appear full entities and intents, or the following entities and intents are not full. According to the corresponding use case type, the corresponding organization or intent is removed, and the corresponding number of use cases is generated according to this rule.

8. The logistics field large model context-based test method according to claim 7, characterized in that: The model success rate The calculation formula is: ； In the calculation formula, represents the success rate of the model; represents the total number of rounds of use cases; represents an indicator function, indicating whether the th task is successful, and the value is 1 when all rounds of the task pass the expected verification, otherwise the value is 0; represents the th task; represents the cumulative value of the indicators from the 1st task to the th task.

9. The logistics field large model context-based test method according to claim 8, characterized in that: The specific steps of the statistics failure distribution feedback are: S1, extract the failed multi-round task list; S2, obtain the detailed information of each failed task, including round sequence, failure round position, model error response, and expected correct result; S3, statistics of the distribution characteristics of failure cases, including intent understanding error, entity recognition error, context inheritance failure, irrelevant response; S4, classify by round position, and statistics of failure concentrated in which round; S5, classify by use case type, including intent inheritance, intent switching, inheritance & switching; S6, generate a feedback report for the above statistics and model success rate for feedback. 10.A logistics domain large model context-based test system, applied to the logistics domain large model context-based test method of any one of claims 1-9, characterized in that, It comprises a data acquisition module, a parameter configuration module, a processing module, a generation module, a calculation module, and a feedback module; The data acquisition module is used to acquire single-round successful test records and form a basic data set; The parameter configuration module is used to determine the parameters for generating multi-round use cases, and supports user-defined test scenarios; The processing module is used to classify single-round test records by entity and intent, and provide structured data for multi-round use case generation; The generation module is used to generate multi-round use cases according to parameters and rules; The calculation module is used to convert multi-round use cases into executable tasks, verify model responses round by round, calculate the overall success rate, and statistics of failure distribution; The feedback module is used to generate a feedback report for the statistics and model success rate for feedback.

Images