Test set creation method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention disclose a test set creation method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a test demand text; generating a description text of a to-be-created test set according to the test demand text through a preset generation model; wherein the description text comprises a total number of materials, a material mode, a material label and a material number corresponding to the material label; and obtaining a target material from a preset material pool according to the description text, and creating the to-be-created test set based on the target material. Low-threshold and high-efficiency test set generation can be realized, and the test set creation effect can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a test set creation method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, the test set is usually created based on the instructions input by the user. The existing method has high professional requirements for the user, low creation efficiency and poor effect. SUMMARY

[0003] The embodiments of the present disclosure provide a test set creation method and device, electronic equipment and storage medium, which can realize low-threshold and high-efficiency test set generation and improve the test set creation effect.

[0004] In a first aspect, the embodiments of the present disclosure provide a test set creation method, which comprises the following steps:

[0005] obtaining a test requirement text;

[0006] generating a description text of a test set to be created according to the test requirement text by using a preset generation model, wherein the description text comprises a total number of materials, a material modality, a material label and a number of materials corresponding to the material label;

[0007] obtaining target materials from a preset material pool according to the description text, and creating the test set to be created based on the target materials.

[0008] In a second aspect, the embodiments of the present disclosure further provide a test set creation device, which comprises:

[0009] an obtaining module configured to obtain a test requirement text;

[0010] a generating module configured to generate a description text of a test set to be created according to the test requirement text by using a preset generation model, wherein the description text comprises a total number of materials, a material modality, a material label and a number of materials corresponding to the material label;

[0011] a creating module configured to obtain target materials from a preset material pool according to the description text, and create the test set to be created based on the target materials.

[0012] In a third aspect, the embodiments of the present disclosure further provide an electronic equipment, which comprises:

[0013] one or more processors;

[0014] a storage device configured to store one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for creating a test set according to any of the embodiments of the present disclosure.

[0016] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions for executing the method for creating a test set according to any of the embodiments of the present disclosure when executed by a computer processor.

[0017] The technical solution of the embodiments of the present disclosure is to obtain a test requirement text; generate a description text of a test set to be created according to the test requirement text through a preset generation model; wherein the description text includes a total number of materials, a material mode, a material label, and a material quantity corresponding to the material label; obtain a target material from a preset material pool according to the description text, and create the test set to be created based on the target material.

[0018] By generating a description text of a test set to be created corresponding to different test requirements through a preset generation model, and determining a target material from a preset material pool based on the description text to create the test set, the automatic creation of the test set based on requirements can be realized, and the test set creation efficiency can be improved. By pre-setting the included items of the description text and determining the target material based on the description text, more boundary conditions and abnormal scenarios can be covered by the test set, and the test set creation effect can be improved. By extracting and combining material data from the preset material pool, a richer and more diversified test set can be created, and the test set creation effect can be further improved. In addition, the threshold can also be reduced, so that non-expert users can also effectively create the test set. That is, low-threshold and high-efficiency test set generation can be realized, and the test set creation effect can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0020] Figure 1 A flowchart of a method for creating a test set provided by the embodiments of the present disclosure;

[0021] Figure 2 A data flow schematic block diagram of a method for creating a test set provided by the embodiments of the present disclosure;

[0022] Figure 3 A flowchart of constructing a preset material pool in a method for creating a test set provided by the embodiments of the present disclosure;

[0023] Figure 4A data flow schematic block diagram for constructing a preset material pool in a test set creation method provided by an embodiment of the present disclosure;

[0024] Figure 5 A structural schematic diagram of a test set creation apparatus provided by an embodiment of the present disclosure;

[0025] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that each step described in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0032] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0033] Figure 1 A flowchart of a method for creating a test set provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case of creating a test set for model testing. The method can be performed by a test set creation device, which can be implemented in the form of software and / or hardware, and can be configured in an electronic device, such as a server of a test platform.

[0034] As shown in Figure 1 The method for creating a test set provided by the embodiment can include:

[0035] S110, obtaining a test requirement text.

[0036] In the embodiment of the present disclosure, the test requirement text can be considered as a natural language text for describing test requirements. The test requirements can be related to specific business scenarios, for example, can include object detection requirements, data classification requirements, and data generation requirements, etc. The test set creation device can provide a front-end user interface to collect user input test requirements. Then, the collected test requirements can be converted into a text mode to obtain the test requirement text.

[0037] The process of collecting test requirements may, for example, include at least one of the following: in response to the triggering of an audio collection control in the collection interface, collecting voice data of the test requirements input by the user; in response to the triggering of a text collection control in the collection interface, collecting text data of the test requirements input by the user; in response to the triggering of an image collection control in the collection interface, collecting image data of the test requirements input by the user. Then, the voice data of the test requirements can be processed through a voice recognition algorithm to convert it into test requirement text; the image data of the test requirements can be processed through an image recognition algorithm to convert it into test requirement text. The text data of the test requirements can be directly used as the test requirement text.

[0038] S120, generating a description text of a test set to be created according to the test requirement text through a preset generation model.

[0039] In the embodiments of the present disclosure, the description text of the to-be-created test set can include structural text for describing various parameters of the to-be-created test set. The description text can include a total number of materials (which can be defined as a total count parameter), a material type (which can be defined as a material type parameter), a material label and a number of materials corresponding to the material label (which can be defined as a map parameter composed of a label and a number), and the like. The total number of materials can be used to describe the total number of materials in the to-be-created test set. The material type can be used to describe the type of the materials in the to-be-created test set, and can include at least one of an image, text, audio, and video.

[0040] The same material can correspond to labels of different dimensions. For example, for an image material, the labels of the light and dark dimensions can be "light" and "dark", and the labels of the detection object dimension can be "cat" and "dog". In the embodiments, the material label can be understood as at least two labels in a dimension related to the test requirement. For example, assuming that the test requirement is face detection, the material label can include a male face label and a female face label. Correspondingly, the number of materials corresponding to the material label can describe the number of materials corresponding to each material label. It can be considered that the sum of the number of materials corresponding to each material label should be equal to the total number of materials.

[0041] In the embodiments of the present disclosure, the preset generation model can include a language model having the ability to generate a description text based on a test requirement text. The preset generation model can be constructed by using a sample test requirement text and a true value of the description text of the to-be-created test set, so that the preset generation model has the ability to generate the description text. Correspondingly, the completed preset generation model is used to generate the corresponding description text according to the obtained test requirement text.

[0042] In S130, target materials are obtained from a preset material pool according to the description text, and the to-be-created test set is created based on the target materials.

[0043] In the embodiments of the present disclosure, the various parameters of the description text can be parsed according to the preset structure of the description text to obtain parameter items including the total number of materials, the material type, the material label, and the number of corresponding materials. The parameter items can be used as an acquisition rule, and the corresponding target materials can be obtained from the preset material pool according to the acquisition rule. For example, a number of target materials corresponding to the total number of materials can be determined from the preset material pool. The type of the target materials should conform to the material type, the material label in the target materials should conform to the material label, and the number of target materials under each material label should conform to the corresponding number of materials.

[0044] After obtaining the target material according to the description text, the test set to be created can be created based on the target material, so as to obtain a test set meeting the test requirement. For example, the set of target materials can be taken as the test set; for another example, the target material can be processed to obtain the test set, and the like.

[0045] Since the preset structure of the description text can predefine more comprehensive parameter items, and the preset generation model constructed based on a large amount of data can generate more reasonable description texts, the target material determined based on the description text can cover more boundary conditions and abnormal scenarios, thereby improving the test set creation effect.

[0046] In some optional implementation manners, the material modality includes an image and / or a video; and the corresponding material label includes an attribute label; wherein the attribute label includes at least one of the following: resolution, format and color histogram.

[0047] The attribute label can be considered as a label of the attribute of the material itself, and this kind of label can be directly determined according to the material data without the need of logical inference according to the labeling model. For example, for an image and / or a video, the resolution (i.e. size) and format labels can be directly obtained from the related attribute information; the color histogram can also be directly determined according to the pixels of the image or video frame. In addition, other attributes determined according to the material data can also be included in the attribute label, which is not enumerated here.

[0048] In these optional implementation manners, for the processing model of the image and / or the video, the attribute of the material needs to be limited in some test scenarios, and at this time, the material label can include the attribute label, so that the created test set meets the test requirement.

[0049] In some optional implementation manners, the description text can also include test set information and / or material richness indicators.

[0050] In these implementation manners, the test set information can include at least one of the following: test set type, test set label and test set introduction.

[0051] The test set type (which can be defined as an evaluation set type parameter, for example) can include an externally created class and a platform created class. The externally created class can be understood as a type of test set created by a user of the test platform; the platform created class can be understood as a type of test set created by the test platform party. In addition, the platform created class can also include a normal class and a recommended class, and the like.

[0052] The creating device can provide a first creating interface of the test set for a user of the test platform, and can provide a second creating interface of the test set for a party of the test platform. The test set created based on the first creating interface can be considered as a specific test set created by the user of the test platform based on the test requirement of the user, and the specific test set can be bound to a user account for maintenance. The test set created based on the second creating interface can be considered as a general test set created and maintained by the party of the test platform, and the general test set can be used by the user of the test platform. The recommended test set can be constructed from the general test set with high usage, and the like. It can be considered that the recommended test set can be updated according to the usage of the general test set within a preset period of time from the current preset period of time. For example, in a case where the number of times that the user of the test platform uses the general test set A within a preset period of time from the current preset period of time is greater than a preset number of times, the general test set A can be updated to the recommended test set A for the user to preferentially select the test set. By setting the parameter item of the type of the test set in the description text, the test set can be conveniently maintained later.

[0053] The test set label (for example, which can be defined as a custom labels parameter) can be automatically generated based on the test requirement text, for example, can be determined based on the keyword of the test requirement text, and the like. For example, if the test requirement is an object detection requirement, the test set label can be an object detection test set, and the like. By setting the parameter item of the test set label in the description text, the test set can be conveniently located and positioned.

[0054] The test set introduction (for example, which can be defined as a description parameter) can include a field for describing the test set, which is customized by the user, for example, a field for describing the creation version and the creation time of the test set, and the like. By setting the parameter item of the test set introduction in the description text, the test set can be conveniently managed.

[0055] In addition, other parameter items related to the test set can also be included in the test set information, which is not exhaustively listed herein.

[0056] In the implementation manners, the material richness index can include a richness requirement for other labels, the other labels being labels other than the material label in at least one dimension label of the target material.

[0057] The richness requirement for the other labels can include, but is not limited to, a dimension requirement for the other labels, a requirement for the category and number of labels under each dimension, and the like. In the test process, it is expected that the material label is used as a label of a substantial dimension for dividing the type of the target material, and the labels of other dimensions (i.e., the other labels) other than the material label need to be as rich as possible to avoid the influence of the other labels on the test process.

[0058] For example, for object detection requirements, it is desirable to use the material labels "cat" and "dog" as the substantive dimension labels for classifying target material types. For materials belonging to "cat" or "dog", it is desirable that their corresponding other labels include both "bright" and "dark" labels to avoid the influence of brightness dimension labels on object detection.

[0059] Setting material abundance indicators in the description text can ensure the rationality of the test set and improve the effectiveness of test set creation.

[0060] The technical solution of this disclosure embodiment is to obtain test requirement text; generate description text of the test set to be created based on the test requirement text using a preset generation model; wherein, the description text includes the total number of materials, material modality, material label and the quantity of materials corresponding to the material label; obtain the target material from the preset material pool based on the description text, and create the test set to be created based on the target material.

[0061] By using a preset generation model to generate descriptive text for corresponding test sets based on different testing requirements, and then determining the target material from a preset material pool based on the descriptive text, the test set can be created automatically based on requirements, improving test set creation efficiency. By pre-setting the included items in the descriptive text and determining the target material based on the descriptive text, the test set can cover more boundary cases and abnormal scenarios, improving the test set creation effect. By extracting and combining material data from the preset material pool, richer and more diverse test sets can be created, further improving the test set creation effect. In addition, it lowers the barrier to entry, allowing non-expert users to effectively create test sets. In short, it achieves low-threshold, high-efficiency test set generation and improves the test set creation effect.

[0062] This disclosure embodiment can be combined with various optional solutions in the test set creation method provided in the above embodiments. The test set creation method provided in this embodiment details the test set creation process. By validating and / or modifying the generated description text, the rationality and accuracy of the description text can be ensured; by prompting for modification of the description text in the event of target material failure, successful test set creation is facilitated; by adding, deleting, modifying, and querying target materials, the quality of test set creation is improved; and by previewing and / or visually analyzing the created test set, the user experience is enhanced.

[0063] For example, Figure 2 This is a data flow diagram illustrating a method for creating a test set provided in an embodiment of this disclosure.

[0064] See Figure 2In some optional implementations, after the description text of the test set to be created is generated, the following operations can be further included: checking the description text; wherein the checking includes at least one of the following: integrity checking, data correspondence checking, and label correctness checking; and / or, displaying the description text; and in response to a modification instruction, modifying the description text.

[0065] In these implementations, after the description text is generated, the description text can be checked, and the target materials can be acquired after the checking is successful.

[0066] The integrity checking can include checking the parameter items of the preset structure in the description text. In the case where there are parameter items missing values, the integrity checking of the description text can be considered to fail. At this time, the user can be prompted to continue inputting the test requirement text, so as to generate a complete description text through the preset generation model. In the case where all the parameter items are assigned values, the integrity checking of the description text can be considered to be successful, and subsequent operations can be performed.

[0067] The data correspondence checking can include checking whether the sum of the material quantities under each material label is equal to the total number of materials. In the case where the sum of the material quantities is not equal to the total number of materials, the data correspondence checking can be considered to fail. At this time, the material quantity and / or the total number of materials can be automatically adjusted to make them equal. In the case where the sum of the material quantities is equal to the total number of materials, the data correspondence checking can be considered to be successful, and subsequent operations can be performed.

[0068] The label correctness checking can include checking whether the material label belongs to the label enumeration set corresponding to the preset material pool. In the case where the material label does not belong to the label enumeration set, the label correctness checking can be considered to fail. At this time, the user can be prompted to continue inputting the test requirement text, so as to generate a reasonable description text through the preset generation model. In the case where the material label belongs to the label enumeration set, the label correctness checking can be considered to be successful, and subsequent operations can be performed.

[0069] In addition, in these implementations, after the description text is generated, the description text can also be modified, and the target materials can be acquired after the modification is complete.

[0070] After each parameter item in the description text is parsed, each parameter item can be displayed on the first preview interface. The modification instruction can be received through the first preview interface, so as to modify the corresponding parameter item according to the modification instruction. After the modification is completed, the confirmation instruction can be received, so as to update the description text according to the modified parameter item, thereby realizing the modification of the description text.

[0071] In these optional implementations, the description text can be considered to be checked and / or modified after being generated, so as to ensure the rationality and accuracy of the description text.

[0072] Referring back to Figure 2 In some optional implementations, before creating the test set to be created based on the target materials, the method can further include: in a case where the target materials are failed to be acquired, generating prompt information for modifying the description text.

[0073] In a case where the description text is successfully parsed, there can still be a case where the target materials are failed to be acquired according to the description text. For example, a case where the number of acquired target materials is less than the total number of materials; for another example, a case where the number of target materials under a part of the material labels is less than the corresponding number of materials, and the like.

[0074] In a case where the target materials are failed to be acquired, prompt information for modifying the description text can be generated, and the prompt information can be displayed in a pop-up control or a prompt interface. Correspondingly, the first preview interface can be jumped to in response to a trigger of a confirmation control in the pop-up control or the prompt interface. At this time, in the first preview interface, a parameter item corresponding to the failure case can be prompted for adjustment (for example, “please set the value below × × ” and the like); correspondingly, a user can input a modification instruction according to the adjustment prompt to modify the corresponding parameter item.

[0075] In these optional implementations, in a case where the target materials are failed to be acquired, the description text can be prompted to be modified, thereby facilitating the successful creation of the test set.

[0076] In some optional implementations, the creating the test set to be created based on the target materials can include: performing at least one of the following processing operations on the target materials: adding materials, deleting materials, modifying materials, and querying materials.

[0077] In these optional implementations, after the target materials are successfully acquired, the target materials can be displayed in the second preview interface. And at least one of the following instructions can be received through the second preview interface: an adding material instruction, a deleting material instruction, a modifying material instruction, and a querying material instruction. Correspondingly, the corresponding adding material, deleting material, modifying material, or querying material processing operation can be performed according to the received instruction. By adding, deleting, modifying, and querying the target materials, the creation quality of the test set can be improved.

[0078] Referring back to Figure 2 In some optional implementations, after the test set to be created is created based on the target materials, the method can further include: previewing the created test set; and / or analyzing the created test set, and visualizing the analysis result.

[0079] In the optional implementation, after the test set is successfully created, the test set can also be displayed in a third preview interface. For example, in the third preview interface, the target data can be displayed according to the material label, and / or the test set can be analyzed in terms of coverage and distribution, and the analysis result can be displayed through a visual means such as an icon. The analysis result can also be presented in the third preview interface, or can be presented in a sub-page of the third preview interface. Through preview and / or visual analysis of the created test set, the user experience can be improved.

[0080] In addition, the test set and the corresponding description text can be stored correspondingly. Accordingly, the third preview interface can be provided with an entry of the first preview interface. Through triggering of the entry, the third preview interface can jump to the first preview interface. In the case that the test requirement is slightly changed, after the third preview interface jumps to the first preview interface, the description text can be modified through the first preview interface to obtain new target materials, and the version of the test set is updated according to the new target materials, so that the test set can be conveniently and quickly maintained.

[0081] The technical solution of the embodiment of the present disclosure describes the creation process of the test set in detail. By checking and / or modifying the description text after it is generated, the rationality and accuracy of the description text can be ensured; by prompting to modify the description text in the case of target material failure, the successful creation of the test set can be facilitated; by adding, deleting, modifying and searching the target material, the creation quality of the test set can be improved; by previewing and / or visualizing the created test set, the user experience can be improved.

[0082] In addition, the test set creation method provided by the embodiment of the present disclosure belongs to the same disclosure concept as the test set creation method provided by the above-mentioned embodiment, and the technical details not described in detail in the present embodiment can be referred to the above-mentioned embodiment, and the same technical features have the same beneficial effects in the present embodiment and the above-mentioned embodiment.

[0083] The test set creation method provided by the embodiment of the present disclosure can be combined with the test set creation method provided by the above-mentioned embodiment. The test set creation method provided by the present embodiment describes the construction of the preset material pool in detail. By performing data preprocessing, the data quality of the preset material pool can be improved; by performing data labeling in multiple ways, a fully labeled material pool can be obtained.

[0084] Figure 3 A flowchart of constructing a preset material pool in a test set creation method provided by the embodiment of the present disclosure is shown in FIG. 1. Figure 3 As shown in FIG. 1, in the test set creation method provided by the present embodiment, the preset material pool is constructed based on the following process:

[0085] S310, pre-process the collected source data to obtain first data.

[0086] In this embodiment, data can be generated based on a data generation algorithm. It can be understood that the data (including but not limited to the data itself, acquisition or use of the data) involved in the embodiments of the present disclosure should comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0087] After the source data is collected, the source data can be pre-processed. The pre-processing can include deduplication processing and / or processing to remove abnormal data. The similarity of the source data can be calculated based on a similarity calculation method, and the source data with a similarity greater than a preset threshold can be deduplicated.

[0088] The division of abnormal data can be different for different modalities. For example, for text data, abnormal data can include data with special characters; for image / video data, abnormal data can include data that cannot be normally displayed due to data loss or format abnormalities. After identifying the abnormal data, the abnormal data can be removed.

[0089] S320, label the first data, and construct a preset material pool according to the first data and the corresponding label.

[0090] In the embodiments of the present disclosure, the pre-processed source data can be referred to as first data. The first data can be labeled in multiple dimensions based on an automatic labeling method. For some special businesses, a labeling interface can be provided for users to input business labels of the first data, so as to ensure the correctness of the labels.

[0091] By labeling the first data and constructing a preset material pool according to the first data and the corresponding label, a material pool with rich labels and a number of orders that can meet the needs of different test sets can be generated.

[0092] Exemplarily, Figure 4 A data flow schematic diagram for constructing a preset material pool in a test set creation method provided by the embodiments of the present disclosure. Referring to Figure 4Firstly, source data can be collected through multiple channels; then, the collected source data can be preprocessed to obtain first data; secondly, the first data can be transmitted to a manual labeling interface through a labeling gateway for labeling of business labels; thirdly, the labeled business labels can be recalled and stored in a platform-related database (for example, a Redis database); then, the first data of different modalities and corresponding business labels can be transmitted to an automatic labeling interface in the form of ordered queue distribution for automatic labeling; finally, the first data and corresponding dimensional labels can also be stored in the form of a queue to construct a preset material pool. Thus, a rich and fully labeled material pool can be obtained to provide material sources for creation of a test set.

[0093] The technical solutions of the embodiments of the present disclosure are described in detail for the construction of the preset material pool. Through data preprocessing, the data quality of the preset material pool can be improved; through data labeling in multiple ways, a fully labeled material pool can be obtained. In addition, the test set creation method provided by the embodiments of the present disclosure belongs to the same disclosure concept as the test set creation method provided by the above embodiments, and the technical details not described in detail in the present embodiment can be referred to the above embodiments, and the same technical features have the same beneficial effects in the present embodiment and the above embodiments.

[0094] Figure 5 A structural schematic diagram of a test set creation device provided by the embodiments of the present disclosure is shown. The test set creation device provided by the present embodiment is suitable for the case of creating a test set for model testing.

[0095] As shown in Figure 5 , the test set creation device provided by the embodiments of the present disclosure can include:

[0096] The acquisition module 510 is configured to acquire a test requirement text.

[0097] The generation module 520 is configured to generate a description text of a test set to be created according to the test requirement text by using a preset generation model; wherein the description text includes a total number of materials, a material modality, a material label, and a number of materials corresponding to the material label.

[0098] The creation module 530 is configured to acquire target materials from a preset material pool according to the description text, and create the test set to be created based on the target materials.

[0099] In some optional implementation manners, the test set creation device can further include:

[0100] The description text processing module is configured to verify the description text after generating the description text of the test set to be created; wherein the verification includes at least one of the following: completeness verification, data correspondence verification, and label correctness verification.

[0101] and / or, presenting the description text; modifying the description text in response to the modification instruction.

[0102] In some optional implementations, the test set creation apparatus can further include:

[0103] The prompt module is configured to, in a case where the target material fails to be acquired, generate prompt information for modifying the description text before creating the test set to be created based on the target material.

[0104] In some optional implementations, the creation module is configured to perform at least one of the following on the target material: adding a material, deleting a material, modifying a material, and querying a material.

[0105] In some optional implementations, the test set creation apparatus can further include:

[0106] The presentation module is configured to, after creating the test set to be created based on the target material, preview the created test set; and / or analyze the created test set and visualize the analysis result.

[0107] In some optional implementations, the material modal includes an image and / or a video; and the corresponding material label includes an attribute label.

[0108] The attribute label includes at least one of the following: resolution, format, and color histogram.

[0109] In some optional implementations, the description text further includes test set information and / or a material richness indicator.

[0110] The test set information includes at least one of the following: test set type, test set label, and test set introduction; and the material richness indicator includes a richness requirement for other labels, the other labels being labels other than the material label in at least one dimension label of the target material.

[0111] In some optional implementations, the test set creation apparatus can further include:

[0112] The material pool construction module is configured to construct a preset material pool based on the following process:

[0113] The source data collected is preprocessed to obtain first data; the preprocessing includes de-duplication processing and / or processing to remove abnormal data.

[0114] The first data is labeled, and the preset material pool is constructed according to the first data and the corresponding label.

[0115] The test set creation apparatus provided in this disclosure can execute the test set creation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0116] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0117] The following is for reference. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 6 The diagram below shows the structure of the terminal device or server 600. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0118] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0119] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6The electronic device 600 is shown with various elements, but it will be understood that not all of the elements shown need be present in every implementation or embodiment. Some elements shown can be omitted in some implementations or embodiments, and other elements can be added to the electronic device 600.

[0120] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the creation method of the test set of embodiments of the present disclosure are executed.

[0121] The electronic device provided by the embodiments of the present disclosure and the creation method of the test set provided by the above-mentioned embodiments belong to the same disclosure concept, and the technical details not described in detail in the present embodiment can be referred to the above-mentioned embodiments, and the present embodiment has the same beneficial effects as the above-mentioned embodiments.

[0122] The embodiments of the present disclosure provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the creation method of the test set provided by the above-mentioned embodiments.

[0123] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take any of a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including, but not limited to, wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0124] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0125] The computer-readable medium described above can be included in the electronic device; alternatively, the computer-readable medium can exist as a separate entity, which is not incorporated into the electronic device.

[0126] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0127] obtain a test requirement text; generate, by a preset generation model, a description text of a to-be-created test set according to the test requirement text; wherein the description text comprises a total number of materials, a material mode, a material label, and a material quantity corresponding to the material label; and obtain target materials from a preset material pool according to the description text, and create the to-be-created test set based on the target materials.

[0128] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ as well as conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0129] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.

[0130] The units involved in the embodiments of the present disclosure can be implemented in a software manner, or can be implemented in a hardware manner. In some cases, the names of the units and modules do not constitute a limitation on the units and modules themselves.

[0131] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] According to one or more embodiments of the present disclosure, a test set creation method is provided, which comprises:

[0134] Obtaining a test requirement text;

[0135] Generating, by a preset generation model, a description text of a test set to be created according to the test requirement text; wherein the description text comprises a total number of materials, a material modality, a material label, and a number of materials corresponding to the material label;

[0136] According to the description text, obtaining a target material from a preset material pool, and creating the test set to be created based on the target material.

[0137] According to one or more embodiments of the present disclosure, a test set creation method is provided, which further comprises:

[0138] In some optional implementations, after the description text of the test set to be created is generated, the method further comprises:

[0139] verifying the description text, wherein the verifying comprises at least one of the following: integrity verification, data correspondence verification, and label correctness verification;

[0140] and / or, presenting the description text; and modifying the description text in response to a modification instruction.

[0141] According to one or more embodiments of the present disclosure, a test set creation method is provided, further comprising:

[0142] In some optional implementations, before the creating the test set to be created based on the target material, further comprising:

[0143] In a case where the target material fails to be acquired, generating prompt information for modifying the description text.

[0144] According to one or more embodiments of the present disclosure, a test set creation method is provided, further comprising:

[0145] In some optional implementations, the creating the test set to be created based on the target material comprises:

[0146] The target material is subjected to at least one of the following treatments: adding a material, deleting a material, modifying a material, and querying a material.

[0147] According to one or more embodiments of the present disclosure, a test set creation method is provided, further comprising:

[0148] In some optional implementations, after the creating the test set to be created based on the target material, further comprising:

[0149] previewing the created test set;

[0150] and / or, analyzing the created test set and visualizing the analysis result.

[0151] According to one or more embodiments of the present disclosure, a test set creation method is provided, further comprising:

[0152] In some optional implementations, the material modality comprises an image and / or a video; and correspondingly, the material label comprises an attribute label.

[0153] The attribute label comprises at least one of the following: resolution, format, and color histogram.

[0154] According to one or more embodiments of the present disclosure, a test set creation method is provided, further comprising:

[0155] In some optional implementations, the description text further comprises test set information and / or a material richness index.

[0156] The test set information comprises at least one of a test set type, a test set label, and a test set introduction; and the material richness index comprises a richness requirement for other labels, the other labels being labels other than the material label in at least one dimension label of the target material.

[0157] According to one or more embodiments of the present disclosure, a test set creation method is provided, further comprising:

[0158] In some optional implementations, the preset material pool is constructed based on the following process:

[0159] The collected source data is preprocessed to obtain first data; the preprocessing comprises de-duplication processing and / or processing to remove abnormal data.

[0160] The first data is labeled, and the preset material pool is constructed according to the first data and the corresponding label.

[0161] According to one or more embodiments of the present disclosure, a test set creation device is provided, comprising:

[0162] An acquisition module is configured to acquire a test requirement text.

[0163] A generation module is configured to generate, by a preset generation model, a description text of a test set to be created according to the test requirement text; the description text comprises a total number of materials, a material modality, a material label, and a number of materials corresponding to the material label.

[0164] A creation module is configured to acquire a target material from a preset material pool according to the description text, and create the test set to be created based on the target material.

[0165] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology applied. It should be understood by those skilled in the art that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.

[0166] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the disclosure. Certain of the operations described in the discussion are combinable into a single operation, and certain operations can be separated into several operations. In some embodiments, the operations described in the discussion can be performed in an order different than presented in the discussion. In some embodiments, the operations described in the discussion can be performed concurrently. Also, while several specific implementation details are discussed in the discussion, these should not be interpreted as limiting the scope of the disclosure. Rather, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0167] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method of creating a test set, characterized by, The method comprises: obtaining a test requirement text; generating a description text of a test set to be created according to the test requirement text by using a preset generation model, wherein the description text comprises a total number of materials, a material modality, a material label, and a material quantity corresponding to the material label; obtaining a target material from a preset material pool according to the description text, and creating the test set to be created based on the target material.

2. The method of claim 1, wherein, After the description text of the test set to be created is generated, the method further comprises: verifying the description text, wherein the verification comprises at least one of the following: completeness verification, data correspondence verification, and label correctness verification; and / or, displaying the description text; and in response to a modification instruction, modifying the description text.

3. The method of claim 1, wherein, Before the test set to be created is created based on the target material, the method further comprises: in a case where the target material fails to be obtained, generating prompt information for modifying the description text.

4. The method of claim 1, wherein, The creating of the test set to be created based on the target material comprises: performing at least one of the following on the target material: adding a material, deleting a material, modifying a material, and querying a material.

5. The method of claim 1, wherein, After the test set to be created is created based on the target material, the method further comprises: previewing the created test set; and / or, analyzing the created test set, and visualizing an analysis result.

6. The method of claim 1, wherein, The material modality comprises an image and / or a video; and correspondingly, the material label comprises an attribute label. The attribute label comprises at least one of the following: resolution, format, and color histogram.

7. The method of claim 1, wherein, The description text further comprises test set information and / or a material richness index. The test set information comprises at least one of the following: a test set type, a test set label, and a test set introduction; and the material richness index comprises a richness requirement for other labels, the other labels being labels other than the material label in at least one dimension label of the target material.

8. The method according to any one of claims 1 to 7, characterized in that, The preset material pool is constructed based on the following process: preprocessing collected source data to obtain first data, wherein the preprocessing comprises de-duplication processing and / or processing to remove abnormal data; annotating the first data, and constructing the preset material pool based on the first data and the corresponding annotation.

9. An apparatus for creating a test set, characterized by The method comprises: an obtaining module configured to obtain a test requirement text; a generating module configured to generate a description text of a test set to be created according to the test requirement text by using a preset generation model, wherein the description text comprises a total number of materials, a material modality, a material label, and a material quantity corresponding to the material label; a creating module configured to obtain a target material from a preset material pool according to the description text, and create the test set to be created based on the target material.

10. An electronic device, comprising: The electronic device comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for creating a test set according to any one of claims 1-8.

11. A storage medium containing computer-executable instructions for performing a method of creating a test set as claimed in any one of claims 1-8 when executed by a computer processor.

Images