Testing methods, systems, and electronic equipment for intelligent sensing systems

Abstract

The present invention provides a testing method, system, and electronic device for an intelligent perception system. The intelligent perception system can output a recognition result for a target based on input data to be recognized, wherein the data to be recognized includes a scene and a target contained in the scene. The method includes: obtaining multiple test samples, wherein each sample includes a score, a label, and first data of the sample; for each sample, determining whether the recognition result is correct to obtain a first score; and summing the first scores of all the samples and dividing the sum by the sum of the scores of all the samples to obtain a second score used to evaluate the perception capability of the intelligent perception system. The present invention can provide an objective and quantitative score for the recognition capability of the intelligent perception system and can comprehensively assess the recognition capability of the intelligent perception system from various scene type levels.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to a testing method, system, and electronic equipment for an intelligent sensing system. Background Art

[0002] Intelligent perception systems can identify specific targets and have widespread applications in many fields. For example, in surveillance, intelligent perception systems can quickly identify specific individuals in video footage captured by cameras. With the development of neural networks and computer hardware, the capabilities of intelligent perception systems have rapidly improved, and their application has expanded to various industries. However, it is difficult to determine whether the recognition capabilities of intelligent perception systems meet the requirements in every specific scenario. Therefore, an accurate assessment of the recognition capabilities of intelligent perception systems is necessary.

[0003] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present invention provides a testing method, system, and electronic equipment for an intelligent sensing system.

[0005] The present invention provides a testing method for an intelligent perception system, wherein the intelligent perception system is capable of outputting a recognition result of a target based on input data to be recognized, wherein the data to be recognized includes a scene and a target contained in the scene. The method comprises:

[0006] Acquire a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample;

[0007] For each sample, perform the following steps:

[0008] a. Based on the first data, generating second data to be identified through scenario-based presentation;

[0009] b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system;

[0010] c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample;

[0011] The first scores of all the samples are summed up and then divided by the sum of the scores of all the samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system.

[0012] According to a testing method for an intelligent perception system provided by the present invention, a plurality of test samples are obtained, including:

[0013] Obtain basic first data, and set basic labels and basic scores in a targeted manner to form a basic sample;

[0014] The data generator generates expansive first data based on the input of the basic first data, and sets expansive labels and expansive scores in a targeted manner to form an expansive sample.

[0015] According to a testing method for an intelligent perception system provided by the present invention, the setting of the scalability score includes:

[0016] Obtaining the validity of the data generator;

[0017] The product of the effectiveness and the basic score is used as the scalability score.

[0018] According to a testing method for an intelligent perception system provided by the present invention, different scores are set for the multiple test samples based on differences in sample types.

[0019] According to a testing method for an intelligent perception system provided by the present invention, the method further includes:

[0020] Classifying the samples according to their scores;

[0021] For the samples of different categories, the perception capabilities of the intelligent perception system for samples of different categories are evaluated in order from low to high scores.

[0022] According to a testing method for an intelligent perception system provided by the present invention, the method further includes:

[0023] For the first batch of test samples, obtain the second score corresponding to the first batch;

[0024] adding additional test samples to the first batch of test samples to form a second batch of test samples;

[0025] For the second batch of test samples, obtaining a second score corresponding to the second batch;

[0026] If the second scores corresponding to the second batches are equal to the second scores corresponding to the first batch, the second scores corresponding to the second batches are used to evaluate the perception ability of the intelligent perception system; if the second scores corresponding to the second batches are not equal to the second scores corresponding to the first batch, it is considered that the perception ability score of the intelligent perception system has not yet stabilized, and additional test samples are added and re-scored.

[0027] According to a testing method for an intelligent perception system provided by the present invention, the first data includes a scene-based real image;

[0028] Correspondingly, through scenario presentation, including:

[0029] Acquire all physical objects in the scene, combine them according to the real images, and provide them to the intelligent perception system for identification;

[0030] and / or,

[0031] directly providing the real image to the intelligent perception system for recognition;

[0032] and / or,

[0033] Part of the physical objects in the scene are obtained, and the missing physical objects are supplemented by virtual reality technology, and provided to the intelligent perception system for recognition.

[0034] The present invention also provides a testing system for an intelligent perception system, wherein the intelligent perception system is capable of outputting a recognition result of a target based on input data to be recognized, wherein the data to be recognized includes a scene and a target contained in the scene. The system comprises:

[0035] A sample module, configured to obtain a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample;

[0036] The loop module is used to perform the following steps for each sample:

[0037] a. Based on the first data, generating second data to be identified through scenario-based presentation;

[0038] b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system;

[0039] c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample;

[0040] The evaluation module is used to add up the first scores of all the samples and then divide the sum by the sum of the scores of all the samples to obtain a second score for evaluating the perception ability of the intelligent perception system.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the test method for the intelligent perception system as described in any one of the above items are implemented.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the test method for the intelligent perception system as described in any one of the above items are implemented.

[0043] The testing method, system, and electronic device for the intelligent perception system provided by the present invention can give an objective quantitative score to the recognition ability of the intelligent perception system, and adopt scenario-based test samples, so that the test samples can be easily expanded and acquired, and the recognition ability of the intelligent perception system can be comprehensively examined from various scenario type levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flow chart of a testing method for an intelligent perception system provided by the present invention;

[0046] Figure 2 A schematic diagram of a process for testing an intelligent perception system provided by the present invention;

[0047] Figure 3 A schematic diagram of the structure of a test system for an intelligent sensing system provided by the present invention;

[0048] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] The following describes in detail the testing method for the intelligent perception system provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0051] Figure 1 A flow chart of a testing method for an intelligent sensing system provided by the present invention is shown as follows: Figure 1 As shown, the present invention provides a testing method for an intelligent perception system. The intelligent perception system can output recognition results about the target based on input data to be recognized, where the data to be recognized includes a scene and a target contained in the scene. The method includes the following steps.

[0052] Preferably, the present invention is applicable to the field of image processing, that is, the data to be recognized is in image format.

[0053] Preferably, the present invention is applicable to the field of sound processing, that is, the data to be recognized is in audio format.

[0054] S100: Acquire multiple test samples, where each sample includes a sample score, a label, and first data;

[0055] Optionally, multiple test samples are obtained, including:

[0056] Obtain basic first data, and set basic labels and basic scores in a targeted manner to form a basic sample;

[0057] Through the data generator, based on the input of the basic first data, the expansive first data is generated, and the expansive label and expansive score are set in a targeted manner to form an expansive sample.

[0058] Preferably, the data generator is capable of synthesizing virtual third spatiotemporal feature data based on the physical first spatiotemporal feature data and the second spatiotemporal feature data, wherein the first spatiotemporal feature data includes a first target and a first background, the second spatiotemporal feature data includes a second target and a second background, and the third spatiotemporal feature data includes the first target and the second background.

[0059] Optionally, the scalability score setting includes:

[0060] Obtaining the validity of the data generator;

[0061] The product of effectiveness and basic score is taken as the scalability score.

[0062] Preferably, the validity value of the data generator is in the interval [0, 1]. For example, if the validity η = 0.8 and the basic score of basic sample i is 10 points, then the expansibility score of the expansibility sample j obtained based on the basic sample i by the data generator is 8 points.

[0063] Optionally, for multiple test samples, different scores are set for the samples based on differences in sample types.

[0064] Preferably, samples can be differentiated by type (difficulty, complexity, etc.) and can be assigned different scores.

[0065] S200: For each sample, perform the following steps:

[0066] a. Based on the first data, second data to be identified is generated through scenario-based presentation;

[0067] b. inputting the second data into the intelligent perception system and obtaining a recognition result output by the intelligent perception system;

[0068] c. Compare the recognition result with the label to determine whether the recognition result is correct. If the recognition is correct, the score is used as the first score of the intelligent perception system for the current sample. If the recognition is incorrect, zero is used as the first score of the intelligent perception system for the current sample.

[0069] S300: Add up the first scores of all samples and divide the sum by the sum of the scores of all samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system.

[0070] Figure 2 A flow chart of a test intelligent perception system provided by the present invention is as follows: Figure 2 As shown, traverse N samples, and for each sample, there is a score R i (i.e., the first score), the cumulative score of all samples is divided by the sum of the sample scores to obtain the second score S.

[0071] The present invention can give an objective quantitative score to the recognition ability of the intelligent perception system, and adopts scenario-based test samples, so that the test samples can be easily expanded and acquired, and the recognition ability of the intelligent perception system can be comprehensively examined from various scenario types and levels.

[0072] Optionally, the method further comprises:

[0073] Classify samples according to their scores;

[0074] For samples of different categories, the perception capabilities of the intelligent perception system are evaluated in order from low to high scores.

[0075] Preferably, the test samples can be graded according to their difficulty, with scores given in grades from simple to difficult. After passing the simple grade test, the next grade with a more difficult one can be tested, thereby determining the gear of the intelligent perception system.

[0076] Optionally, the method further comprises:

[0077] For the first batch of test samples, obtain the second score corresponding to the first batch;

[0078] adding test samples to the first batch of test samples to form a second batch of test samples;

[0079] For the second batch of test samples, obtain a second score corresponding to the second batch;

[0080] If the second score corresponding to the second batch is equal to the second score corresponding to the first batch, the second score corresponding to the second batch is used to evaluate the perception ability of the intelligent perception system; if the second score corresponding to the second batch is not equal to the second score corresponding to the first batch, it is considered that the perception ability score of the intelligent perception system has not yet stabilized, and additional test samples will be added and re-scored.

[0081] Appropriately increase the number of test samples. If the second score stabilizes and no longer increases, use the second score as the final score.

[0082] Optionally, the first data includes a scene-based real image;

[0083] Correspondingly, through scenario presentation, including:

[0084] Obtain all physical objects in the scene, combine them according to real images, and provide them to the intelligent perception system for identification;

[0085] and / or,

[0086] Provide real images directly to intelligent perception systems for recognition;

[0087] and / or,

[0088] Obtain some physical objects in the scene, use virtual reality technology to complete the missing physical objects, and provide them to the intelligent perception system for recognition.

[0089] Preferably, the first data may also be video, and further extended to other data in the optical field.

[0090] Preferably, the first data can also be data from the acoustics field, such as enabling the intelligent perception system to simulate hearing and distinguish audible sounds; the first data can also be data from the chemistry field, such as enabling the intelligent perception system to simulate smell and distinguish the odor of gases; the first data can also be data from the mechanics field, such as enabling the intelligent perception system to simulate touch, the source of discrimination; the first data can also be data from the biochemistry field, such as enabling the intelligent perception system to simulate taste and distinguish the flavor of substances. Accordingly, the second data is obtained by presenting the first data in a scenario-based manner. The scenario-based process can be considered as adding various interference factors of the same type to the first data, or adding noise that affects the intelligent perception system's discrimination.

[0091] The test system for the intelligent perception system provided by the present invention is described below. The test system for the intelligent perception system described below and the test method for the intelligent perception system described above can be referenced to each other.

[0092] Figure 3 This is a structural diagram of a test system for an intelligent sensing system provided by the present invention, such as Figure 3 As shown, the present invention also provides a test system for an intelligent perception system. The intelligent perception system can output a recognition result about a target based on input data to be recognized, where the data to be recognized includes a scene and a target contained in the scene. The system includes:

[0093] A sample module, configured to obtain a plurality of test samples, wherein each sample includes a sample score, a label, and first data;

[0094] The loop module is used to perform the following steps for each sample:

[0095] a. Based on the first data, second data to be identified is generated through scenario-based presentation;

[0096] b. inputting the second data into the intelligent sensing system and obtaining a recognition result output by the intelligent sensing system;

[0097] c. Compare the recognition result with the label to determine whether the recognition result is correct. If the recognition result is correct, the score is used as the first score of the intelligent perception system for the current sample. If the recognition result is incorrect, zero is used as the first score of the intelligent perception system for the current sample.

[0098] The evaluation module is used to add up the first scores of all samples and then divide it by the sum of the scores of all samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system.

[0099] This embodiment can give an objective quantitative score to the recognition ability of the intelligent perception system, and adopts scenario-based test samples, so that the test samples can be easily expanded and acquired, and the recognition ability of the intelligent perception system can be comprehensively examined from various scenario type levels.

[0100] Figure 4 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a test method for an intelligent perception system, wherein the intelligent perception system is capable of outputting a recognition result about the target based on input data to be recognized, the data to be recognized including a scene and a target contained in the scene. The method includes:

[0101] Acquire a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample;

[0102] For each sample, perform the following steps:

[0103] a. Based on the first data, generating second data to be identified through scenario-based presentation;

[0104] b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system;

[0105] c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample;

[0106] The first scores of all the samples are summed up and then divided by the sum of the scores of all the samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system.

[0107] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0108] On the other hand, the present invention further provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions. When the program instructions are executed by a computer, the computer can perform the testing method for the intelligent perception system provided by the above methods. The intelligent perception system can output a recognition result about the target based on input data to be recognized, the data to be recognized comprising a scene and a target contained in the scene. The method comprises:

[0109] Acquire a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample;

[0110] For each sample, perform the following steps:

[0111] a. Based on the first data, generating second data to be identified through scenario-based presentation;

[0112] b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system;

[0113] c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample;

[0114] The first scores of all the samples are summed up and then divided by the sum of the scores of all the samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system.

[0115] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program is implemented to perform the above-mentioned testing methods for the intelligent perception system. The intelligent perception system can output a recognition result about the target based on input data to be recognized, the data to be recognized including a scene and a target contained in the scene. The method includes:

[0116] Acquire a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample;

[0117] For each sample, perform the following steps:

[0118] a. Based on the first data, generating second data to be identified through scenario-based presentation;

[0119] b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system;

[0120] c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample;

[0121] The first scores of all the samples are summed up and then divided by the sum of the scores of all the samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0123] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A testing method for an intelligent perception system, characterized in that: The intelligent perception system is capable of outputting a recognition result about the target based on input data to be recognized, wherein the data to be recognized includes a scene and a target contained in the scene. The method includes: Acquire a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample; For each sample, perform the following steps: a. Based on the first data, generating second data to be identified through scenario-based presentation; b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system; c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample; The first scores of all the samples are summed and then divided by the sum of the scores of all the samples to obtain a second score, which is used to evaluate the perception ability of the intelligent perception system; the method further includes: For the first batch of test samples, obtain the second score corresponding to the first batch; adding additional test samples to the first batch of test samples to form a second batch of test samples; For the second batch of test samples, obtaining a second score corresponding to the second batch; If the second score corresponding to the second batch is equal to the second score corresponding to the first batch, the second score corresponding to the second batch is used to evaluate the perception ability of the intelligent perception system; if the second score corresponding to the second batch is not equal to the second score corresponding to the first batch, it is considered that the perception ability score of the intelligent perception system has not yet stabilized, and additional test samples are added and re-scored.

2. The method for testing an intelligent perception system according to claim 1, wherein: Obtain multiple test samples, including: Obtain basic first data, and set basic labels and basic scores in a targeted manner to form a basic sample; The data generator generates expansive first data based on the input of the basic first data, and sets expansive labels and expansive scores in a targeted manner to form an expansive sample.

3. The testing method for the intelligent perception system according to claim 2, characterized in that: The setting of the scalability score includes: Obtaining the validity of the data generator; The product of the effectiveness and the basic score is used as the scalability score.

4. The testing method for an intelligent perception system according to claim 1, characterized in that: For the plurality of test samples, different scores are set for the samples based on differences in sample types.

5. The testing method for an intelligent perception system according to claim 1, characterized in that: The method further comprises: Classifying the samples according to their scores; For the samples of different categories, the perception capabilities of the intelligent perception system for samples of different categories are evaluated in order from low to high scores.

6. The testing method for an intelligent perception system according to claim 1, characterized in that: The first data includes a scene-based real image; Correspondingly, through scenario presentation, including: Acquire all physical objects in the scene, combine them according to the real images, and provide them to the intelligent perception system for identification; and / or, directly providing the real image to the intelligent perception system for recognition; and / or, Part of the physical objects in the scene are obtained, and the missing physical objects are supplemented by virtual reality technology, and provided to the intelligent perception system for recognition.

7. A test system for an intelligent perception system, used to execute the test method for an intelligent perception system according to any one of claims 1 to 6, characterized in that: The intelligent perception system is capable of outputting a recognition result about the target based on input data to be recognized, wherein the data to be recognized includes a scene and a target contained in the scene. The system includes: A sample module, configured to obtain a plurality of test samples, wherein each of the samples includes a score, a label, and first data of the sample; The loop module is used to perform the following steps for each sample: a. Based on the first data, generating second data to be identified through scenario-based presentation; b. inputting the second data into the intelligent perception system to obtain a recognition result output by the intelligent perception system; c. comparing the recognition result with the label to determine whether the recognition result is correct; if the recognition result is correct, using the score as the first score of the intelligent perception system for the current sample; if the recognition result is incorrect, using zero as the first score of the intelligent perception system for the current sample; The evaluation module is used to add up the first scores of all the samples and then divide the sum by the sum of the scores of all the samples to obtain a second score for evaluating the perception ability of the intelligent perception system.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the testing method for the intelligent perception system as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the testing method for an intelligent perception system according to any one of claims 1 to 6 are implemented.

Images