A verification method, device, electronic device and storage medium

Abstract

The present disclosure provides a verification method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology, and particularly to fields such as deep learning, autonomous driving, and intelligent transportation. The specific implementation solution is as follows: obtaining a test sample and a verification sample according to a sample database; performing batch processing on sample annotation for the test sample to obtain a first batch of processing results; performing batch processing on sample annotation for the verification sample to obtain a second batch of processing results; comparing the first batch of processing results with the second batch of processing results to obtain a comparison result; and performing verification according to the comparison result. By adopting the present disclosure, the verification cost can be reduced and the verification accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and particularly to fields such as deep learning, autonomous driving, and intelligent transportation. Background Art

[0002] Data annotation is a key technology in artificial intelligence technology. Currently, most samples are manually constructed and data is annotated for the samples. Even if batch processing can be added to reduce labor costs, most of the batch processing strategies are also manually formulated. In the verification scenario, all the necessary links in a series of artificial intelligence technologies such as sample generation, data annotation, batch processing, and verification require a large amount of manpower and material resources, with high costs and low accuracy. Summary of the Invention

[0003] The present disclosure provides a verification method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, a verification method is provided, including:

[0005] Obtaining a test sample and a verification sample from a sample database;

[0006] Performing batch processing on sample annotation of the test sample to obtain a first batch processing result;

[0007] Performing batch processing on sample annotation of the verification sample to obtain a second batch processing result;

[0008] Performing a difference comparison between the first batch processing result and the second batch processing result to obtain a comparison result;

[0009] Performing verification according to the comparison result.

[0010] According to another aspect of the present disclosure, a verification apparatus is provided, including:

[0011] A sample generation unit, configured to obtain a test sample and a verification sample from a sample database;

[0012] A first batch processing unit, configured to perform batch processing on sample annotation of the test sample to obtain a first batch processing result;

[0013] A second batch processing unit, configured to perform batch processing on sample annotation of the verification sample to obtain a second batch processing result;

[0014] A comparison unit, configured to perform a difference comparison between the first batch processing result and the second batch processing result to obtain a comparison result;

[0015] A verification unit, configured to perform verification according to the comparison result.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided in any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method provided in any embodiment of the present disclosure.

[0021] According to another aspect of the present disclosure, there is provided a computer program product including computer instructions that, when executed by a processor, implement the method provided in any embodiment of the present disclosure.

[0022] By adopting the present disclosure, test samples and verification samples can be obtained from a sample database, batch processing of sample annotation is performed on the test samples to obtain a first batch of processing results, and batch processing of sample annotation is performed on the verification samples to obtain a second batch of processing results. The first batch of processing results can be compared with the second batch of processing results for difference comparison to obtain a comparison result, and verification is performed according to the comparison result. Thus, the verification cost can be reduced and the verification accuracy can be improved.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0025] Figure 1 is a schematic diagram of an application scenario of an autonomous vehicle communicating with the cloud according to an embodiment of the present disclosure;

[0026] Figure 2 is a schematic flowchart of a verification method according to an embodiment of the present disclosure;

[0027] Figure 3 is a schematic flowchart of a verification method in an application example according to an embodiment of the present disclosure;

[0028] Figure 4 is a schematic flowchart of another verification method in an application example according to an embodiment of the present disclosure;

[0029] Figure 5 This is a schematic diagram of a guide arrow located on a roadside equipment sign in an application example according to an embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram of a connectivity relationship having a semantic relationship with a guide arrow in an application example according to an embodiment of the present disclosure;

[0031] Figure 7 is a flow chart of a verification method using a guide arrow as a sample in an application example according to an embodiment of the present disclosure;

[0032] Figure 8 is a schematic diagram of the structure of a verification device according to an embodiment of the present disclosure;

[0033] Figure 9 is a block diagram of an electronic device used to implement the verification method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them, and do not mean to limit the order or to limit to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0036] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0037] According to an embodiment of the present disclosure, Figure 1 FIG. Figure 1 is a schematic diagram of an application scenario where an autonomous vehicle communicates with the cloud according to an embodiment of the present disclosure. As shown, it includes a background server 100, multiple vehicles (such as vehicle 107 - vehicle 109), and a "cloud" 106 for communication between the background server and the multiple vehicles. On one side of the background server, the distributed cluster system can be adopted. Exemplarily, it is described that the distributed cluster system can be used to receive autonomous driving data reported by multiple vehicles (such as navigation request data, electronic map data, parking data, etc.) and perform data processing tasks related to autonomous driving scenarios. As shown in FIG. Figure 1 , the distributed cluster system includes multiple nodes (such as server cluster 101, server 102, server cluster 103, server 104, server 105). Multiple nodes can jointly execute one or more verification tasks. Through the verification tasks, correct sample annotation data and correct strategies for sample annotation batch processing of samples to be processed can be obtained. As essential links in a series of artificial intelligence technologies such as sample generation, data annotation, batch processing, and verification, it reduces a large amount of manpower and material resources, has low verification costs and high verification accuracy, and is applicable to any model training related to autonomous driving scenarios. Providing the correct sample annotation data to the model training task can improve the accuracy of model training. By the correct batch processing strategy, the generation speed of sample annotation data can be accelerated and provided to the model training task, which can also accelerate the iteration speed of model training. Optionally, in addition to performing verification tasks, multiple nodes in the distributed cluster system can also perform model training. The multiple nodes can execute model training tasks based on data parallelism, and the multiple nodes can also execute model training tasks based on model parallelism. Optionally, after each round of model training tasks is completed, data exchange (such as data synchronization) can be performed between multiple nodes.

[0038] According to an embodiment of the present disclosure, a verification method is provided. Figure 2 FIG. Figure 2 is a schematic flowchart of the verification method according to an embodiment of the present disclosure. This method can be applied to a verification device. For example, when the device is deployed on a single machine, multiple machines, or a terminal, server, or other processing device in a cluster system to execute, verification and other processing can be achieved. Among them, the terminal can be a user equipment (UE), a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, this method can also be implemented by a processor calling computer-readable instructions stored in a memory. As shown in FIG. Figure 1 , this method is applied to Figure 1 ​​​​​​In any node or electronic device (such as a mobile phone or a desktop computer) in the cluster system shown, it includes:

[0039] S201. Obtain a test sample and a verification sample according to the sample database.

[0040] S202. Perform batch processing on the sample annotation of the test sample to obtain a first batch of processing results.

[0041] S203. Perform batch processing on the sample annotation of the verification sample to obtain a second batch of processing results.

[0042] S204. Compare the differences between the first batch of processing results and the second batch of processing results to obtain a comparison result.

[0043] S205. Perform verification according to the comparison result.

[0044] In an example of S201 - S205, the source of the test sample can be the sample database, and the source of the verification sample can be a backup database obtained by backing up the sample database. Perform batch processing on the sample annotation of the test sample to obtain a first batch of processing results, and the first batch of processing results includes first sample annotation data obtained by performing sample annotation on the test sample. Perform batch processing on the sample annotation of the verification sample to obtain a second batch of processing results, and the second batch of processing results includes second sample annotation data obtained by performing sample annotation on the verification sample. Comparing the differences between the first batch of processing results and the second batch of processing results can actually be comparing the first sample annotation data and the second sample annotation data, and perform verification according to the obtained comparison result. Among them, the verification includes at least one verification process such as verifying whether the data is abnormal and verifying whether the batch processing strategy is abnormal.

[0045] By using the present disclosure, a test sample and a verification sample can be obtained according to the sample database, batch processing on the sample annotation of the test sample can be performed to obtain a first batch of processing results, and batch processing on the sample annotation of the verification sample can be performed to obtain a second batch of processing results. The differences between the first batch of processing results and the second batch of processing results can be compared to obtain a comparison result, and verification can be performed according to the comparison result. Thus, the verification cost can be reduced and the verification accuracy can be improved.

[0046] In one embodiment, a test sample and a verification sample are obtained based on a sample database, including: extracting a sample to be processed from the sample database, and obtaining the test sample according to the sample to be processed. The sample database is backed up to obtain a backup database, and the verification sample is obtained according to the backup database. By adopting this embodiment, the source of the test sample is the sample database. In the case of a map navigation scenario, the test sample can be navigation-related data (such as guiding arrows). The source of the verification sample is the backup database obtained by backing up the sample database. Since the verification sample is derived from the same sample database and retains data that comprehensively covers and adapts to various different scenarios, it not only makes the verification more general (suitable for verification in various different scenarios), but also has comprehensive data and there is no possibility of data loss. Therefore, compared with the test sample, the accuracy is higher.

[0047] In one embodiment, the sample to be processed is an image sample to be processed. Obtaining the test sample according to the sample to be processed includes: distinguishing a foreground image and a background image from the image sample to be processed, and taking the target object extracted from the foreground image as the test sample. By adopting this embodiment, taking the image sample as an example, the background image is usually not the focus of concern in the verification task. The foreground image, such as lanes, guiding arrows, roadside devices, pedestrians, etc., is the focus of concern in the verification task. It is necessary to extract the valid data (such as lanes, guiding arrows, roadside devices, pedestrians, etc.) in the foreground image. Taking the navigation scenario as an example, adding correct labels to these valid data through the verification task can provide data support for model training such as autonomous driving and intelligent transportation services, and improve the accuracy of verification.

[0048] In one embodiment, verification is performed according to the comparison result, including: when the comparison result shows that there is no difference between the first batch of processing results and the second batch of processing results, the verification passes, and the sample annotation data obtained based on the batch processing of the sample annotation is correct data. By adopting this embodiment, one of the verification tasks is to verify whether the data is abnormal. For example, when batch processing is used for sample annotation, it is necessary to determine whether the sample annotation data obtained by batch processing is correct. If there is no difference between the first batch of processing results and the second batch of processing results, the verification passes, and the sample annotation data obtained by batch processing is correct data. By automatically performing this verification task without manual intervention, the verification cost is reduced.

[0049] In one embodiment, verification is performed according to the comparison result, including: when the comparison result shows that there are differences between the first batch of processing results and the second batch of processing results, the verification fails, the batch processing strategy used for batch processing based on the sample annotation is an incorrect strategy, and a first prompt message for updating the batch processing strategy is sent. By adopting this embodiment, the second verification task is: to verify whether the batch processing strategy is abnormal. For example, when batch processing is used for sample annotation, it is necessary to determine whether the batch processing strategy used for sample annotation is correct. If there are differences between the first batch of processing results and the second batch of processing results, it indicates that the batch processing strategy is incorrect, and the verification fails, and the batch processing strategy needs to be further optimized. By automatically executing this verification task, a first prompt message for updating the batch processing strategy can be sent when the verification fails, without manual intervention, reducing the verification cost.

[0050] In one embodiment, it further includes: in response to the first prompt message, updating the batch processing strategy. The update of the batch processing strategy is stopped until the verification passes. By adopting this embodiment, a first prompt message for updating the batch processing strategy can be sent when the verification fails, and in response to this first prompt message, without manual intervention, reducing the verification cost.

[0051] In one embodiment, verification is performed according to the comparison result, including: when the comparison result shows that there are differences between the first batch of processing results and the second batch of processing results, the verification fails, the sample annotation data obtained by batch processing based on the sample annotation is incorrect data, and a second prompt message for deleting the sample annotation data is sent. By adopting this embodiment, the situation that causes the verification to fail may also be caused by data anomalies. For example, the sample annotation data obtained by batch processing based on the sample annotation is incorrect data, and this part of incorrect data needs to be deleted, improving the verification accuracy. A second prompt message for deleting this part of incorrect data is sent, without manual intervention, reducing the verification cost.

[0052] In one embodiment, it further includes: in response to the second prompt message, deleting the incorrect sample annotation data. By adopting this embodiment, a second prompt message for deleting the sample annotation data can be sent when the verification fails, and in response to this second prompt message, without manual intervention, reducing the verification cost.

[0053] The verification method provided in the above embodiments of the present disclosure will be illustrated by examples below.

[0054] Figure 3 is a schematic flowchart of a verification method in an application example according to an embodiment of the present disclosure, as Figure 3As shown in the figure, it includes: batch processing the manually created sample data to achieve data annotation of the sample data. Manually analyze the results of the batch processing. If the data annotation meets the expectations, it means that the verification passes, and the annotated data obtained through batch processing (denoted as batch-marked) and the corresponding batch processing strategy can be used for model training; if the data annotation does not meet the expectations and it is impossible to automatically distinguish whether the data annotation is incorrect or the batch processing strategy is incorrect, further manual analysis is required, and finally the batch processing strategy is optimized. Among them, corresponding to the annotated data obtained through batch processing (denoted as batch-marked) and the unannotated data (denoted as un-batch-marked), in addition to being able to judge whether the batch-marked meets the expectations, it is also possible to further judge whether the un-batch-marked meets the expectations. If it meets the expectations, the strategy is put into use; otherwise, the strategy is further optimized.

[0055] Considering that in the above verification method, the sample data is manually created, the results of the batch processing of the sample data are also manually analyzed, and whether the results of the batch processing meet the expectations is also manually judged, the verification cost is high and the verification accuracy is low. Another verification method is provided, where the generation of the sample data, the analysis of the results of the batch processing of the sample data, and the judgment of whether the results of the batch processing meet the expectations are all automatically executed without manual intervention. Among them, verification is performed through the differential comparison of the batch processing of the sample data to judge whether the results of the batch processing meet the expectations, which can not only automatically optimize the strategy but also delete the incorrect annotation data, thereby improving the accuracy of the sample database.

[0056] Figure 4 It is a schematic flow chart of another verification method in the application example according to the embodiment of the present disclosure, as Figure 4As shown in the figure, it includes: extracting samples to be processed from the sample database (the sample database is a complete database, including existing data and newly added data), and obtaining the test samples after converting the formats of the samples to be processed. The test samples are newly added data for batch processing of sample annotation. Taking image samples as an example, the newly added data does not include background image data, but only valid data in the foreground image data, such as guiding arrows, lanes, pedestrians, vehicles, etc. Deleting the previous batch processing results by clearing the base map elements is also to ensure that the difference comparison of the batch processing results is only based on the newly added data. Backing up the sample database to obtain a backup database, and obtaining the verification samples according to the backup database. Conducting batch processing of sample annotation on the test samples to obtain the first batch of processing results, the labeled data obtained through batch processing (denoted as batch Marked), and the unlabeled data (denoted as unbatch Marked). Conducting batch processing of sample annotation on the verification samples to obtain the second batch of processing results. Comparing the differences between the first batch of processing results and the second batch of processing results to obtain a comparison result. Conducting verification based on the comparison result. If the comparison result shows that there is no difference between the first batch of processing results and the second batch of processing results, the verification passes, and the sample annotation data obtained based on the batch processing of sample annotation is correct data; if the comparison result shows that there is a difference between the first batch of processing results and the second batch of processing results, the verification fails. Specifically, further difference analysis is performed on the different parts. If it is yes, it means that the strategy is abnormal, that is, the batch processing strategy used for the batch processing of sample annotation is the wrong strategy, and strategy optimization is required, and a first prompt message for updating the batch processing strategy can be sent; if the comparison result shows that there is a difference between the first batch of processing results and the second batch of processing results, the verification fails. Specifically, further difference analysis is performed on the different parts. If it is no, it means that the sample annotation data obtained based on the batch processing of sample annotation is wrong data, and a second prompt message for deleting the sample annotation data can be sent.

[0057] The above verification method of this application example is used in the production process of navigation electronic map data for autonomous driving scenarios. Through automated sample data generation, batch processing, and verification, it can ensure that the production process of navigation electronic map data not only meets the expected quality requirements, but also can continuously and rapidly iterate data and optimize the batch processing strategy, making all links in the entire production process in a closed-loop and correctable state, and all links in the entire production process develop more benignly, with low cost and high accuracy.

[0058] In the production process of navigation electronic map data, guiding arrows are indispensable. Guiding arrows are used to indicate the driving direction of vehicles. Figure 5 It is a schematic diagram of a guiding arrow on a roadside device signboard in the application example according to an embodiment of the present disclosure, such as Figure 5As shown, different guiding arrows have different pointing meanings. The special guiding arrow for U-turn is used to indicate the special lane where vehicles are allowed to make a U-turn; the straight guiding arrow is used to indicate the special lane where vehicles are allowed to drive straight; the special guiding arrow for right turn is used to indicate the special lane where vehicles turn right. On this special lane, vehicles can only turn right and are prohibited from going straight, turning left, and driving in other directions. It may also include a dedicated bus lane, etc.

[0059] Figure 6 It is a schematic diagram of the connection relationship with semantic relationship to the guiding arrow in the application example according to the embodiment of the present disclosure. The connection relationship is used to characterize the semantic relationship between the guiding arrow and the lane where the guiding arrow is located. In the autonomous driving scenario, users can achieve autonomous driving based on the voice prompts obtained from the navigation electronic map data. Machines do not know what a guiding arrow is and what a lane is, and it needs to be converted into a language that the machine can recognize. For example, when the guiding arrow indicates that the vehicle should make a U-turn at the next intersection, it is necessary to exit the current lane. Converting it into a language that the machine can recognize means: finding the semantic connection relationship between the guiding arrow and the current lane. Figure 6 It is an example of the semantic connection relationship between the guiding arrow and the current lane. After obtaining the connection relationship, a voice prompt can be generated to indicate the current vehicle to exit the current lane through the voice prompt, and drive to the next intersection according to the direction indicated by the guiding arrow and then make a U-turn.

[0060] Figure 7 It is a schematic flowchart of the verification method with the guiding arrow as a sample in the application example according to the embodiment of the present disclosure. Taking the guiding arrow as an example, the verification sample (this verification sample can be the guiding arrow) is obtained from the backup data of the sample database. The first connection relationship is obtained according to this verification sample. The test sample (this test sample can also be the guiding arrow) is extracted from the sample database. The existing guiding arrows in the sample database are cleared, and batch processing is performed on the newly added guiding arrows to obtain the batch processing result for the newly added guiding arrows (denoted as the already batch-marked for the newly added guiding arrows). The second connection relationship is obtained according to the batch processing result of the newly added guiding arrows. An automated difference comparison is performed between the first connection relationship and the second connection relationship, and verification is performed according to the obtained comparison result.

[0061] It should be noted that the calibration samples are sourced from the backup data of the sample database (which includes the old master database data). This backup data includes: the guiding arrows that are complete in the old master database data. The test samples are sourced from the sample database (which includes the old master database data), but the existing guiding arrows in the sample database are cleared, and only the newly added guiding arrows are retained. After batch processing the newly added guiding arrows, the sample database is updated to obtain the updated sample database (which includes the new master database data). Only the newly added batch processing results are retained in the updated sample database, enabling the test samples and batch processing to be continuously updated. Correspondingly, the sample database from which the test samples are sourced is also in an updated state. As a result, the data volume is reduced, the calibration cost is lowered, and the data is continuously updated, improving the calibration accuracy.

[0062] According to an embodiment of the present disclosure, a calibration device is provided. Figure 8 It is a schematic diagram of the composition structure of the calibration device according to an embodiment of the present disclosure, as Figure 8 shown, the calibration device 800 includes: a sample generation unit 801, configured to obtain test samples and calibration samples according to the sample database; a first batch processing unit 802, configured to perform batch processing on the sample annotation of the test samples to obtain a first batch processing result; a second batch processing unit 803, configured to perform batch processing on the sample annotation of the calibration samples to obtain a second batch processing result; a comparison unit 804, configured to perform a difference comparison between the first batch processing result and the second batch processing result to obtain a comparison result; and a calibration unit 805, configured to perform calibration according to the comparison result.

[0063] In one embodiment, the sample generation unit is configured to extract the samples to be processed from the sample database; obtain the test samples according to the samples to be processed; back up the sample database to obtain a backup database;

[0064] obtain the calibration samples according to the backup database.

[0065] In one embodiment, when the samples to be processed are image samples to be processed, the sample generation unit is configured to distinguish the foreground image and the background image from the image samples to be processed; and use the target object extracted from the foreground image as the test samples.

[0066] In one embodiment, the calibration unit is configured to pass the calibration when the comparison result shows that there is no difference between the first batch processing result and the second batch processing result, and the sample annotation data obtained based on the batch processing of the sample annotation is correct data.

[0067] In one embodiment, the verification unit is used to fail the verification when the comparison result shows a difference between the first batch of processing results and the second batch of processing results. Based on this, the batch processing strategy used for the batch processing of sample annotation is an incorrect strategy, and a first prompt message for updating the batch processing strategy is sent.

[0068] In one embodiment, it further includes a strategy update unit, which is used to respond to the first prompt message and update the batch processing strategy; and stop the update of the batch processing strategy until the verification passes.

[0069] In one embodiment, the verification unit is used to fail the verification when the comparison result shows a difference between the first batch of processing results and the second batch of processing results. Based on this, the sample annotation data obtained from the batch processing of sample annotation is incorrect data, and a second prompt message for deleting the sample annotation data is sent.

[0070] In one embodiment, it further includes a data deletion unit, which is used for:

[0071] Respond to the second prompt message and delete the incorrect sample annotation data.

[0072] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0073] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0074] Figure 9 FIG. shows a schematic block diagram of an exemplary electronic device 900 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0075] As Figure 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0076] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disc, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0077] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 executes the various methods and processes described above, such as the verification method. For example, in some embodiments, the verification method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the verification method described above can be executed. Alternatively, in other embodiments, the computing unit 901 can be configured to execute the verification method in any other appropriate way (e.g., by means of firmware).

[0078] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0079] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0080] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0082] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0083] A computer system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, or a server of a distributed system, or a server incorporating a blockchain.

[0084] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0085] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A verification method, comprising: Obtaining a test sample and a verification sample according to a sample database; wherein, the test sample is new data to be batch processed for sample annotation; Performing batch processing on the sample annotation of the test sample to obtain a first batch of processing results; Performing batch processing on the sample annotation of the verification sample to obtain a second batch of processing results; Performing a difference comparison between the first batch of processing results and the second batch of processing results to obtain a comparison result; Performing verification according to the comparison result; Wherein, obtaining a test sample and a verification sample according to a sample database includes: Extracting samples to be processed from the sample database; the samples to be processed are image samples to be processed; Obtaining the test sample according to the samples to be processed; Backing up the sample database to obtain a backup database; Obtaining the verification sample according to the backup database.

2. The method according to claim 1, wherein The obtaining the test sample according to the samples to be processed includes: Distinguishing a foreground image and a background image from the image samples to be processed; Taking the target object extracted from the foreground image as the test sample.

3. The method according to any one of claims 1-2, wherein, The performing verification according to the comparison result includes: When the comparison result shows that there is no difference between the first batch of processing results and the second batch of processing results, the verification passes, and the sample annotation data obtained based on the batch processing of the sample annotation is correct data.

4. The method according to any one of claims 1-2, wherein The performing verification according to the comparison result includes: When the comparison result shows that there is a difference between the first batch of processing results and the second batch of processing results, the verification fails, the batch processing strategy used for the batch processing of the sample annotation is an incorrect strategy, and a first prompt message for updating the batch processing strategy is sent.

5. The method according to claim 4, further comprising: Responding to the first prompt message and updating the batch processing strategy; Stopping the update of the batch processing strategy until the verification passes.

6. The method according to any one of claims 1-2, wherein, The performing verification according to the comparison result includes: When the comparison result shows that there is a difference between the first batch of processing results and the second batch of processing results, the verification fails, the sample annotation data obtained based on the batch processing of the sample annotation is incorrect data, and a second prompt message for deleting the sample annotation data is sent.

7. The method according to claim 6, further comprising: Responding to the second prompt message and deleting the incorrect sample annotation data.

8. A verification device, comprising: A sample generation unit for obtaining a test sample and a verification sample according to a sample database; wherein, the test sample is new data to be batch processed for sample annotation; A first batch processing unit for performing batch processing on the sample annotation of the test sample to obtain a first batch of processing results; A second batch processing unit for performing batch processing on the sample annotation of the verification sample to obtain a second batch of processing results; A comparison unit for performing a difference comparison between the first batch of processing results and the second batch of processing results to obtain a comparison result; A verification unit for performing verification according to the comparison result; Wherein, the sample generation unit is used for: Extract the sample to be processed from the sample database; the sample to be processed is an image sample to be processed; Obtain the test sample according to the sample to be processed; Back up the sample database to obtain a backup database; Obtain the verification sample according to the backup database.

9. The apparatus according to claim 8, wherein, The sample generation unit is used for, when the sample to be processed is an image sample to be processed, Distinguish the foreground image and the background image from the image sample to be processed; Use the target object extracted from the foreground image as the test sample.

10. The device according to any one of claims 8-9, wherein, The verification unit is used for: When the comparison result shows that there is no difference between the first batch of processing results and the second batch of processing results, the verification passes, and the sample annotation data obtained based on the batch processing of the sample annotation is correct data.

11. The device according to any one of claims 8-9, wherein, The verification unit is used for: When the comparison result shows that there is a difference between the first batch of processing results and the second batch of processing results, the verification fails, the batch processing strategy used for the batch processing of the sample annotation is an incorrect strategy, and a first prompt message for updating the batch processing strategy is issued.

12. The device according to claim 11, further comprising a strategy update unit for: Respond to the first prompt message and update the batch processing strategy; Stop updating the batch processing strategy until the verification passes.

13. The device according to any one of claims 8-9, wherein, The verification unit is used for: When the comparison result shows that there is a difference between the first batch of processing results and the second batch of processing results, the verification fails, the sample annotation data obtained based on the batch processing of the sample annotation is incorrect data, and a second prompt message for deleting the sample annotation data is issued.

14. The device according to claim 13, further comprising a data deletion unit for: Respond to the second prompt message and delete the incorrect sample annotation data.

15. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.

Images