Model testing method, device and equipment based on service grid and storage medium

Through service mesh technology, the inference request data in industrial production is mirrored and collected and stored uniformly, which solves the problem of model accuracy reduction and development complexity, realizes unified management and performance testing of real inference data, and reduces development complexity and maintenance difficulty.

CN118939528AInactive Publication Date: 2024-11-12YITONG TECH (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410878651.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In industrial production, the accuracy of the model decreases with time and data changes. The existing technology requires each inference program to realize data collection and management functions, resulting in an increase in the complexity of inference service development, and different services adopt their own storage methods, which increases Platform maintenance difficulty.

Method used

The service grid-based method is used to mirror the inference request data generated on the production line and uniformly store it. By matching the mirror data entities and labels, the performance test of the model to be tested is realized, reducing the complexity of the development of inference services.

Benefits of technology

It realizes unified collection and management of real inference data on the production line, reduces the development complexity of each inference service, improves the authenticity and efficiency of model testing, and reduces the difficulty of platform maintenance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939528A_ABST
    Figure CN118939528A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a model testing method and device based on a service grid, equipment and a storage medium, and the method comprises the steps: obtaining a first reasoning request of a first production line client, carrying out the mirror image collection of the first reasoning request based on the service grid, and obtaining first mirror image data; wherein the first mirror image data comprises a first mirror image data entity and a first mirror image data label; storing the first mirroring data entity and the first mirroring data tag in a collection container; obtaining a first to-be-tested model and preset screening reference information corresponding to the first to-be-tested model; matching with the first mirror image data label in the collection container based on the screening reference information to obtain a first mirror image data entity; and performing performance test on the to-be-tested model through the first mirror image data entity to obtain model test data. The real reasoning data generated on the production line is collected and managed in a unified manner, a program does not need to be compiled independently for each reasoning service, and the development complexity of the reasoning service is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of model testing technology, and in particular to a service grid-based model testing method, device, equipment and storage medium. Background Art

[0002] In recent years, more and more models have been deployed in real business scenarios. In industrial scenarios such as PCBA, the accuracy of the model will gradually decrease due to the continuous changes in products or processes or the change of time and inference data. Therefore, it is necessary to test the accuracy and performance of the model using real request data in the industrial production process.

[0003] In related technologies, the callee, that is, the inference program corresponding to the model, can collect the request data and record the request data in a file or database for subsequent testing of the model. However, this method requires each inference program to implement a set of request data collection, storage and management functions, which greatly increases the complexity of inference service development. Summary of the invention

[0004] The embodiments of the present application provide a service grid-based model testing method, apparatus, device and storage medium, which can realize the unified collection and management of real reasoning data generated on the production line, and apply it to the performance testing of the production model, thereby reducing the development complexity of the reasoning service.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model testing method based on a service grid, the method comprising:

[0006] Obtain a first inference request from a first production line client, and perform image collection on the first inference request based on the service grid to obtain first image data; wherein the first image data includes a first image data entity and a first image data label;

[0007] storing the first mirror data entity and the first mirror data label in a collection container;

[0008] Acquire a first model to be tested and screening benchmark information preset corresponding to the first model to be tested;

[0009] Based on the screening reference information, the first image data label is matched in the collection container to obtain the first image data entity;

[0010] The performance test of the model to be tested is performed through the first mirror data entity to obtain model test data.

[0011] In some embodiments, the collection container includes an entity database and a collection database, and storing the first mirror data entity and the first mirror data tag in the collection container includes:

[0012] Store the first mirror data entity in the entity database, and obtain a first entity storage address;

[0013] The first mirror data tag and the first entity storage address are stored in the acquisition database.

[0014] In some embodiments, the first image data tag includes a first request creation time, and storing the first image data entity and the first image data tag in a collection container includes:

[0015] Store the first mirror data entity in the entity database, and obtain a first entity storage address;

[0016] Obtaining the creation time of the first request;

[0017] When the creation time of the first request is within a first preset time period, the first mirror data tag and the first entity storage address are stored in a first sub-table corresponding to the first preset time period in the acquisition database.

[0018] In some embodiments, the screening benchmark information includes a target time period, wherein the target time period matches the first preset time period, and the first image data entity is obtained by matching the first image data tag in the collection container based on the screening benchmark information, including:

[0019] Matching the target time period with the first preset time period in the collection database to obtain the first sub-table;

[0020] The first entity storage address is obtained from the first sub-table, and the first mirror data entity is obtained according to the first entity storage address.

[0021] In some embodiments, the screening benchmark information further includes a target production line identifier and a target machine identifier, the first image data tag further includes a first production line identifier and a first machine identifier, and the method further includes:

[0022] In the first sub-table, the target production line identifier is compared with the first production line identifier to obtain a first comparison result, and the target machine identifier is compared with the first machine identifier to obtain a second comparison result;

[0023] Selecting a target image data tag from the plurality of first image data tags according to the first comparison result and the second comparison result;

[0024] A target entity storage address corresponding to the target image data tag is determined, and the corresponding target image data entity is acquired from the target entity storage address.

[0025] In some embodiments, the model to be tested is a printed circuit board assembly visual inspection model, and the step of testing the model to be tested by using the first mirror data entity to obtain model test data includes:

[0026] Calling the printed circuit board assembly visual inspection model through the first mirror data entity to perform defect inspection and obtain defect inspection results;

[0027] Comparing the defect detection result with the labeled data corresponding to the first mirror data entity to obtain a defect detection rate, a false detection rate, and a missed detection rate;

[0028] The model test data is generated according to the defect detection rate, the false detection rate and the missed detection rate.

[0029] In some embodiments, obtaining a first inference request from a first production line client, and performing image collection on the first inference request based on the service grid to obtain first image data includes:

[0030] Obtaining a first inference request sent by the first production line client;

[0031] Based on the service grid, mirroring the first reasoning request according to a preset sampling rate to obtain the first mirror data entity;

[0032] A first mirror data tag is configured for the first mirror data entity to obtain the first mirror data.

[0033] To achieve the above purpose, a second aspect of an embodiment of the present application proposes a model testing device based on a service grid, comprising:

[0034] An image collection module, used to obtain a first reasoning request from a first production line client, and perform image collection on the first reasoning request based on the service grid to obtain first image data;

[0035] A data storage module, used to store the first mirror data entity and the first mirror data label in a collection container; wherein the first mirror data includes the first mirror data entity and the first mirror data label;

[0036] A model acquisition module, used to acquire a first printed circuit board assembly visual inspection model and screening reference information preset for the first printed circuit board assembly visual inspection model;

[0037] A data screening module, configured to screen and obtain the first mirror data entity according to the screening reference information and the first mirror data tag;

[0038] The model testing module is used to test the model to be tested through the first mirror data entity to obtain model testing data.

[0039] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the service grid-based model testing method as described in the first aspect.

[0040] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the service grid-based model testing method described in the first aspect above.

[0041] The service grid-based model testing method, device, equipment and storage medium proposed in the embodiments of the present application, the method first obtains a first reasoning request from a first production line client, and performs mirror collection on the first reasoning request based on the service grid to obtain first mirror data; wherein the first mirror data includes a first mirror data entity and a first mirror data label; then, the first mirror data entity and the first mirror data label are stored in a collection container; next, a first model to be tested and screening benchmark information preset corresponding to the first model to be tested are obtained; then, based on the screening benchmark information, the first mirror data label is matched in the collection container to obtain the first mirror data entity; finally, a performance test is performed on the model to be tested through the first mirror data entity to obtain model test data. The embodiment of the present application uses a service grid to mirror the inference request data generated on the production line, and stores its data entities and data labels in a collection container. After obtaining the model to be tested, the corresponding data entities are collected in the collection container based on a specific data screening benchmark for the model to be tested, and model test data is obtained, so as to realize the unified collection and management of the real inference data generated on the production line, and apply it to the performance test of the production model. It is no longer necessary to compile the program separately for each inference service, thereby reducing the development complexity of the inference service.

[0042] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of a service grid provided in an embodiment of the present application.

[0044] Figure 2 It is a flowchart of a service grid-based model testing method provided in one embodiment of the present application.

[0045] Figure 3 yes Figure 2 Flow chart of step 201 in FIG.

[0046] Figure 4 yes Figure 2 Flow chart of step 202 in FIG.

[0047] Figure 5 yes Figure 4 Flow chart of step 402 in FIG.

[0048] Figure 6 yes Figure 2 Flow chart of step 204 in FIG.

[0049] Figure 7 This is a flow chart of a screening machine and a production line provided by another embodiment of the present application.

[0050] Figure 8 yes Figure 2 Flow chart of step 205 in FIG.

[0051] Fig. 9 A schematic diagram of a model testing device provided in one embodiment of the present application.

[0052] Fig.10 This is a schematic diagram of the hardware structure of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those understood by a person skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, some nouns involved in this application are analyzed:

[0057] Service Mesh is an infrastructure layer for microservice architecture, designed to handle communication and coordination between microservices. Its main goal is to provide a simple and efficient way to manage communication, traffic control, security, and monitoring between microservices. Figure 1 As shown in the figure, the service mesh is divided into a control plane and a data plane. The black squares in the figure represent application services, and the gray squares represent sidecar proxies. The dotted arrows represent the communication paths between services, which are controlled and managed by the sidecar proxies. Among them, the data plane layer is divided into two parts, application services and sidecar proxies. Application services are responsible for handling specific business logic, while sidecar proxies are responsible for handling communication between services. Sidecar proxies are deployed next to each application service and are responsible for handling various network communication functions between services, such as routing, load balancing, fuses, timeout retries, etc. This separation design allows application developers to focus on the implementation of business logic without paying too much attention to the underlying network communication issues. The sidecar proxy assumes this part of the responsibility, and the "sidecar" separates the network communication function from the application to form an independent infrastructure layer - the service mesh. This separation design allows the service mesh to better manage and control the communication between services. Common sidecar implementations include Istio's Envoy, Linkerd's Linkerd, and Consul's Connect.

[0058] AI inference refers to the process of using a trained machine learning model to make actual predictions or decisions. During the model training phase, the machine learning algorithm will learn to extract data features and build a prediction model based on a large amount of training data. This process is also called model training; the trained model will be deployed to the production environment to receive new input data and make predictions or decisions based on the learned rules. This process is called AI inference or model inference. The inference speed is very fast because the model has already learned the rules and does not need to repeat the training process. The results will be used as a basis for supporting business decisions.

[0059] Request replay refers to the process in the field of network and network security where a request that has already been executed is sent or replayed again. This may be for legitimate or malicious reasons. In this application, request replay technology is used in legitimate ways. During AI reasoning, these reasoning requests are recorded and then replayed in a test environment to verify the behavior and performance of the model.

[0060] In recent years, people have paid more and more attention to the implementation of machine / deep learning projects, and more and more models have been deployed in real business scenarios. However, when the business actually starts to use it, there will be various demand feedbacks for the model, or as time goes by and the inference data changes (for example, in industrial scenarios, due to the continuous changes in products or processes), the accuracy of the model will gradually decrease, thus failing to meet the accuracy requirements of online services. Therefore, algorithm engineers need to continuously iterate development and frequently deploy online. As the business develops, the number of model application scenarios increases, and managing and maintaining these model systems has become a real challenge. Model iteration is generally divided into several steps:

[0061] 1) Model monitoring: monitor the relevant indicators of the model, such as confidence, defect category distribution, etc., to find problems with the model

[0062] 2) Obtaining difficult data: Obtaining difficult data generated in the production system. This data needs to be re-labeled and trained to improve the accuracy and generalization ability of the model.

[0063] 3) Data labeling: Relabeling of difficult data

[0064] 4) Training & Evaluation: Incremental training and evaluation on labeled hard examples

[0065] 5) Model online testing and deployment: Use A / B testing, grayscale release, and other technologies and monitoring methods to test the model in many aspects before it goes online to ensure that the model's indicators, performance, and interfaces meet the requirements before it can go online.

[0066] 6) Data feedback: After the model is launched, the inference results are fed back for subsequent model analysis and iteration

[0067] Among them, step 5) model online testing is a very important link. Sufficient means must be taken to ensure that the accuracy and performance of the model can meet the requirements of online services before the model goes online. It is particularly important to note that in industrial production (such as the PCBA field), it is best to use real request data on the production line to test the accuracy and performance of the model, so that the test indicators are more credible and convincing.

[0068] In related technologies, reasoning services bear too many data collection and management responsibilities. Each reasoning program needs to implement its own data collection and storage logic, which increases the complexity of reasoning service development and causes duplication of work. At the same time, different reasoning services use their own data storage methods, which increases the difficulty of maintenance at the platform level.

[0069] In order to achieve unified collection and management of real reasoning data generated on the production line, apply it to the performance test of the production model, and reduce the development complexity of the reasoning service, the embodiment of the present application uses a service grid to mirror the reasoning request data generated on the production line, and stores its data entities and data labels in a collection container. After obtaining the model to be tested, the corresponding data entities are collected in the collection container based on a specific data screening benchmark for the model to be tested, and the model test data is obtained, so as to achieve unified collection and management of the real reasoning data generated on the production line, and apply it to the performance test of the production model. It is no longer necessary to compile the program separately for each reasoning service, thereby reducing the development complexity of the reasoning service.

[0070] The following will further describe the service grid-based model testing method, device, equipment and storage medium provided in the embodiments of the present application.

[0071] Reference Figure 2 , is an optional flowchart of a service grid-based model testing method provided in an embodiment of the present application, Figure 2 The method in the embodiment may include but is not limited to steps 201 to 205. Figure 2 The order of step 201 to step 205 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0072] Step 201: Obtain a first inference request from a first production line client, and perform image collection on the first inference request based on a service grid to obtain first image data.

[0073] In some embodiments, the first inference request from the first production line client is intercepted through the proxy function of the service grid. The service grid will automatically perform image collection on these requests, generate copy data of the requests, obtain the first inference request data collected from these images from the service grid, including the original content of the request, request header, response result and other information, and encapsulate these data into the first image data entity for subsequent storage and processing. By using the image collection function of the service grid, there is no need to add additional data collection logic to the application, reducing the complexity of development. The real inference request data can be obtained, which improves the authenticity and effectiveness of the test.

[0074] The "first production line client" here refers to the actual business system, which can be an application deployed in the production environment, and the "first inference request" refers to a specific inference request sent by the production line client to the model service. Through this step, real inference request data can be obtained as a test sample without intruding the business system. These mirror data have high authenticity and representativeness, and can provide a reliable basis for subsequent model testing.

[0075] In some embodiments, the system first needs to capture the real-time reasoning requests sent by the first production line client. These requests can be generated by the visual inspection system on the equipment of the production line, including the task of analyzing and reasoning about a certain PCB component image. The service grid technology intercepts these reasoning requests and copies them as mirror requests without interfering with the execution of the original request. The generation of mirror data helps to collect data for testing the model under test without affecting the normal operation of the production system.

[0076] In some embodiments, Istio can be used to replicate inference requests. Istio is an open source service mesh platform that can solve the complexity of service communication in a microservice architecture and provide a unified way to manage and monitor the interaction of microservices. Istio's ability to replicate traffic can be used. After the online request reaches the istio gateway, istio will copy the traffic and send it to the collection container, which will then be recorded on the disk.

[0077] like Figure 3 As shown, in some embodiments, step 201 may include but is not limited to steps 301 to 303 .

[0078] Step 301: Obtain a first inference request sent by a first production line client.

[0079] Among them, during the operation of the production line client (such as PCB inspection equipment), when image reasoning is required (such as detecting defects on PCB), the equipment will generate a reasoning request. This reasoning request contains the image data to be analyzed and other meta information that may be included, such as the equipment number, request timestamp, etc.

[0080] Step 302: Based on the service grid, mirror collection is performed on the first reasoning request according to a preset sampling rate to obtain a first mirror data entity.

[0081] In some embodiments, considering the occupancy of CPU and network bandwidth, the service grid is configured with a preset sampling rate, which means that not all inference requests will be used to generate mirror data. According to this sampling rate, the service grid will select a portion of all received inference requests for mirror collection. The sampling rate can be set based on different requirements, such as percentages, time intervals, or specific request characteristics. In this way, the amount and frequency of test data can be controlled to avoid generating a large amount of mirror data that may cause waste of resources. These data obtained by sampling constitute the "first mirror data entity", and because the sampling rate is random sampling, it will not reduce the validity of the overall data, but it can effectively reduce the occupancy of CPU and network bandwidth.

[0082] In some embodiments, the sampling rate is calculated as a percentage, for example, a sampling rate of 10 represents 10%, and 100 represents 100%. When a request arrives at the service grid, it will decide whether to sample this time based on the sampling rate. If it is 10%, the service grid will use a pseudo-random number. When the pseudo-random number falls within the range of 10%, then the sampling is determined. If it falls outside the range, no sampling is performed this time. From a statistical point of view, when the number of requests is large enough (for example, 100,000), a sampling rate of 10% will be approximately equal to a sampling number of about 10,000.

[0083] Among them, the service grid can be configured with advanced network agents such as Istio, and sidecar agents such as Envoy can be configured to implement this mirroring processing. Such mirroring requests will not be sent to the real service for processing, but directly to the test or monitoring system, so that real-time traffic replication and analysis can be achieved without affecting production services.

[0084] Step 303: configuring a first mirror data tag for the first mirror data entity to obtain first mirror data.

[0085] In some embodiments, after the first mirror data entities are generated, the next operation is to assign tags to these entities to generate complete first mirror data. Tags are configured to save contextual information related to the entity data, such as the initiation time, source, target service identifier, etc. of the request. They can be used in subsequent steps to identify, filter, and sort data, or when data needs to be selected based on specific attributes for further analysis or testing. After being assigned with mirror data tags, the first mirror data entities and their corresponding tags together constitute the complete first mirror data.

[0086] In some embodiments, the system assigns a UUID as a label to each first mirror data entity obtained from the service grid. The generation of UUID can utilize ready-made functions in various programming languages ​​or libraries, such as libraries in Python, or in Java, etc. Assigning a UUID to each data entity gives each mirror data a unique identity, which can maintain the uniqueness and traceability of each data entity in the entire system in clusters and large-scale distributed environments. After obtaining the first mirror data with a UUID label, this data can then be used for various tests and analyses in a service grid environment without interfering with the actual production environment, such as testing the performance of an AI model, or analyzing the impact of production line operating parameters on product quality.

[0087] Step 202: Store the first mirror data entity and the first mirror data tag in a collection container.

[0088] In some embodiments, the first mirror data entity obtained in the previous step 201 needs to be saved. This mirror data entity contains the original content, request header, response result and other information of the real reasoning request collected from the service grid, and is an important data sample for subsequent testing. The first mirror request tag can include metadata such as the time when the request occurred, the production line name of the request source, and the type of request. These tag information helps to classify, retrieve and filter these mirror data in the future, and improve the efficiency of data management. Then, the mirror data entity and its corresponding tag information are uniformly stored in a special collection container, which can be a database or object storage service for persistent storage of these mirror data.

[0089] like Figure 4 As shown, in some embodiments, the collection container includes an entity database and a collection database, and step 202 may include but is not limited to steps 401 to 402.

[0090] Step 401: Store the first mirror data entity in an entity database, and obtain the first entity storage address.

[0091] Step 402: Store the first mirror data tag and the first entity storage address in the acquisition database.

[0092] In steps 401 to 402 of some embodiments, the system first needs to store the first mirror data entities generated by the above steps, and these data entities are stored in a so-called entity database. The entity database can be a SQL or NoSQL database, which is used to persistently store actual data content, such as image files, log records, system indicators, etc. Once the data entity is stored, the system will obtain the storage address corresponding to each data entity. This address can be a row identifier in the database (such as a primary key ID) or a file path in the storage system (such as a URI in an object store). Next, the system will store the label of the first mirror data, which specifically refers to the UUID assigned to the data entity, together with the corresponding storage address in another database, namely the acquisition database. The acquisition database is responsible for maintaining the metadata of the data entity, including the label (UUID) of each first mirror data entity and the address pointing to the actual storage content (the first entity storage address). Such a structure facilitates the indexing and rapid retrieval of data, and also supports the implementation of complex query and data management functions. Since the acquisition database only stores metadata rather than the data itself, it is generally small and has a fast query speed.

[0093] like Figure 5 As shown, in some embodiments, the first image data tag includes the first request creation time, and step 402 may include but is not limited to steps 501 to 502.

[0094] Step 501: Obtain the first request creation time;

[0095] Step 502: When the first request creation time is within a first preset time period, the first mirror data tag and the first entity storage address are stored in a first sub-table corresponding to the first preset time period in the acquisition database.

[0096] In some embodiments, the first image data tag also includes the first request creation time. When storing the collected data, it can be determined whether the time of request creation is within a pre-set first time period. If so, the tag information and storage address are stored in a sub-table corresponding to the time period in the collection database, so as to perform data query and analysis for a specific time period later. Through such a hierarchical storage method, the separate management of the image data entity and tag information can be achieved, which improves the flexibility and scalability of data storage. The entity data and tag data can be optimized independently, and the data partitioning based on the time dimension can quickly locate and filter the data of a specific time period.

[0097] Step 203: Acquire a first model to be tested and preset screening benchmark information corresponding to the first model to be tested.

[0098] In some embodiments, the first model to be tested may be a newly developed model or an optimized version of an existing model. Before preparing for model testing, the system needs to obtain the model to be tested and its screening benchmark information. The screening benchmark information may include test case identification, timestamp range, production line information, etc. This information will be used to screen out a mirror data set suitable for testing in step 204.

[0099] Step 204: matching the first image data tag in the collection container based on the screening reference information to obtain the first image data entity.

[0100] In some embodiments, the system can search and match relevant image data tags in the collection container. In this step, the data management system of the collection container will quickly locate and extract the required data set based on screening conditions, such as timestamp matching, request source matching, data type matching, etc.

[0101] like Figure 6 As shown, in some embodiments, step 204 may include but is not limited to steps 601 to 602.

[0102] Step 601: Match the target time period with the first preset time period in the collection database to obtain a first sub-table;

[0103] In some embodiments, based on the target time period for which data needs to be acquired, a query is performed in the acquisition database to match the corresponding first preset time period. Since the data is stored in partitions (as described in step 502), the purpose of this step is to find the sub-table containing the data for the required time period. For example, if the acquisition database structure is partitioned by month or other time units, this step will identify the specific partition table containing the data for the target time period.

[0104] Step 602: Obtain a first entity storage address from a first sub-table, and obtain a first mirror data entity according to the first entity storage address.

[0105] In some embodiments, the screening benchmark information includes a target time period, and the target time period matches the first preset time period. It can be understood that the match can be equal to or within the first preset time period. The first preset time period set when the table is divided is matched by the target time period, and the first sub-table corresponding to the target time period is queried. The corresponding first entity storage address is queried in the first sub-table according to the first mirror data tag, and the first mirror data entity is obtained according to the storage address.

[0106] like Figure 7As shown, in some embodiments, the screening benchmark information also includes a target production line identifier and a target machine identifier, the first mirror data label also includes a first production line identifier and a first machine identifier, and the model testing method provided in the embodiment of the present application may also include but is not limited to steps 701 to 703.

[0107] Step 701: In the first sub-table, the target production line identifier is compared with the first production line identifier to obtain a first comparison result, and the target machine identifier is compared with the first machine identifier to obtain a second comparison result.

[0108] In some embodiments, the system can search for data records that match the target production line identifier and the target machine identifier in a specific sub-table of the acquisition database. These identifiers can be represented as part of the data tag and included in the first mirror data tag. The target production line identifier and the target machine identifier can be preset by the user or the test scenario. The system traverses all records in the sub-table and compares the first production line identifier and the first machine identifier in the record one by one to ensure that the selected data is derived from the specific production line and machine. The records that obtain successful matching are called the first comparison result and the second comparison result.

[0109] Step 702: Selecting a target image data tag from a plurality of first image data tags according to the first comparison result and the second comparison result;

[0110] In some embodiments, the system will select the target image data label that meets all the screening conditions from the multiple candidate image data obtained in the above screening process according to the first comparison result and the second comparison result. This step ensures that the screened data label is completely consistent with the preset target production line and machine, and meets the test requirements.

[0111] Step 703: Determine the target entity storage address corresponding to the target image data tag, and obtain the corresponding target image data entity from the target entity storage address.

[0112] In some embodiments, the system determines the target entity storage address corresponding to the selected target image data tag, which has been stored in the acquisition database in steps 401 and 402. With the correct storage address, the system can retrieve and obtain the specific target image data entity from the entity database. The data entity will be used for the next step of the performance test of the model to be tested.

[0113] Step 205: Perform a performance test on the model to be tested through the first mirror data entity to obtain model test data.

[0114] In some embodiments, the system can use these data entities as input to call the model to be tested for performance testing. For example, in the PCBA component visual inspection scenario, the model needs to identify possible defects in the image and give a diagnosis result. By comparing and analyzing the results output by the model with the real annotated data, the system can calculate the key performance indicators of the model, such as defect detection rate, false detection rate, and missed detection rate. Finally, these performance indicators are summarized into model test data to provide a basis for evaluating model performance.

[0115] like Figure 8 As shown, in some embodiments, the model to be tested is a printed circuit board assembly visual inspection model, and step 205 may include but is not limited to steps 801 to 803.

[0116] Step 801: Calling a printed circuit board assembly visual inspection model through a first mirror data entity to perform defect inspection and obtain defect inspection results.

[0117] In some embodiments, after the target image data entities are determined, the system will use these data entities to test the printed circuit board (PCB) component visual inspection model. In this step, the model is called to analyze the image in the first image data entity to identify and mark potential defects. The defect detection results may include detailed information such as the type, location, size, etc. of the defect, which are automatically identified by the visual inspection model.

[0118] Step 802: Compare the defect detection result with the labeled data corresponding to the first mirror data entity to obtain the defect detection rate, false detection rate and missed detection rate.

[0119] In some embodiments, after defect detection is completed, the system needs to evaluate the accuracy of the detection model. To this end, the defect detection results generated by the model are compared with the actual annotated data previously prepared for the mirror data entity. The annotated data can be provided by professional annotators with accurate defect information, which serves as the true value to verify the detection results of the model. By comparing the detection results and the true value, the system can calculate the defect detection rate (the proportion of defects correctly identified), the false detection rate (the proportion of non-defects that are incorrectly marked), and the missed detection rate (the proportion of actual defects not discovered by the model) of the model in actual applications.

[0120] Step 803: Generate model test data according to the defect detection rate, false detection rate and missed detection rate.

[0121] Based on the comparison results in step 802, the system then generates model test data. These test data are intended to quantify the performance of the model. By statistically analyzing the defect detection rate, false detection rate, and missed detection rate of the model, we can fully understand the performance of the model in actual scenarios. These data provide key information for further development and optimization of the model, allowing developers to adjust the model to improve detection accuracy and reduce false positives.

[0122] like Fig. 9 As shown, the embodiment of the present application also provides a model testing device 900 based on a service grid, including:

[0123] The image collection module 901 is used to obtain a first inference request from a first production line client, and to collect images of the first inference request based on the service grid to obtain first image data; wherein the first image data includes a first image data entity and a first image data label;

[0124] In some embodiments, the module is responsible for implementing the image collection function, which is used to collect reasoning requests from production line clients and replicate these requests using the capabilities of the service grid to generate image data. These data are used to simulate actual work scenarios so that model testing can be performed without affecting the normal operation of the production line. The generated first image data will contain the actual data content (first image data entity) and related metadata (first image data label, such as UUID, timestamp, etc.).

[0125] The data storage module 902 is used to store the first mirror data entity and the first mirror data label in the collection container;

[0126] In some implementations, the data storage module processes the image data entities and tags received from the image acquisition module 901. It stores these data securely in a collection container, including an entity database and a metadata database or collection database. This module ensures data persistence for subsequent retrieval and analysis.

[0127] The model acquisition module 903 is used to acquire the first printed circuit board assembly visual inspection model and the screening reference information preset for the first printed circuit board assembly visual inspection model;

[0128] In some embodiments, the model acquisition module 903 acquires the visual inspection model of the PCB assembly to be tested and the screening benchmark information related thereto. For example, the screening benchmark may include a specific version of the model, a specific time period for testing, or a production line machine identification, etc. Such information is essential for subsequent identification and screening of data for testing.

[0129] A data screening module 904, configured to screen and obtain a first mirror data entity according to screening reference information and a first mirror data tag;

[0130] In some embodiments, the data screening module 904 selects and extracts the data stored in the collection container according to the screening benchmark information. It determines which data entities are suitable for model testing by analyzing the image data labels (such as production line and machine identification). In this way, data for specific usage scenarios can be screened out from a large amount of data to provide an accurate data set for model testing.

[0131] The model testing module 905 is used to test the model to be tested through the first mirror data entity to obtain model testing data.

[0132] The service grid-based model testing method, device, equipment and storage medium proposed in the embodiments of the present application, the method first obtains a first reasoning request from a first production line client, and performs mirror collection on the first reasoning request based on the service grid to obtain first mirror data; wherein the first mirror data includes a first mirror data entity and a first mirror data label; then, the first mirror data entity and the first mirror data label are stored in a collection container; next, a first model to be tested and screening benchmark information preset corresponding to the first model to be tested are obtained; then, based on the screening benchmark information, the first mirror data label is matched in the collection container to obtain the first mirror data entity; finally, a performance test is performed on the model to be tested through the first mirror data entity to obtain model test data. The embodiment of the present application uses a service grid to mirror the inference request data generated on the production line, and stores its data entities and data labels in a collection container. After obtaining the model to be tested, the corresponding data entities are collected in the collection container based on a specific data screening benchmark for the model to be tested, and model test data is obtained, so as to realize the unified collection and management of the real inference data generated on the production line, and apply it to the performance test of the production model. It is no longer necessary to compile the program separately for each inference service, thereby reducing the development complexity of the inference service.

[0133] The present application also provides an electronic device, including:

[0134] at least one memory;

[0135] at least one processor;

[0136] at least one program;

[0137] The program is stored in the memory, and the processor executes at least one program to implement the model testing method based on the service grid in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0138] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0139] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0140] The memory 1002 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1002 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1002, and the processor 1001 calls and executes the service grid-based model testing method of the embodiment of this application;

[0141] Input / output interface 1003, used to implement information input and output;

[0142] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0143] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0144] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0145] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned service grid-based model testing method is implemented.

[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0148] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0149] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0151] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0152] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one item (one) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0157] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A model testing method based on service grid, characterized in that: The method comprises: Obtain a first inference request from a first production line client, and perform image collection on the first inference request based on the service grid to obtain first image data; wherein the first image data includes a first image data entity and a first image data label; storing the first mirror data entity and the first mirror data label in a collection container; Acquire a first model to be tested and screening benchmark information preset corresponding to the first model to be tested; Based on the screening reference information, the first image data label is matched in the collection container to obtain the first image data entity; The performance test of the model to be tested is performed through the first mirror data entity to obtain model test data.

2. A service grid-based model testing method according to claim 1, characterized in that: The collection container includes an entity database and a collection database, and storing the first mirror data entity and the first mirror data tag in the collection container includes: Store the first mirror data entity in the entity database, and obtain a first entity storage address; The first mirror data tag and the first entity storage address are stored in the acquisition database.

3. A service grid-based model testing method according to claim 2, characterized in that: The first image data tag includes a first request creation time, and storing the first image data tag and the first entity storage address in the acquisition database includes: Obtaining the creation time of the first request; When the creation time of the first request is within a first preset time period, the first mirror data tag and the first entity storage address are stored in a first sub-table corresponding to the first preset time period in the acquisition database.

4. A service grid-based model testing method according to claim 3, characterized in that: The screening benchmark information includes a target time period, wherein the target time period matches the first preset time period, and the first image data label is matched in the collection container based on the screening benchmark information to obtain the first image data entity, including: Matching the target time period with the first preset time period in the collection database to obtain the first sub-table; The first entity storage address is obtained from the first sub-table, and the first mirror data entity is obtained according to the first entity storage address.

5. A service grid-based model testing method according to claim 4, characterized in that: The screening benchmark information also includes a target production line identifier and a target machine identifier, the first mirror data tag also includes a first production line identifier and a first machine identifier, and the method further includes: In the first sub-table, the target production line identifier is compared with the first production line identifier to obtain a first comparison result, and the target machine identifier is compared with the first machine identifier to obtain a second comparison result; Selecting a target image data tag from the plurality of first image data tags according to the first comparison result and the second comparison result; A target entity storage address corresponding to the target image data tag is determined, and the corresponding target image data entity is acquired from the target entity storage address.

6. A service grid-based model testing method according to claim 1, characterized in that: The model to be tested is a printed circuit board assembly visual inspection model, and the model to be tested is tested by using the first mirror data entity to obtain model test data, including: Calling the printed circuit board assembly visual inspection model through the first mirror data entity to perform defect inspection and obtain defect inspection results; Comparing the defect detection result with the labeled data corresponding to the first mirror data entity to obtain a defect detection rate, a false detection rate, and a missed detection rate; The model test data is generated according to the defect detection rate, the false detection rate and the missed detection rate.

7. A service grid-based model testing method according to claim 1, characterized in that: The obtaining of a first inference request from a first production line client and performing image collection on the first inference request based on the service grid to obtain first image data includes: Obtaining a first inference request sent by the first production line client; Based on the service grid, mirroring the first reasoning request according to a preset sampling rate to obtain the first mirror data entity; A first mirror data tag is configured for the first mirror data entity to obtain the first mirror data.

8. A model testing device based on a service grid, characterized in that: include: An image collection module, used to obtain a first inference request from a first production line client, and perform image collection on the first inference request based on the service grid to obtain first image data; wherein the first image data includes a first image data entity and a first image data label; A data storage module, used for storing the first mirror data entity and the first mirror data label in a collection container; A model acquisition module, used to acquire a first printed circuit board assembly visual inspection model and screening reference information preset for the first printed circuit board assembly visual inspection model; A data screening module, configured to screen and obtain the first mirror data entity according to the screening reference information and the first mirror data tag; The model testing module is used to test the model to be tested through the first mirror data entity to obtain model testing data.

9. An electronic device, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the service grid-based model testing method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the service grid-based model testing method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Artificial intelligence model test method and device, electronic equipment and storage medium

    CN114492764A

  • Vision algorithm model test method and system, electronic equipment and storage medium

    CN115185797A

  • Service calling method for grid environment application, computer equipment and storage medium

    CN115964193A

  • Detection method and device of online banking system and nonvolatile storage medium

    CN117745401A