A method for scheduling inspection and testing tests based on reinforcement learning agents
By adopting a scheduling method based on reinforcement learning agents in the inspection and testing test, the problem of low scheduling efficiency of inspection and testing tests is solved, intelligent scheduling and resource coordination is achieved, and detection efficiency and traceability are improved.
Patent Information
- Application Number
- CN202411835600.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-01
AI Technical Summary
In the prior art, inspection and testing test schedule efficiency is low, and there are problems such as repetitive labor, low automation level, equipment management difficulties and lack of inspection process tracking and management.
The inspection and testing test schedule method based on reinforcement learning agents is adopted to design the inspection and testing task schedule environment, collect and manage the test management data, human resources data and test equipment data of the task, establish a reward function model, realize the schedule decision-making agent through the neural network model, maximize the cumulative reward score, and train and adjust the decision parameters of the schedule decision-making agent.
The efficiency of inspection and testing test schedules has been improved, intelligent schedules have been achieved, the environmental limitations of the test schedules have been met, laboratory resources have been coordinated, and the traceability of testing and the manageability of sample storage have been improved.
Smart Images
Figure CN119740822A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of test scheduling, and in particular to a method for scheduling inspection and testing tests based on reinforcement learning agents. Background Art
[0002] In the traditional inspection and testing process of instruments and meters, a large amount of manual records and data analysis are often relied on, which is prone to errors and inefficient. Inspection and testing involve many professional standards, methods and experiences. These knowledge are often scattered among different personnel and departments, making it difficult to effectively integrate and share. It is urgent to build knowledge at different levels to promote centralized storage, rapid retrieval and cross-departmental sharing of knowledge. The operating standards between different laboratories or inspectors may not be unified, which affects the comparability and reliability of the inspection results. This requires the implementation of standardized processes and quality control systems to ensure the consistency and accuracy of inspection and testing.
[0003] In particular, the scheduling of inspection and testing tests is difficult due to repetitive labor, low automation level, difficult equipment management, and lack of tracking management of the testing process. The automated scheduling method used to improve scheduling efficiency usually fails to properly handle whether personnel and equipment can effectively apply the inspection and testing tasks, which may result in poor scheduling of personnel and equipment maintenance. Therefore, it is still necessary to manually review and adjust the scheduling management repeatedly; on the other hand, if it is sorted linearly, it may cause delays in the inspection and testing schedule; the lack of tracking management of the testing process makes it impossible to record and manage the entire flow of samples during the testing process, which also makes it impossible for the automated scheduling of inspection and testing to be executed as expected.
[0004] With the surge in the amount of testing data, traditional data management and analysis methods can no longer cope with the needs of inspection and testing test management and scheduling. It is necessary to use new-generation information technology to effectively process and analyze multi-type and multi-modal data, timely discover trends and anomalies, support deeper insights and decision-making, and solve the problem of intelligent scheduling of inspection and testing tests in a targeted manner. Summary of the invention
[0005] The purpose of the present invention is to provide a method for scheduling inspection and testing tests based on reinforcement learning agents in order to solve the problem of low efficiency in test scheduling, in view of the deficiencies of the above-mentioned prior art.
[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0007] The present invention provides a method for scheduling inspection and testing tests based on a reinforcement learning agent, the method comprising:
[0008] Step S1, design an inspection and testing task scheduling environment, collect and manage the test management data, human resources data and test equipment data of the task, simulate task scheduling and execution, and establish a reward function model to calculate the efficiency of scheduling production and the corresponding test result reward;
[0009] Step S2: Establish a state embedding agent, correspond the existing scheduling state to a feature space, and provide a decision basis for the scheduling decision agent in step S3 through state embedding;
[0010] Step S3: Establish a scheduling decision agent implemented by a neural network model, search all scheduling options to select relevant personnel and equipment, and set the time of the experiment. According to the reward mechanism of the task management environment, the scheduling decision agent conducts state search in the scheduling management environment, and trains and adjusts the decision parameters of the scheduling decision agent by maximizing the cumulative reward score;
[0011] Step S4: Continue training the neural network model until the model meets the preset training and verification conditions.
[0012] Optionally, in step S1, for data collection and management of the task scheduling environment, by collecting and organizing test management data, human resources data and test equipment data, the current inspection and testing test scheduling status is determined, including whether the test status is in progress, the remaining test days, the priority of the test, whether the tester is idle, whether the equipment is idle, and the equipment operation and maintenance status.
[0013] Optionally, in step S1, for data collection and management of the task scheduling environment, the relationship between human resources and test equipment and test testing is marked, and for fixed test requirements, whether the test personnel or test equipment are suitable for the relevant test is marked in the form of multi-hot coding; the minimum number of applicable personnel required for the relevant test is marked; for personnel who are currently not suitable for performing the test, when simulating task execution, they can obtain corresponding training by participating in relevant tests, and the task scheduling environment marks the number of trainings required for the relevant tests, and increases rewards for those who complete the training; personnel participating in the test as training are not counted in the number of applicable personnel required for the test.
[0014] Optionally, in step S1, the reward function of the task scheduling environment is modeled, and when simulating task execution, the exclusive conditions that need to be met in the scheduling under human resource and equipment constraints are set, including that testers who are already fully tasked cannot perform more tests, and that equipment that is already performing test tasks cannot perform two or more tests at the same time; when the scheduling of the simulated task violates the relevant constraints, the execution of the simulated task is stopped, and a negative feedback reward of a first preset order of magnitude is given.
[0015] Optionally, in step S1, for the reward function of the task scheduling environment, rewards are given for completing the test task within a preset time. Starting from the moment the test task is issued, every time the test is not completed after a certain moment, the reward function accumulates a negative feedback of a second preset order of magnitude. Combined with the priority of the test, the negative feedback is proportional to the priority of the test. The reward function of the task scheduling environment is also related to the maintenance status of the equipment. Based on the historical operation data and functional safety data of the equipment, the equipment status is simulated and updated at each moment to determine whether the test task can continue to be executed.
[0016] Optionally, in step S2, for the multi-hot encoding of the environment state, an embedding model from the multi-hot encoding of the state to the feature space is pre-trained using an automatic codec, and the encoding result of the automatic codec is used as the input of the scheduling decision agent.
[0017] Optionally, in step S3, for the scheduling decision agent implemented by the neural network model, all scheduling options are searched based on a neural network according to the task status, including testers, equipment to be used, and the insertion position of the new test task in the existing scheduling queue.
[0018] Optionally, the neural network model outputs a real value a between 0 and 1 as the basis for insertion in the queue; if it is a fixed insertion method, the task is inserted into the queue at the ath position in proportion; if it is a random training method, starting from the head of the queue, a random number between 0 and 1 is generated and compared with a. If a is less than the generated random number, the task is inserted into the corresponding position, otherwise the insertion position is moved back one position, and random numbers are continuously generated to confirm whether to insert, until the task is inserted or the insertion position is moved to the end of the queue.
[0019] Optionally, in step S4, model training allocates test tasks in a fixed or random insertion method according to the results of the scheduling decision agent, calculates the accumulated rewards based on the reward mechanism, and maximizes the reward value of the scheduling decision agent during the training process.
[0020] The beneficial effects of the present invention include:
[0021] The inspection and testing test scheduling method based on reinforcement learning agent provided by the present invention includes: step S1, designing an inspection and testing test task scheduling environment, collecting and managing test management data, human resources data and test equipment data of the task, simulating task scheduling and execution, and establishing a reward function model to calculate the efficiency of production based on scheduling and the corresponding test result reward; step S2, establishing a state embedding agent, corresponding the existing scheduling state to a feature space, and providing a decision basis for the scheduling decision agent in step S3 through state embedding; step S3, establishing a scheduling decision agent implemented by a neural network model, searching all scheduling options to select relevant personnel and equipment, and setting the time of the test. According to the reward mechanism of the task management environment, the scheduling decision agent performs state search in the scheduling management environment, and trains and adjusts the decision parameters of the scheduling decision agent by maximizing the cumulative reward score; step S4, continuously training the neural network model until the model meets the preset training and verification conditions. This application uses reinforcement learning technology based on neural networks to design an inspection and testing test scheduling environment, integrates the test management data, human resources data and test equipment data of the inspection and testing tests, as well as the constraints that need to be met for test scheduling, and establishes a scheduling decision neural network model to achieve intelligent scheduling that meets the constraints of the test scheduling environment, thereby improving the efficiency of test scheduling. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 A framework diagram of a method for scheduling inspection and testing tests based on a reinforcement learning agent provided in an embodiment of the present invention is shown;
[0024] Figure 2 A flow chart of a method for scheduling inspection and testing tests based on a reinforcement learning agent provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] In the traditional inspection and testing process of instruments and meters, a large amount of manual records and data analysis are often relied on, which is prone to errors and inefficient. With the surge in the amount of test data, traditional data management and analysis methods have been unable to cope with the needs of test management and scheduling. It is necessary to use new generation information technology to solve the problem of intelligent scheduling of test. To this end, the present invention proposes a method for scheduling test based on reinforcement learning agent.
[0027] Figure 1 A framework diagram of a method for scheduling inspection and testing tests based on a reinforcement learning agent provided in an embodiment of the present invention is shown; Figure 2 A flow chart of a method for scheduling inspection and testing tests based on a reinforcement learning agent provided in an embodiment of the present invention is shown.
[0028] like Figure 1 and Figure 2 As shown, the inspection and testing test scheduling method based on reinforcement learning agent provided by the present invention includes:
[0029] Step S1, design an inspection and testing task scheduling environment, collect and manage the test management data, human resources data and test equipment data of the task, simulate task scheduling and execution, and establish a reward function model to calculate the efficiency of scheduled production and the corresponding test result rewards.
[0030] Step S2: Establish a state embedding agent to map the existing scheduling state to a feature space. Through state embedding, a decision basis is provided for the scheduling decision agent in step S3.
[0031] Step S3, establish a scheduling decision agent implemented by a neural network model, search all scheduling options to select relevant personnel and equipment, and set the time for the experiment. According to the reward mechanism of the task management environment, the scheduling decision agent conducts state search in the scheduling management environment, and trains and adjusts the decision parameters of the scheduling decision agent by maximizing the cumulative reward score.
[0032] Step S4: Continue training the neural network model until the model meets the preset training and verification conditions.
[0033] In step S1, for data collection and management of the task scheduling environment, by collecting and organizing test management data, human resources data and test equipment data, the current inspection and testing test scheduling status is determined, including whether the test status is in progress, the remaining test days, the priority of the test, whether the tester is idle, whether the equipment is idle, and the equipment operation and maintenance status.
[0034] In step S1, for data collection and management of the task scheduling environment, the relationship between human resources and test equipment and test testing is marked. For fixed test requirements, whether the test personnel or test equipment are suitable for the relevant test is marked in the form of multi-hot coding; the minimum number of applicable personnel required for the relevant test is marked; for personnel who are currently not suitable for performing the test, they can obtain corresponding training by participating in relevant tests during the simulation task execution. The task scheduling environment marks the number of trainings required for the relevant test, and increases rewards for the personnel who complete the training. Personnel who participate in the test as training are not counted in the number of applicable personnel required for the test.
[0035] For the information on the number of training sessions, when executing the simulation task, you can consider increasing rewards for those who have completed the training; however, during the actual scheduling, whether a person is suitable for participating in the experiment cannot be automatically updated by the intelligent agent, and must be updated in the system after confirmation by an authorized person.
[0036] In step S1, the reward function of the task scheduling environment is modeled. When simulating the execution of the task, the exclusive conditions that need to be met in the scheduling under the human resource and equipment constraints are set, including that testers who are already fully tasked cannot perform more tests, and that the equipment that is already performing test tasks cannot perform two or more tests at the same time; when the scheduling of the simulated task violates the relevant constraints, the execution of the simulated task is stopped, and a negative feedback reward of a first preset order of magnitude (large order of magnitude) is given.
[0037] In step S1, for the reward function of the task scheduling environment, rewards are given for completing the test task within the preset time (as quickly as possible). Starting from the moment the test task is issued, every time the test is not completed after a certain moment, the reward function accumulates a negative feedback of a second preset order of magnitude (a small order of magnitude, obviously, the second preset order of magnitude is smaller than the first preset order of magnitude). Combined with the priority of the test, the negative feedback is proportional to the priority of the test. The reward function of the task scheduling environment also takes into account the maintenance status of the equipment, that is, the reward function of the task scheduling environment is also related to the maintenance status of the equipment. Based on the historical operation data and functional safety data of the equipment (such as mean time between failures, mean repair time, etc.), the equipment status is simulated and updated at each moment to determine whether the test task can continue to be executed.
[0038] In step S2, for the multi-hot encoding of the environment state, an embedding model from the multi-hot encoding of the state to the feature space is pre-trained using the automatic codec, and the encoding result of the automatic codec is used as the input of the scheduling decision agent.
[0039] In step S3, for the scheduling decision agent implemented by the neural network model, according to the task status, based on a neural network, it searches for possible scheduling options, including testers, equipment to be used, and the insertion position of the new test task in the existing scheduling queue.
[0040] The neural network model outputs a real value a between 0 and 1 as the basis for insertion in the queue. If it is a fixed insertion method (used for decision-making or training), insert the task into the queue in proportion to the ath position; if it is a random training method, start from the head of the queue and generate a random number between 0 and 1 to compare with a. If a is less than the generated random number, insert the task into the corresponding position, otherwise move the insertion position back one position and continue to generate random numbers to confirm whether to insert until the task is inserted or the insertion position moves to the end of the queue. In particular, when using the agent to decide the insertion position of the experimental task, the problem uses a fixed insertion method.
[0041] In step S4, the model training allocates the test tasks in a fixed or random insertion method according to the results of the scheduling decision agent, calculates the accumulated rewards based on the reward mechanism, and maximizes the reward value of the scheduling decision agent during the training process.
[0042] To summarize, this application utilizes reinforcement learning technology based on neural networks to design an inspection and testing test scheduling environment, integrates the test management data, human resources data, and test equipment data of the inspection and testing tests, as well as the constraints that need to be met for test scheduling, and establishes a scheduling decision neural network model to achieve intelligent scheduling that meets the constraints of the test scheduling environment, thereby improving the efficiency of test scheduling.
[0043] Introducing an artificial intelligence scheduling method based on reinforcement learning and intelligent agents in the inspection and testing test, through the simulation test scheduling environment and efficiency reward mechanism, it can effectively improve the efficiency of sample testing, and realize the coordination of various resources in the laboratory, and understand the working status of each device and personnel in the laboratory, the number of samples to be tested in different links, and other information in real time, which can timely reflect the progress of work, realize the rational arrangement of work, and shorten the sample testing cycle; at the same time, improve the traceability of detection and the manageability of sample storage. The present invention can combine the capabilities and data of the existing laboratory information system, the regulations and resources of the enterprise, and apply artificial intelligence technology to achieve flexible scheduling management; the flexible scheduling module can use artificial intelligence algorithms to automatically generate and dynamically adjust the optimal execution plan of the inspection and testing project according to the current resource status, project priority, equipment capabilities and customer needs.
[0044] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it. They cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for scheduling inspection and testing tests based on reinforcement learning agents, characterized in that: The method comprises: Step S1, design an inspection and testing task scheduling environment, collect and manage the test management data, human resources data and test equipment data of the task, simulate task scheduling and execution, and establish a reward function model to calculate the efficiency of scheduling production and the corresponding test result reward; Step S2: Establish a state embedding agent, correspond the existing scheduling state to a feature space, and provide a decision basis for the scheduling decision agent in step S3 through state embedding; Step S3: Establish a scheduling decision agent implemented by a neural network model, search all scheduling options to select relevant personnel and equipment, and set the time of the experiment. According to the reward mechanism of the task management environment, the scheduling decision agent conducts state search in the scheduling management environment, and trains and adjusts the decision parameters of the scheduling decision agent by maximizing the cumulative reward score; Step S4: Continue training the neural network model until the model meets the preset training and verification conditions.
2. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 1 is characterized in that: In step S1, for data collection and management of the task scheduling environment, by collecting and organizing test management data, human resources data and test equipment data, the current inspection and testing test scheduling status is determined, including whether the test status is in progress, the remaining test days, the priority of the test, whether the tester is idle, whether the equipment is idle, and the equipment operation and maintenance status.
3. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 1 is characterized in that: In step S1, for data collection and management of the task scheduling environment, the relationship between human resources and test equipment and test testing is marked. For fixed test requirements, whether the test personnel or test equipment are suitable for the relevant test is marked in the form of multi-hot coding; the minimum number of applicable personnel required for the relevant test is marked; for personnel who are currently not suitable for performing the test, when simulating task execution, they can obtain corresponding training by participating in relevant tests. The task scheduling environment marks the number of trainings required for the relevant test and increases rewards for those who have completed the training; personnel participating in the test as training are not counted in the number of applicable personnel required for the test.
4. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 1 is characterized in that: In step S1, the reward function of the task scheduling environment is modeled. When simulating the execution of the task, the exclusive conditions that need to be met in the scheduling under the human resource and equipment constraints are set, including that testers who are already fully tasked cannot perform more tests, and that the equipment that is already performing test tasks cannot perform two or more tests at the same time; when the scheduling of the simulated task violates the relevant constraints, the execution of the simulated task is stopped, and a negative feedback reward of a first preset order of magnitude is given.
5. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 1 is characterized in that: In step S1, for the reward function of the task scheduling environment, rewards are given for completing the test task within the preset time. Starting from the moment the test task is issued, every time the test is not completed after a certain moment, the reward function accumulates a negative feedback of a second preset order of magnitude. Combined with the priority of the test, the negative feedback is proportional to the priority of the test. The reward function of the task scheduling environment is also related to the maintenance status of the equipment. Based on the historical operation data and functional safety data of the equipment, the equipment status is simulated and updated at each moment to determine whether the test task can continue to be executed.
6. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 3 is characterized in that: In step S2, for the multi-hot encoding of the environment state, an embedding model from the multi-hot encoding of the state to the feature space is pre-trained using the automatic codec, and the encoding result of the automatic codec is used as the input of the scheduling decision agent.
7. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 1 is characterized in that: In step S3, for the scheduling decision agent implemented by the neural network model, according to the task status, based on a neural network, all scheduling options are searched, including testers, equipment to be used, and the insertion position of the new test task in the existing scheduling queue.
8. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 7 is characterized in that: The neural network model outputs a real value a between 0 and 1 as the basis for insertion in the queue; if it is a fixed insertion method, the task is inserted into the queue at the ath position in proportion; if it is a random training method, it starts from the head of the queue and generates a random number between 0 and 1 and compares it with a. If a is less than the generated random number, the task is inserted into the corresponding position, otherwise the insertion position is moved back one position, and random numbers are continuously generated to confirm whether to insert, until the task is inserted or the insertion position is moved to the end of the queue.
9. The inspection and testing test scheduling method based on reinforcement learning agent according to claim 1 is characterized in that: In step S4, the model training allocates the test tasks in a fixed or random insertion method according to the results of the scheduling decision agent, calculates the accumulated rewards based on the reward mechanism, and maximizes the reward value of the scheduling decision agent during the training process.