Generative Training Contextual Simulation Testing System
Patent Information
- Authority / Receiving Office
- TW · TW
- Patent Type
- Applications
- Current Assignee / Owner
- AI NEXT MINE TECHNOLOGY CO LTD
- Filing Date
- 2025-01-13
- Publication Date
- 2026-07-16
AI Technical Summary
Existing systems face challenges in efficiently training new employees due to labor shortages of experienced personnel, necessitating effective training methods for new employees, particularly in fields requiring on-site operational capabilities.
A generative training scenario simulation test system utilizing a camera device, processing device, and servers storing pre-trained models to generate test images, recognition results, and text questions, jointly verifying practical ability and background knowledge of trainees.
The system efficiently trains and tests trainees by detecting errors and generating targeted questions, reinforcing learning outcomes and improving training efficiency.
Smart Images
Figure TWG2TA001068061_001 
Figure TWG2TA001068061_002 
Figure TWG2TA001068061_003
Abstract
Description
Technical Field
[0001] The present invention relates to a test system, in particular to a generative training scenario simulation test system. Prior Art
[0002] Generally speaking, new employees often require training and testing from senior staff upon entering the workplace. For example, mechanical maintenance personnel require not only a certain level of professional knowledge but also on-site operational capabilities.
[0003] However, due to labor shortages, it is difficult to allocate experienced personnel to conduct training and testing for new employees. Clearly, how to effectively train new employees is a critical issue that cannot be ignored today. Summary of the Invention
[0004] Therefore, the purpose of the present invention is to provide a generative training scenario simulation test system that can efficiently train a group of trainees.
[0005] The generative training scenario simulation test system of the present invention is suitable for use in an examination room and is signal-connected to a first server storing a pre-trained object detection model, a pre-trained action detection model, a pre-trained behavior classification model, a pre-trained behavior detection model, and standard operating procedure information, and a second server storing a natural language generation model. The generative training scenario simulation test system includes a camera device and a processing device. The camera device is configured to be placed in the examination room and to generate a test image corresponding to a real-time image of the examination room. The processing device includes a data collection module that is signal-connected to the camera device, a storage module for storing a prompt template, and a computing module that is electrically connected to the data collection module and the storage module and is signal-connected to the first server end and the second server end. The data collection module is used to receive the test image from the camera device, and the computing module is used to receive the test image from the data collection module and send the test image to the first server end to receive a recognition result from the first server end. The recognition result is generated by the pre-trained object detection model, the pre-trained action detection model, the pre-trained behavior classification model and the pre-trained behavior detection model according to the test image relative to the standard operating procedure information. The computing module is used to send the prompt template and the recognition result to the second server end to receive a text test question from the second server end and generated by the natural language generation model according to the prompt template and the recognition result.
[0006] Therefore, the generative training scenario simulation test system of the present invention is suitable for use in an examination room. The generative training scenario simulation test system includes a camera device and a processing device. The camera device is configured to be installed in the examination room and to generate a test image corresponding to a real-time image of the examination room. The processing device includes a data collection module, a storage module, and a computing module electrically connected to the data collection module and the storage module. The data collection module is signal-connected to the camera device and is configured to receive the test image from the camera device. The storage module is configured to store a prompt template, standard operating procedure information, a built-in pre-trained object detection model, a built-in pre-trained action detection model, a built-in pre-trained behavior classification model, a built-in pre-trained behavior detection model, and a built-in natural language generation model. The computing module is configured to receive the test image from the data collection module, generate a recognition result corresponding to the test image using the built-in pre-trained object detection model, and then generate a text test question using the built-in natural language generation model based on the prompt template and the recognition result.
[0007] The effectiveness of the present invention is that by arranging the camera device and the processing device to receive or generate the recognition result and the text test question, the practical ability and background knowledge of the trainee are jointly verified, thereby more efficiently training and testing the trainee. Simple diagram description
[0008] Other features and effects of the present invention will be clearly presented in the embodiments with reference to the accompanying drawings, in which: Figure 1 is a block diagram of a first embodiment of a generative training scenario simulation test system of the present invention; and FIG2 is a block diagram of a second embodiment of the generative training scenario simulation test system of the present invention. Implementation Method
[0009] Before the present invention is described in detail, it should be noted that similar elements are represented by the same reference numerals in the following description.
[0010] Referring to FIG. 1 , a first embodiment of a generative training scenario simulation test system according to the present invention is adapted for use in an examination room (not shown) and is signal-connected to a first server 91 and a second server 92 storing a natural language generation model 921. The first server 91 stores a pre-trained object detection model 911, a pre-trained action detection model 912, a pre-trained behavior classification model 913, a pre-trained behavior detection model 914, standard operating procedure information, and an efficiency evaluation algorithm.
[0011] In this embodiment, the testing area provides a simulated testing area for trainees to perform simulated mechanical maintenance. The pre-trained object detection model 911 is implemented using existing object detection algorithms such as YOLO or Faster R-CNN. The pre-trained object detection model 911 is trained by providing annotated training images, thereby achieving accurate object detection and recognition capabilities. The pre-trained action detection model 912 is implemented using existing open-source tools such as OpenPose or Mediapipe. The pre-trained action classification model 913 is developed based on existing deep learning models such as CNN or RNN to distinguish different action segments within an image. The pre-trained action detection model 914 is developed based on a long short-term memory network (LSTM) model to determine the differences between each action segment and the standard operating procedure information. The efficiency evaluation algorithm uses the dynamic time warping (DTW) method. The natural language generation model 921 uses the existing GPT model combined with a corpus in the mechanical field. It fine-tunes prompts through the tools provided by the ChatGPT API to generate different text test questions.
[0012] The first embodiment includes a camera device 1 , a sensing device 2 , a processing device 3 , a temperature and humidity adjustment device 4 , and a brightness adjustment device 5 .
[0013] The camera device 1 is used to be installed in the test field. The camera device 1 is used to generate a test image corresponding to the real-time image of the test field. In this embodiment, the camera device 1 is a surveillance camera (CCTV).
[0014] The sensing device 2 is configured to be installed in the test field. The sensing device 2 includes a temperature sensor 21 for generating temperature data, a humidity sensor 22 for generating humidity data, and a brightness sensor 23 for generating brightness data. The temperature data corresponds to the real-time temperature of the test field. The humidity data corresponds to the real-time humidity of the test field. The brightness data corresponds to the real-time brightness of the test field. In variations of this embodiment, the sensing device 2 may include only one or two of the temperature sensor 21, the humidity sensor 22, and the brightness sensor 23.
[0015] The processing device 3 includes a data collection module 31, a storage module 32 for storing a prompt template, and a computing module 33 electrically connected to the data collection module 31 and the storage module 32 and signal-connected to the temperature and humidity control device 4 and the brightness control device 5. The data collection module 31 is signal-connected to the camera 1, the temperature sensor 21, the humidity sensor 22, and the brightness sensor 23. The computing module 33 is signal-connected to the first server terminal 91 and the second server terminal 92.
[0016] In this embodiment, the processing device 3 is implemented using an integrated circuit, computer, or server with computing and storage capabilities, and illustratively includes hardware components such as a processor, memory, communication circuit, input interface, and output interface, which are not shown in the figure.
[0017] The temperature and humidity control device 4 is used to be installed in the test site. In this embodiment, the temperature and humidity control device 4 is a smart air conditioner with Internet of Things function, which can be remotely controlled by the processing device 3 to adjust the temperature and humidity of the test site.
[0018] The brightness adjustment device 5 is used to be set in the test field. In this embodiment, the brightness adjustment device 5 is a smart lamp with Internet of Things function, which can be remotely controlled by the processing device 3 to adjust the brightness of the test field.
[0019] During actual operation, the data collection module 31 receives at least one of the temperature data, the humidity data, and the brightness data, and the test image from the camera device 1 and the sensing device 2, and then the computing module 33 receives at least one of the temperature data, the humidity data, and the brightness data, and the test image from the data collection module 31.
[0020] Secondly, the computing module 33 controls the temperature and humidity adjustment device 4 and the brightness adjustment device 5 according to at least one of the temperature data, the humidity data, and the brightness data, thereby simulating the temperature, humidity, and brightness of a standard test site.
[0021] Next, the computing module 33 sends the test image to the first server 91, which then waits for a recognition result from the first server 91. The recognition result is generated by the pre-trained object detection model 911, the pre-trained action detection model 912, the pre-trained action classification model 913, and the pre-trained action detection model 914 based on the correspondence between the test image and the standard operating procedure information. For example, the test image is a recording of the trainee performing simulated mechanical maintenance. The test image shows that the trainee did not securely tighten a screw on the machine. The first server 91 uses the pre-trained object detection model 911 to identify an object feature (e.g., a hand tool, a screw) that appears during a specific time period in the test image, and uses the pre-trained action detection model 912 to identify an action feature (turning a screw) that appears during a specific time period in the test image. Next, the pre-trained behavior classification model 913 and the pre-trained behavior detection model 914 are used to determine whether the object features and the action features match the corresponding steps in the standard operating procedure information, thereby determining whether the object features and the action features presented in the test image are "correct" or "incorrect." Finally, if the first server 91 determines that the object features and the action features presented in the test image are correct, the efficiency evaluation algorithm is further used to calculate the degree of match between the training personnel's operation time and the time in the standard operating procedure information, thereby determining whether the object features and the action features presented in the test image are "unskilled." Therefore, the recognition result indicates whether the behavior feature formed by the object features and the action features presented in the test image is "correct," "incorrect," or "unskilled."
[0022] Finally, the computing module 33 sends the prompt template and the recognition result to the second server 92, which then waits to receive a text question generated by the natural language generation model 921 from the second server 92. The text question corresponds to the prompt template and the recognition result. For example, the prompt template sent by the computing module 33 is "Please analyze the following content and infer and generate a relevant question," and the recognition result sent is "The behavior characteristic presented by the test image is incorrect." The second server 92 uses the natural language generation model 921 to generate content related to the behavior characteristic and use it to generate the text question, thereby strengthening the trainee's understanding of the operating principles or standard procedures.
[0023] Therefore, the first embodiment can detect "errors" or "lack of proficiency" in the test image (implementation) and dynamically generate text questions corresponding to the trainee, thereby reinforcing training focus and improving learning outcomes. By providing the recognition results and text questions, the first embodiment jointly verifies the trainee's practical skills and background knowledge, thereby more efficiently training and testing the trainee.
[0024] Referring to FIG2 , a second embodiment of the generative training scenario simulation test system of the present invention is also applicable to the test site. This second embodiment also includes the camera device 1 , the sensor device 2 , the processing device 3 , the temperature and humidity adjustment device 4 , and the brightness adjustment device 5 .
[0025] The main difference between the second embodiment and the first embodiment is that the processing device 3 includes a data collection module 31, a storage module 32, and a computing module 33 electrically connected to the data collection module 31 and the storage module 32, and signal-connected to the temperature and humidity control device 4 and the brightness control device 5. The data collection module 31 is signal-connected to the camera 1, the temperature sensor 21, the humidity sensor 22, and the brightness sensor 23. The storage module 32 is used to store the prompt template, the standard operating procedure information, the efficiency evaluation algorithm, a built-in pre-trained object detection model 321, a built-in pre-trained action detection model 322, a built-in pre-trained behavior classification model 323, a built-in pre-trained behavior detection model 324, and a built-in natural language generation model 325. The computing module 33 is signal-connected to the first server 91 and the second server 92.
[0026] In this embodiment, the processing device 3 is implemented using an integrated circuit, computer, or server with computing and storage capabilities, and illustratively includes hardware components (not shown) such as a processor, memory, communication circuits, input interfaces, and output interfaces. The built-in pre-trained object detection model 321 is also implemented using existing object detection algorithms such as YOLO or Faster R-CNN. The built-in pre-trained action detection model 322 is also implemented using existing open-source tools such as OpenPose or Mediapipe. The built-in pre-trained behavior classification model 323 is also developed based on existing deep learning models such as CNN or RNN. The built-in pre-trained behavior detection model 324 is also developed based on the long short-term memory network model (LSTM). The efficiency evaluation algorithm also uses the dynamic time warping (DTW) method. The built-in natural language generation model 325 also uses an existing GPT model combined with a corpus in the mechanical field. It uses the tools provided by the ChatGPT API to fine-tune prompts to generate different text test questions.
[0027] During actual operation, the data collection module 31 receives at least one of the temperature data, the humidity data, and the brightness data, and the test image from the camera device 1 and the sensing device 2, and then the computing module 33 receives at least one of the temperature data, the humidity data, and the brightness data, and the test image from the data collection module 31.
[0028] Secondly, the computing module 33 controls the temperature and humidity adjustment device 4 and the brightness adjustment device 5 according to at least one of the temperature data, the humidity data, and the brightness data, thereby simulating the temperature, humidity, and brightness of a standard test site.
[0029] Then, the computing module 33 utilizes the built-in pre-trained object detection model 321, the built-in pre-trained action detection model 322, the built-in pre-trained behavior classification model 323 and the built-in pre-trained behavior detection model 324 to generate the recognition result according to the test image.
[0030] Finally, the computing module 33 utilizes the built-in natural language generation model 325 to generate the text test question based on the prompt template and the recognition result. It is worth noting that after the trainee completes the text test question, they can be asked to perform simulated mechanical maintenance again, thereby obtaining an updated test image. The processing device 3 then generates an updated recognition result based on the test image. If the updated recognition result still indicates that the behavioral characteristics presented in the updated test image are "error" or "unskilled," an updated text test question is generated based on the prompt template and the updated recognition result, forming a new learning cycle. In this way, the second embodiment can provide specific data on the improvement in the new round of simulated mechanical maintenance compared to the previous round, such as reduced time consumption, fewer errors, and so on. Furthermore, the text test questions generated in each round are recorded by the computing module 33 in the storage module 32 for subsequent analysis.
[0031] Therefore, the second embodiment can also detect "errors" or "lack of proficiency" in the test image (practice), and dynamically generate the text test questions corresponding to the trainee, effectively reinforcing training points and improving learning outcomes. By providing the recognition results and the text test questions, the second embodiment jointly verifies the trainee's practical skills and background knowledge, thereby more efficiently training and testing the trainee.
[0032] Therefore, the first embodiment and the second embodiment of the generative training scenario simulation test system of the present invention have the following effects:
[0033] (1) In the first embodiment, the camera device 1 and the processing device 3 are arranged to cooperate with the first server end 91 and the second server end 92 to receive the recognition result and the text test question, thereby jointly verifying the practical ability and background knowledge of the trainee, thereby more efficiently training and testing the trainee.
[0034] (2) In the second embodiment, by setting up the camera device 1 and the processing device 3, and utilizing the built-in pre-trained object detection model 321, the built-in pre-trained action detection model 322, the built-in pre-trained behavior classification model 323, the built-in pre-trained behavior detection model 324 and the built-in natural language generation model 325 to generate the recognition result and the text test question, thereby jointly verifying the practical ability and background knowledge of the trainee, thereby more efficiently training and testing the trainee.
[0035] (3) In the first and second embodiments, the sensing device 2 is provided to generate the temperature data, the humidity data, and the brightness data, and the temperature and humidity adjustment device 4 and the brightness adjustment device 5 are provided to adjust the temperature, humidity, and brightness of the test field, so that the processing device 3 can cooperate with the temperature, humidity, and brightness of the simulated standard test field, thereby more efficiently training and testing the trainees.
[0036] To sum up, the generative training scenario simulation test system of the present invention can indeed significantly improve training efficiency and can indeed achieve the purpose of the present invention.
[0037] However, the above is merely an embodiment of the present invention and should not be used to limit the scope of implementation of the present invention. All simple equivalent changes and modifications made according to the scope of the patent application and the content of the patent specification of the present invention are still within the scope of the patent of the present invention.
[0038] 1: Camera 2: Sensing device 21: Temperature sensor 22: Humidity sensor 23: Brightness sensor 3: Processing device 31: Data Collection Module 32: Storage module 321: Built-in pre-trained object detection model 322: Built-in pre-trained motion detection model 323: Built-in pre-trained behavior classification model 324: Built-in pre-trained behavior detection model 325: Built-in natural language generation model 33: Computing module 4: Temperature and humidity control device 5: Brightness adjustment device 91: First server 911: Pre-trained object detection model 912: Pre-trained motion detection model 913: Pre-trained behavior classification model 914: Pre-trained behavior detection model 92: Second server 921: Natural Language Generation Model
Claims
1. A generative training scenario simulation test system, suitable for an examination room, and signal-connected to a first server end storing a pre-trained object detection model, a pre-trained action detection model, a pre-trained behavior classification model, a pre-trained behavior detection model, and standard operating procedure information, and a second server end storing a natural language generation model, the generative training scenario simulation test system comprising: a camera device for being set in the examination room, the camera device for generating a test image corresponding to a real-time image of the examination room; and a processing device, including a data collection module signal-connected to the camera device, a storage module for storing a prompt template, and a computing module electrically connected to the data collection module and the storage module and for signal-connecting the first server end and the second server end, the data collection module for receiving the test image from the camera device, the computing module for receiving the test image from the data collection module, and sending the test image to the processing device. The image is sent to the first server end to receive a recognition result from the first server end, the recognition result is generated by the pre-trained object detection model, the pre-trained action detection model, the pre-trained behavior classification model and the pre-trained behavior detection model according to the corresponding test image relative to the standard operating procedure information, and the computing module is used to send the prompt template and the recognition result to the second server end to receive a text test question generated by the natural language generation model from the second server end according to the prompt template and the recognition result.
2. The generative training scenario simulation test system as described in claim 1 further includes a sensing device and a temperature and humidity adjustment device for being set up in the examination room, the sensing device includes a temperature sensor for generating temperature data, the temperature data corresponds to the real-time temperature of the examination room, the temperature and humidity adjustment device signal is connected to the computing module, the data collection module signal is connected to the temperature sensor to receive the temperature data from the temperature sensor, the computing module is used to receive the temperature data from the data collection module, and control the temperature and humidity adjustment device according to the temperature data, thereby adjusting the temperature of the examination room.
3. The generative training scenario simulation test system as described in claim 1 further includes a sensing device and a temperature and humidity adjustment device for being set up in the examination room, the sensing device includes a humidity sensor for generating humidity data, the humidity data corresponds to the real-time humidity of the examination room, the temperature and humidity adjustment device signal is connected to the computing module, the data collection module signal is connected to the humidity sensor to receive the humidity data from the humidity sensor, the computing module is used to receive the humidity data from the data collection module, and control the temperature and humidity adjustment device according to the humidity data, thereby adjusting the humidity of the examination room.
4. The generative training scenario simulation test system as described in claim 1 further includes a sensing device and a brightness adjustment device for being set up in the examination room, the sensing device includes a brightness sensor for generating brightness data, the brightness data corresponds to the real-time brightness of the examination room, the brightness adjustment device signal is connected to the computing module, the data collection module signal is connected to the brightness sensor to receive the brightness data from the brightness sensor, the computing module is used to receive the brightness data from the data collection module, and control the brightness adjustment device according to the brightness data, thereby adjusting the brightness of the examination room.
5. A generative training scenario simulation test system, suitable for an examination room, the generative training scenario simulation test system comprising: a camera device for being set in the examination room, the camera device for generating a test image corresponding to a real-time image of the examination room; and a processing device, comprising a data collection module, a storage module, and a computing module electrically connected to the data collection module and the storage module, the data collection module being signal-connected to the camera device and for receiving the test image from the camera device, the storage module being used to store a prompt template, standard operating procedure information, a built-in pre-trained object detection model, a built-in pre-trained action detection model, a built-in pre-trained behavior classification model, a built-in pre-trained behavior detection model, and a built-in natural language generation model, the computing module being used to receive the test image from the data collection module, and using the built-in pre-trained object detection model to generate a recognition result corresponding to the test image, and then using the built-in natural language generation model to generate a text test question according to the prompt template and the recognition result.
6. The generative training scenario simulation test system as described in claim 5 further includes a sensing device and a temperature and humidity adjustment device for being set up in the examination room, the sensing device includes a temperature sensor for generating temperature data, the temperature data corresponds to the real-time temperature of the examination room, the temperature and humidity adjustment device signal is connected to the computing module, the data collection module signal is connected to the temperature sensor to receive the temperature data from the temperature sensor, the computing module is used to receive the temperature data from the data collection module, and control the temperature and humidity adjustment device according to the temperature data, thereby adjusting the temperature of the examination room.
7. The generative training scenario simulation test system as described in claim 5 further includes a sensing device and a temperature and humidity adjustment device for being set up in the examination room, the sensing device includes a humidity sensor for generating humidity data, the humidity data corresponds to the real-time humidity of the examination room, the temperature and humidity adjustment device signal is connected to the computing module, the data collection module signal is connected to the humidity sensor to receive the humidity data from the humidity sensor, the computing module is used to receive the humidity data from the data collection module, and control the temperature and humidity adjustment device according to the humidity data, thereby adjusting the humidity of the examination room.
8. The generative training scenario simulation test system as described in claim 5 further includes a sensing device and a brightness adjustment device for being set up in the examination room, the sensing device includes a brightness sensor for generating brightness data, the brightness data corresponds to the real-time brightness of the examination room, the brightness adjustment device signal is connected to the computing module, the data collection module signal is connected to the brightness sensor to receive the brightness data from the brightness sensor, the computing module is used to receive the brightness data from the data collection module, and control the brightness adjustment device according to the brightness data, thereby adjusting the brightness of the examination room.