Detection system and detection method of interlocking human-computer interface operation display system

Abstract

The invention relates to the technical field of rail transit signal system detection, in particular to a detection system and a detection method of an interlocking human-computer interface operation display system, and the detection system comprises a central processing unit, a data storage module, a data acquisition module, an image recognition module, a character recognition module, a voice recognition module and a simulation mouse operation module. And a man-machine interaction processing module. The detection method comprises the steps of S1, constructing a standard database, S2, initializing an interlocking human-computer interface to obtain a to-be-detected image, a to-be-detected character and a to-be-detected voice, S3, comparing the to-be-detected image, the to-be-detected character and the to-be-detected voice with a standard image, a standard character and a standard voice character respectively; s4, respectively setting a similarity threshold value for the image, the character and the voice; s5, mouse operation test; and S6, detection report output. By means of image recognition, character recognition and voice recognition, detection of an interlocking human-computer interface and detection of mouse operation are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of railway traffic signal system detection, and specifically relates to a detection system and a detection method for an interlocking human-machine interface operation display system. Background Art

[0002] A computer interlocking system (hereinafter referred to as interlocking) is a signal system that realizes station interlocking by using a computer as the main technical means. The interlocking device performs logical operations on the operation commands of the station operator and the information indicating the status of on-site equipment, realizes centralized control of signal lamps, switches, and routes, etc., and completes the basic functions of interlocking such as route establishment, locking, unlocking, switch control, and signal lamp control to ensure train operation safety and improve transportation efficiency. Interlocking is applied in high-speed railways, conventional railways, and urban rail transit.

[0003] The interlocking human-machine interface mainly displays the topological structure and status information of station signal equipment, the settings and distributions of operation buttons, the names of the station yard and equipment, relevant prompt and alarm information, etc., and provides a human-computer interaction interface for relevant interlocking operations. The operation display interface should give full play to the advantages of computers in processing images and texts, use colors with more than 16 color bits and standard font library fonts, provide clear and accurate displays, ensure that the geometric dimensions and fonts of the same type of identifiers are consistent, update the information that has changed in a timely manner, and respond to relevant operations. It meets the display and operation requirements for the interlocking human-machine interface in standards such as "TB / T 3027-2015 Technical Conditions for Railway Station Computer Interlocking", "TB / T3027-2015 Amendment No. 1 to Technical Conditions for Railway Station Computer Interlocking", "TB / T 3537-2018 Test Specifications for Railway Station Computer Interlocking", "TB / T 3578-2022 Operation Display Specifications for Railway Station Computer Interlocking", "TJ / DW188-2016 Technical Specifications for Operation Display of Station Computer Interlocking", and "Q / CR 930-2024 Operation Display Specifications for Railway Station Computer Interlocking".

[0004] At present, the display test of the interlocking human-machine interface requires manual operation of the mouse, relying on the human eye to judge the test results and manual recording, with low efficiency, easy to make mistakes, and unable to accurately calculate the RGB values of the display interface. The combination of a camera and a manipulator is also a common method. The process of inputting test cases is cumbersome, the actions performed by the manipulator are time-consuming, the accuracy of the camera image recording is not high, the test volume is large and there are a large number of repetitive operations, especially when traversing the entire station yard for testing. At the same time, due to the uneven knowledge levels and test experiences of testers, it is difficult to ensure the completeness and consistency of the test, and since there are still many operations that require manual intervention, the quality and credibility of the interlocking test are difficult to guarantee. Summary of the Invention

[0005] To solve the above problems, the present invention provides a detection system and a detection method for an interlocking human-machine interface operation display system, which realize the detection of the interlocking human-machine interface and the detection of mouse operations through image recognition, character recognition, and voice recognition.

[0006] The technical solution adopted by the present invention is to provide a detection system for an interlocking human-machine interface operation display system, including

[0007] a central processing unit for data processing and receiving and sending instructions,

[0008] a data storage module for storing standard data, including standard images, standard characters, and standard voice texts;

[0009] a data acquisition module for acquiring images, characters, and voices of service elements to be measured in the interlocking human-machine interface, obtaining the images to be measured, the characters to be measured, and the voices to be measured;

[0010] an image recognition module for comparing the images to be measured with the standard images to obtain a similarity;

[0011] a character recognition module for matching the characters to be measured with the standard characters to obtain a similarity;

[0012] a voice recognition module for converting the voice to be measured into text through SpeechSDK and matching it with the standard voice text to obtain a similarity;

[0013] a simulated mouse operation module for detecting mouse operation instructions through manual mouse operation, including left mouse button click, right mouse button click, mouse movement, mouse scroll, and mouse drag;

[0014] a human-computer interaction processing module for receiving user input instructions and sending them to the central processing unit;

[0015] The central processing unit is respectively connected to the data storage module, the data acquisition module, the image recognition module, the character recognition module, the voice recognition module, the simulated mouse operation module, and the human-computer interaction processing module for data communication.

[0016] The data acquisition module acquires images of service elements to be measured in the interlocking human-machine interface, specifically including initializing the interlocking human-machine interface;

[0017] controlling the interlocking human-machine interface to sequentially display images of each service element,

[0018] sequentially taking screenshots of images of each service element;

[0019] processing the images obtained by screenshotting to obtain the images to be measured with the same size, resolution, and orientation as the standard images.

[0020] The data acquisition module obtains the text of the service elements to be measured in the interlocking man-machine interface. Specifically, it includes initializing the interlocking man-machine interface;

[0021] Controlling the interlocking man-machine interface to sequentially display the images of each service element,

[0022] Sequentially taking screenshots of the images of each service element;

[0023] Extracting text from the screenshots.

[0024] The image recognition module compares the image to be measured with the standard image by using two methods: image template matching and image feature matching, and calculates the matching degree between the image to be measured and the standard image.

[0025] Respectively set the similarity thresholds for images, text, and voice according to the detection requirements. If the similarity is greater than the threshold, it is determined that the detection is qualified.

[0026] The service elements include buttons, signal lights, switches, routes, indicator lights, signs, and warning prompts.

[0027] The present invention also provides a detection method for an interlocking man-machine interface operation display system, which is implemented in the above detection system and includes the following steps:

[0028] S1. Construct a standard database, including standard images, standard text, and standard voice text;

[0029] S2. Initialize the interlocking man-machine interface, control the interlocking man-machine interface to sequentially display the images, text, and voice of each service element, and obtain the image to be measured, the text to be measured, and the voice to be measured.

[0030] S3. Compare the image to be measured, the text to be measured, and the voice to be measured with the standard image, the standard text, and the standard voice text respectively to obtain the similarity;

[0031] S4. Respectively set the similarity thresholds for images, text, and voice. When the obtained similarity from the comparison is higher than the threshold, it is determined that the detection is qualified;

[0032] S5. Mouse operation test, detecting mouse operation instructions, including left mouse button click, right mouse button click, mouse movement, mouse scrolling, and mouse dragging;

[0033] S6. Output a detection report after the detection ends.

[0034] In step S2, obtain the image screenshots of each service element by taking screenshots, and after processing the image screenshots, obtain the image to be measured; extract text from the image screenshots to obtain the text to be measured.

[0035] In the step S3, the method for comparing the image to be measured and the standard image adopts image template matching and image feature matching, and the average value after adding the two matching values is used as the similarity value.

[0036] In the step S3, when comparing the voice to be measured with the standard voice text, the voice to be measured is converted into text by SpeechSDK and then compared with the standard voice text.

[0037] The beneficial effect of the present invention is to provide a detection system and a detection method for an interlocking human-machine interface operation display system. By using image recognition, text recognition and voice recognition methods, the images, texts and voices in the interlocking human-machine interface are recognized and compared with the standard images, texts and voices. Whether it is qualified is judged by the similarity. At the same time, through the simulated mouse operation module, the mouse operation can be detected, realizing high-precision and high-efficiency automatic detection of the interlocking human-machine interface and the mouse, saving a large amount of manpower and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is the system structure diagram of the present invention;

[0039] Figure 2 is the method flow chart of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0040] As Figure 1 shown, the present invention provides a detection system for an interlocking human-machine interface operation display system, including a central processing unit for data processing and receiving and sending instructions,

[0041] a data storage module for storing standard data, including standard images, standard texts and standard voice texts;

[0042] a data acquisition module for acquiring images, texts and voices of the service elements to be measured in the interlocking human-machine interface to obtain the images, texts and voices to be measured;

[0043] an image recognition module for comparing the image to be measured with the standard image to obtain the similarity;

[0044] a text recognition module for matching the text to be measured with the standard text to obtain the similarity;

[0045] a voice recognition module for converting the voice to be measured into text by SpeechSDK and matching it with the standard voice text to obtain the similarity;

[0046] a simulated mouse operation module for detecting mouse operation instructions by manually operating the mouse, including left mouse button click, right mouse button click, mouse movement, mouse scroll and mouse drag;

[0047] The human-computer interaction processing module is used to receive user input instructions and send them to the central processing unit;

[0048] The central processing unit is respectively connected to the data storage module, the data acquisition module, the image recognition module, the character recognition module, the speech recognition module, the analog mouse operation module and the human-computer interaction processing module for data communication.

[0049] As Figure 1 shown, the data acquisition module acquires the images of the service elements to be measured in the interlocking man-machine interface, specifically including,

[0050] Initializing the interlocking man-machine interface;

[0051] Controlling the interlocking man-machine interface to display the images of each service element in sequence,

[0052] Taking screenshots of the images of each service element in sequence;

[0053] Processing the images obtained by screenshot to obtain the images to be measured with the same size, resolution and orientation as the standard images.

[0054] As Figure 1 shown, the data acquisition module acquires the characters of the service elements to be measured in the interlocking man-machine interface, specifically including,

[0055] Initializing the interlocking man-machine interface;

[0056] Controlling the interlocking man-machine interface to display the images of each service element in sequence,

[0057] Taking screenshots of the images of each service element in sequence;

[0058] Extracting characters from the screenshots.

[0059] As Figure 1 shown, the image recognition module compares the images to be measured with the standard images, and adopts two methods of image template matching and image feature matching to calculate the matching degree between the images to be measured and the standard images,

[0060] As Figure 1 shown, the similarity thresholds of images, characters and voices are respectively set according to the detection requirements. If the similarity is greater than the threshold, it is determined that the detection is qualified.

[0061] As Figure 1 shown, the service elements include buttons, signal lights, switches, routes, indicator lights, signs and warning prompts.

[0062] As Figure 2 shown, the present invention provides a detection method for an interlocking man-machine interface operation display system, which is implemented in the above detection system. The detection method includes the following steps,

[0063] S1. Build a standard database, including standard images, standard texts, and standard speech texts;

[0064] S2. Initialize the interlocking human-machine interface, and control the interlocking human-machine interface to display the images, texts, and voices of each service element in sequence to obtain the images to be tested, the texts to be tested, and the voices to be tested.

[0065] S3. Compare the images to be tested, the texts to be tested, and the voices to be tested with the standard images, standard texts, and standard speech texts respectively to obtain the similarity.

[0066] S4. Set similarity thresholds for images, texts, and voices respectively. When the similarity obtained by comparison is higher than the threshold, it is determined that the detection is qualified.

[0067] S5. Mouse operation test, detect mouse operation instructions, including left mouse button click, right mouse button click, mouse movement, mouse scroll, and mouse drag;

[0068] S6. Output a test report after the detection is completed.

[0069] As Figure 2 shown, in the step S2, obtain the image screenshots of each service element by taking screenshots, and after processing the image screenshots, obtain the images to be tested; extract texts from the image screenshots to obtain the texts to be tested.

[0070] As Figure 2 shown, in the step S3, the comparison method between the image to be tested and the standard image adopts image template matching and image feature matching, and the average value after adding the two matching values is used as the similarity value.

[0071] As Figure 2 shown, in the step S3, when comparing the voice to be tested with the standard speech text, convert the voice to be tested into text through SpeechSDK and then compare it with the standard speech text.

[0072] The detection system of the present invention adopts a three-layer architecture, namely, an application layer, a service layer, and a storage layer;

[0073] The application layer is mainly the tappet human-computer interaction port, corresponding to the central processing unit, the human-computer interaction processing module, and the data acquisition module, and realizes the detection process control, including a test framework, remote operation display, and coordinate configuration tool.

[0074] The test framework is the core application of the application layer, mainly realizing test case writing, initiating the test process, controlling the test logic, feedback the test results, and exporting the test report.

[0075] The main function of the user operation display terminal is to receive commands from the custom interface library, implement screenshot and mouse operation commands, and at the same time call the plugin to implement voice recognition logic processing.

[0076] The main function of the coordinate configuration tool is to compile a coordinate configuration file for positioning in image recognition, text recognition, and voice recognition.

[0077] The main function of the business layer is to implement image recognition, text recognition, and voice recognition functions, corresponding to the image recognition module, text recognition module, voice recognition module, and simulated mouse operation module. The functions include the custom interface library of the open-source text recognition library, the open-source image recognition library, and the voice recognition plugin.

[0078] The main function of the open-source text recognition library is to provide a text recognition interface for recognizing the text content of the interlocking human-machine interface.

[0079] The main function of the open-source image recognition library is to provide an image recognition interface for recognizing the image information in the interlocking human-machine interface.

[0080] The main function of the voice recognition plugin is to collect voice information and convert the voice information into text content for the voice recognition function.

[0081] The main function of the custom interface library is to provide interfaces for image recognition, text recognition, mouse operation, and voice content comparison, which are used to assist in writing cases in the test framework and send commands to the remote display screen mirroring software.

[0082] The main function of the storage layer is to store coordinates, images, and voice translated into text information, corresponding to the data storage module. The function is mainly to provide relevant data information to the business layer, including the coordinate configuration file, screenshots of the interlocking simulation software human-machine interface, and voice translation text files.

[0083] Prerequisites for testing:

[0084] The prerequisite for testing is that after the test environment is available, some settings need to be made manually to the test environment or the interlocking simulation before the test script runs automatically. Only through this setting can the test requirements of certain specific cases be met. These preconditions are either global or specific to individual test cases. The global preconditions are set before all test cases are executed, and the settings specific to individual test cases can only be made before testing that case. Otherwise, it may affect the execution results of other test cases.

[0085] Regarding the image recognition module:

[0086] First, select the test object and determine whether the current man-machine interface of the interlocking software is in the initial state. Second, start screenshot processing. According to the test object, remotely operate the interlocking simulation software, and display the service elements to be tested, such as signal lights, turnouts, routes, etc. on the man-machine interface of the interlocking simulation software. Finally, according to the standard graphic template, determine whether the service elements displayed on the interlocking man-machine interface are consistent with the expectations in the screenshot, and restore the interlocking man-machine interface to the initial state.

[0087] The image recognition process mainly relies on the Opencv2 module of the Python open-source library, including two recognition methods, namely image template matching and image feature matching.

[0088] Image template matching mainly locates the position of the standard image in the target image through the Opencv2 module, and crops the target image according to the range of the standard image. The cropped image and the standard image are set as input parameters into the Opencv2 module for template matching. After the matching is completed, compare the matching degree in the matching result with the standard matching degree to determine whether the matching is successful.

[0089] For image feature matching, there are always unique pixel points in an image. These points can be considered as the features of this image, called feature points, such as the color and shape of the image. Image feature matching is based on feature points. Image feature recognition mainly locates the range to be recognized in the target image through the Opencv2 module, and performs recognition by counting the number of color pixels in the recognized range. Usually, the background color corresponds to the most color pixels, and the text corresponds to the second most color pixels.

[0090] Regarding the text recognition module

[0091] There is a large amount of text on the interlocking man-machine interface, mainly on each button, near signal lights, turnouts, sections, various indicator lights, warnings, and within prompt boxes. The interlocking operation display technical specifications have clear requirements for this text. In addition to checking whether this text meets the specifications, relevant devices such as signal lights, turnouts, and sections nearby can also be located by searching for the text.

[0092] The text processing process mainly relies on the easyocr module and the Opencv2 module of the Python open-source library. The Opencv2 module mainly locates the position of the text, crops the target image according to the text area, and extracts the image containing only the text part. Then, the cropped image is used as the input parameter for the easyocr module to recognize the text, reads the recognized text content, and compares the recognized text content with the expected standard text character by character.

[0093] Regarding the voice recognition module

[0094] The interlocking human-machine interface will send out alarm or prompt voice messages in certain situations, and display the service elements to be tested, such as routes, etc. on the interlocking simulation software human-machine interface. These voices are recognized through Microsoft's SpeechSDK software, and the voice messages are translated into text and then compared.

[0095] Regarding the simulated mouse operation module

[0096] According to the interlocking operation display specification, it is necessary to operate the interlocking human-machine interface so as to display the test target graphics on the human-machine interface. Therefore, the interlocking human-machine interface automatic test system supports the operation interfaces of left mouse button click, right mouse button click, mouse movement, mouse scrolling, and mouse dragging, which are referenced in the test cases to meet the needs of automatic testing of the interlocking control and display software.

Claims

1. A detection system for an interlocking human-machine interface operation display system, characterized in that: including, a central processing unit for data processing and receiving and sending instructions, a data storage module for storing standard data, including standard images, standard texts, and standard speech texts; a data acquisition module for acquiring images, texts, and voices of service elements to be measured in an interlocking man-machine interface, obtaining the images to be measured, the texts to be measured, and the voices to be measured; an image recognition module for comparing the image to be measured with the standard image to obtain a similarity; a text recognition module for matching the text to be measured with the standard text to obtain a similarity; a voice recognition module for converting the voice to be measured into text through SpeechSDK and matching it with the standard speech text to obtain a similarity; a simulated mouse operation module for detecting mouse operation instructions through manual mouse operation, including left mouse button click, right mouse button click, mouse movement, mouse scroll, and mouse drag; a man-machine interaction processing module for receiving user input instructions and sending them to the central processing unit; The central processing unit is respectively connected to the data storage module, the data acquisition module, the image recognition module, the text recognition module, the voice recognition module, the simulated mouse operation module, and the man-machine interaction processing module for data communication.

2. The detection system of an interlocking human-machine interface operation display system according to claim 1, characterized in that: The data acquisition module acquires the image of the service element to be measured in the interlocking man-machine interface, specifically including, initializing the interlocking man-machine interface; controlling the interlocking man-machine interface to sequentially display the images of each service element; sequentially taking screenshots of the images of each service element; processing the images obtained by screenshotting to obtain the images to be measured with the same size, resolution, and orientation as the standard images.

3. The detection system of an interlocking human-machine interface operation display system according to claim 1, characterized in that: The data acquisition module acquires the text of the service element to be measured in the interlocking man-machine interface, specifically including, initializing the interlocking man-machine interface; controlling the interlocking man-machine interface to sequentially display the images of each service element; sequentially taking screenshots of the images of each service element; extracting text from the screenshots.

4. The detection system of an interlocking human-machine interface operation display system according to claim 1, characterized in that: The image recognition module compares the image to be measured with the standard image by using two methods: image template matching and image feature matching, and calculates the matching degree between the image to be measured and the standard image.

5. The detection system of an interlocking human-machine interface operation display system according to claim 1, characterized in that: Set the similarity thresholds for images, texts, and voices respectively according to the detection requirements. If the similarity is greater than the threshold, it is determined that the detection is qualified.

6. The detection system of an interlocking human-machine interface operation display system according to claim 1, characterized in that: The service elements include buttons, signal machines, turnouts, routes, indicator lights, signs, and warning prompts.

7. A detection method for an interlocking human-machine interface operation display system, implemented in the detection system according to any one of claims 1-6, characterized in that: The detection method is implemented on the basis of a detection system, including the following steps, S1. Construct a standard database, including standard images, standard texts, and standard speech texts; S2. Initialize the interlocking man-machine interface, control the interlocking man-machine interface to sequentially display the images, texts, and voices of each service element, obtaining the images to be measured, the texts to be measured, and the voices to be measured; S3. Compare the images to be measured, the texts to be measured, and the voices to be measured with the standard images, standard texts, and standard speech texts respectively to obtain similarities; S4. Set similarity thresholds for images, texts, and voices respectively. When the similarity obtained by comparison is higher than the threshold, it is determined that the detection is qualified; S5. Mouse operation test, detecting mouse operation instructions, including left mouse button click, right mouse button click, mouse movement, mouse scroll, and mouse drag; S6. Output a detection report after the detection is completed.

8. The detection method of an interlocking human-machine interface operation display system according to claim 7, characterized in that: In the step S2, image screenshots of each service element are obtained by taking screenshots, and after processing the image screenshots, a to-be-tested image is obtained; text is extracted from the image screenshots to obtain to-be-tested text.

9. The detection method of an interlocking human-machine interface operation display system according to claim 7, characterized in that: In the step S3, the method for comparing the to-be-tested image with the standard image adopts image template matching and image feature matching, and the average value after adding the two matching values is used as the similarity value.

10. The detection method of an interlocking human-machine interface operation display system according to claim 7, characterized in that: In the step S3, when comparing the to-be-tested speech with the standard speech text, the to-be-tested speech is converted into text by SpeechSDK and then compared with the standard speech text.

Images