A dual-mode image acquisition and processing method and system for a nuclear instrument control human-machine interface

Abstract

The application belongs to the technical field of nuclear power digital instrument control system testing and automation, and relates to a nuclear instrument control man-machine interaction interface dual-mode image acquisition and processing method and system. First, spatial calibration is performed on the dual camera to obtain a mapping matrix. During the testing process, a high-frame-rate camera is used to capture pixel changes in real time, and time sequence analysis is used to determine the validity of the changes, extract time characteristics and position characteristics of the state changes. The mapping matrix is used to convert the coordinates of the changed area to a high-definition coordinate system, and the ROI is intercepted in the high-definition image. Real-time semantic recognition is performed on the ROI to determine the identity and content of the changed target icon and extract spatial semantic features. Finally, the time characteristics and spatial semantic characteristics are cooperatively fused to determine the response time and change results. The application does not need to pre-scan and construct a reference library, and realizes real-time and efficient automatic testing.

Description

Technical Field

[0001] This application belongs to the field of nuclear power digital instrumentation and control system testing and automation technology, and more specifically, relates to a dual-modal image acquisition and processing method and system for nuclear instrumentation and control human-machine interface. Background Technology

[0002] The human-machine interface of the digital instrumentation and control system in a nuclear power plant is a key window for operators to monitor and control the nuclear reactor. The accuracy of the screen and the response speed directly affect nuclear safety. Therefore, during the development and acceptance phase of the nuclear instrumentation and control system, the human-machine interface must be rigorously tested, including static screen display checks and dynamic response tests, such as changes in icon colors and indicator light flipping.

[0003] Currently, the testing of dynamic changes in the human-machine interface for nuclear instrument control has the following problems and shortcomings: Manual testing is inefficient and prone to errors. It relies on visual observation and manual operation by testers, which is labor-intensive, can easily lead to visual fatigue, resulting in missed detections or misjudgments, and the results are difficult to quantify. Existing automated testing processes are cumbersome, typically requiring the pre-collection of full-screen images and the identification of all graphic symbol positions to establish a benchmark library before guiding cameras to perform fixed-point monitoring. This approach requires a significant amount of preliminary work, and the process needs to be re-executed when the interface layout changes, severely impacting the continuity of testing. Furthermore, a single sensor cannot simultaneously capture both speed and detail; high-definition cameras with low frame rates are prone to missing transient changes, while high-frame-rate cameras with low resolution struggle to accurately identify graphic symbol details, such as text and color, leading to false alarms. Summary of the Invention

[0004] To overcome the problems of poor real-time performance and cumbersome preprocessing in existing technologies, this invention provides a dual-modal image acquisition and processing method and system for nuclear instrument control human-machine interface.

[0005] On one hand, the present invention provides a dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface, comprising the following steps: S1: The field of view of two cameras covers the same display area of ​​the human-machine interface of the nuclear instrument control system. The two cameras are jointly calibrated to obtain the spatial mapping matrix between the coordinate systems of the two cameras. The two cameras are a high frame rate camera and a high-definition camera, respectively. S2: After starting the test, the high frame rate camera is used to collect the video stream of the nuclear instrument control human-machine interface in real time. By calculating the difference between two adjacent frames, the pixel change area in the image is detected. S3: Perform time-series tracking analysis on the detected pixel mutation region. When the duration of the mutation state in the pixel mutation region exceeds a preset time-series threshold, it is determined to be a valid change event. At the same time, record the time characteristics of the occurrence of the valid change event and the center position coordinates of the pixel mutation region. S4: Using the acquired spatial mapping matrix, the center position coordinates are transformed to the image coordinate system of the high-definition camera, and the high-definition camera is controlled to capture the current nuclear instrument control human-machine interface screen, and the region of interest corresponding to the center position coordinates is captured in the captured screen. S5: Perform real-time semantic analysis on the captured region of interest to identify the target identity and displayed content of the symbols within the region, forming spatial semantic features; S6: Perform collaborative matching between the temporal features of the effective change event and the extracted spatial semantic features to determine whether the symbol has undergone the expected state change at the time point corresponding to the recorded temporal features.

[0006] This invention employs a dual-modal collaborative acquisition architecture using a high-frame-rate camera and a high-definition camera. The two cameras are jointly calibrated to obtain a spatial mapping matrix. The high-frame-rate camera captures the interface video stream in real time, and pixel abrupt change regions are detected by calculating inter-frame differences. Valid change events are then selected through time-series tracking, and time and location information are recorded. Next, coordinate transformation using the spatial mapping matrix triggers the high-definition camera to capture images only from the abrupt change regions and extract the region of interest. After semantic analysis to extract the spatial semantic features of the region's icon identity and displayed content, time features are combined with collaborative matching to determine whether the icon state change is compliant. This replaces traditional manual visual observation and testing. This method avoids the drawbacks of manual testing, such as low efficiency, high labor intensity, visual fatigue leading to missed detections and misjudgments, and difficulty in quantifying results. It also eliminates the need for pre-collecting full-screen images, establishing a comprehensive icon benchmark library, and pre-setting monitoring points, saving tedious pre-processing work. When the interface layout changes, there is no need to redeploy the entire system, ensuring the continuity of testing. Furthermore, the dual-camera system complements each other, with a high-frame-rate camera capturing transient changes to avoid missing short-term interface dynamics, and a high-definition camera focusing on local areas of interest to supplement detail recognition capabilities. This solves the technical challenges of a single sensor not being able to balance acquisition speed and image detail, and being prone to transient missed detections and false alarms in icon and text color recognition.

[0007] Preferably, the high frame rate camera and the high-definition camera are connected via a GPIO hardware trigger interface to synchronously acquire commands with a microsecond-level delay.

[0008] Preferably, the steps for obtaining the spatial mapping matrix are as follows: A standard calibration board is used to jointly calibrate a high frame rate camera and a high-definition camera. The homography matrix between the image coordinate systems of the two cameras is calculated by feature point matching as a spatial mapping matrix. The spatial mapping matrix is ​​a non-singular matrix with 8 degrees of freedom, which is obtained by solving at least 4 pairs of non-collinear corresponding feature points.

[0009] Preferably, detecting pixel abrupt change regions in the image includes the following steps: The pixel difference between the current image and the previous image is calculated in real time to generate an absolute difference matrix; a preset pixel threshold is set, and the absolute difference matrix is ​​binarized according to the pixel threshold to generate a binarized mask image. The regions with a value of 1 in the binarized mask image are the pixel abrupt change regions; wherein the pixel threshold is set according to the image noise level.

[0010] Preferably, the time-series tracking analysis includes the following steps: Define a time-series cumulative matrix, update the time-series cumulative matrix in real time, and count the number of frames in which the same pixel position shows a continuous difference; When the value of a connected region in the time-series cumulative matrix exceeds a preset time-series threshold, it is determined to be a valid change event; at the same time, by utilizing the visual persistence characteristic of the state changes of the icons in the nuclear instrumentation human-machine interface, it is ensured that when the high-definition camera captures the image, the image of the nuclear instrumentation human-machine interface is still in a stable state after the change.

[0011] Preferably, the step of extracting the region of interest includes: Based on the classic dimensions of symbols in the design specifications for the human-machine interface of nuclear instrumentation control, a fixed-size region of interest is extracted; The region of interest (ROI) of a fixed size is dynamically adjusted in the current frame based on the bounding box of the symbol in the recognition result of the previous frame and the target motion prediction.

[0012] Preferably, the real-time semantic analysis includes the following steps: First, the region of interest (ROI) is preprocessed. Then, an optical character recognition (OCR) algorithm is used to extract the device label and status text information within the ROI. Simultaneously, the current display content of the symbol is identified to determine its real-time presentation status. Next, a shape feature matching algorithm is used to identify the symbol type within the ROI. Finally, the OCR results and shape feature matching results are semantically fused to comprehensively determine the target identity of the changed symbol and its current display content.

[0013] Preferably, the collaborative matching includes the following steps: The high frame rate camera is responsible for providing the temporal characteristics of effective change events, which are used for testing the response time of the nuclear instrument control human-machine interface. The high-definition camera is responsible for providing target identity confirmation and status content features after the icon changes, which is used to verify the functional correctness of the nuclear instrument control human-machine interface. After collaborative matching is completed, if the comparison is consistent, the temporal features and spatial semantic features are fused to generate a complete test record.

[0014] Preferably, by using the low-latency task scheduling of the real-time operating system, the total processing latency of effective change determination, coordinate mapping and instruction transmission is controlled within 20 milliseconds. By utilizing the visual persistence characteristic of the changes in the state of the symbols in the human-machine interface of the nuclear instrument control, the state of the symbol after the change is maintained for more than 100 milliseconds, ensuring that the symbol is still in a stable state after the change when the high-definition camera is collecting data.

[0015] Another invention provides a dual-modal image acquisition and processing system for a nuclear instrumentation control human-machine interface, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the dual-modal image acquisition and processing method for a nuclear instrumentation control human-machine interface described in this invention.

[0016] The beneficial effects of this invention include: This invention employs a dual-modal collaborative acquisition architecture using a high-frame-rate camera and a high-definition camera. The two cameras are jointly calibrated to obtain a spatial mapping matrix. The high-frame-rate camera captures the interface video stream in real time, and pixel abrupt change regions are detected by calculating inter-frame differences. Valid change events are then selected through time-series tracking, and time and location information are recorded. Next, coordinate transformation using the spatial mapping matrix triggers the high-definition camera to capture images only from the abrupt change regions and extract the region of interest. After semantic analysis to extract the spatial semantic features of the region's icon identity and displayed content, time features are combined with collaborative matching to determine whether the icon state change is compliant. This replaces traditional manual visual observation and testing. This method avoids the drawbacks of manual testing, such as low efficiency, high labor intensity, visual fatigue leading to missed detections and misjudgments, and difficulty in quantifying results. It also eliminates the need for pre-collecting full-screen images, establishing a comprehensive icon benchmark library, and pre-setting monitoring points, saving tedious pre-processing work. When the interface layout changes, there is no need to redeploy the entire system, ensuring the continuity of testing. Furthermore, the dual-camera system complements each other, with a high-frame-rate camera capturing transient changes to avoid missing short-term interface dynamics, and a high-definition camera focusing on local areas of interest to supplement detail recognition capabilities. This solves the technical challenges of a single sensor not being able to balance acquisition speed and image detail, and being prone to transient missed detections and false alarms in icon and text color recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 The overall flowchart provided for embodiments of the present invention.

[0019] Figure 2 This is a coordinate mapping diagram provided for an embodiment of the present invention. Detailed Implementation

[0020] To make the technical problems, solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] Example 1 This embodiment provides a dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface. By using a high frame rate camera and a high-definition camera working together, it achieves rapid capture, accurate positioning, and real-time recognition of dynamic changes. The specific steps are as follows: S1: Build a dual-modal vision acquisition platform. Install a high-frame-rate camera and a high-definition camera directly in front of the human-machine interface of the instrumentation control system. Adjust the angle and focal length of the two cameras to ensure that their fields of view completely cover the same display area of ​​the human-machine interface of the instrumentation control system, and the overlap rate of the fields of view is not less than 95%, to avoid blind spots in the acquisition. Among them, the high-frame-rate camera is an industrial camera with a frame rate of not less than 200fps and a resolution of 1280×720, which can meet the real-time capture of transient changes. The high-definition camera is an industrial camera with a resolution of not less than 1920×1080 and a frame rate of not less than 30fps.

[0022] Secondly, in this embodiment, a standard calibration board (e.g., a checkerboard calibration board with a checkerboard size of 10mm×10mm and a total of 10×8 squares) is used to jointly calibrate the two cameras. The calibration board is placed at different positions and angles on the nuclear instrument control human-machine interface, and at least 10 images of the calibration board are captured by the high frame rate camera and the high-definition camera respectively, to ensure that the calibration board is clearly visible in the field of view of both cameras and covers all corners of the field of view.

[0023] The feature points in each calibration board image are then extracted using a feature point matching algorithm. At least four pairs of non-collinear corresponding feature points are selected, and the homography matrix between the coordinate systems of the two camera images is calculated as the spatial mapping matrix. In this embodiment, the feature matching algorithm is the SIFT feature point matching algorithm.

[0024] In this embodiment, the spatial mapping matrix is ​​a 3×3 nonsingular matrix containing 8 degrees of freedom. Its solution process is based on the principle of projective transformation, describing the one-to-one correspondence between the high frame rate camera image coordinate system and the high-definition camera image coordinate system. The coordinate transformation formula is as follows: ; In the formula: Indicates a non-zero scaling factor; This represents the coordinates of the corresponding point in the high-definition camera image coordinate system, in homogeneous coordinate format. ,in These are two-dimensional coordinates in the high-definition camera image coordinate system. Represents the spatial mapping matrix; This represents the coordinates of any point in the high frame rate camera image coordinate system, in homogeneous coordinate format. ,in These are two-dimensional coordinates in the high frame rate camera image coordinate system.

[0025] In this embodiment, non-collinear feature points can ensure the uniqueness and accuracy of the mapping matrix, avoiding matrix solution failure caused by collinear feature points.

[0026] Finally, the obtained spatial mapping matrix is ​​stored in the system memory for mapping and transformation of coordinates in the changing region in subsequent steps, ensuring the real-time performance and accuracy of coordinate transformation.

[0027] S2: By rapidly capturing the dynamic changes of the nuclear instrumentation control human-machine interface, pixel abrupt change regions where icon states may change are identified. In this embodiment, the dynamic changes of the nuclear instrumentation control human-machine interface occur instantaneously. If a high-definition camera is used for acquisition, its low frame rate will lead to missed transient changes. However, the high sampling rate of a high frame rate camera can accurately capture each instantaneous change. Then, through inter-frame differential detection, abrupt change regions can be quickly located, providing a basis for subsequent effective change determination. The specific steps are as follows: After starting the test of the nuclear instrument control human-machine interface, the high frame rate camera is controlled to continuously collect the video stream of the human-machine interface at a sampling rate of 200fps, ensuring the acquisition speed while reducing the amount of data processing.

[0028] In this embodiment, the video stream captured by the high frame rate camera is read in real time, and the pixel difference between the current image and the previous image is calculated frame by frame to generate an absolute difference matrix. Each pixel in the absolute difference matrix corresponds to the difference in grayscale value at the same pixel position in the two frames. The calculation formula is as follows: ; In the formula: The element in the i-th row and j-th column of the absolute difference matrix represents the difference in grayscale values ​​of the pixels in the i-th row and j-th column between two frames of images. This represents the grayscale value of the pixel in the i-th row and j-th column in frame t at the current time. This represents the grayscale value of the pixel in the i-th row and j-th column in the previous frame t-1.

[0029] Furthermore, since there is slight noise during the image acquisition process, it can cause false pixel differences in the difference matrix. If not filtered, it can lead to subsequent misjudgments. Therefore, a preset pixel threshold is set according to the image noise level. For example, the pixel threshold is set to 30. When the pixel difference is greater than 30, it is judged as a real change; when the pixel difference is less than or equal to 30, it is judged as noise interference.

[0030] In this embodiment, the absolute difference matrix is ​​binarized based on the aforementioned pixel threshold to generate a binarized mask image. The binarization formula is as follows: ; In the formula: This represents the element in the i-th row and j-th column of the binarized mask image, used to mark whether a sudden change occurs at the corresponding pixel position; This represents the difference in pixel grayscale values ​​in the i-th row and j-th column of the absolute difference matrix; This indicates the preset pixel threshold.

[0031] In this embodiment, the area with a value of 1 in the binarized mask image is the pixel abrupt change area, which corresponds to the area in the human-machine interface of the nuclear instrument control that may have a change in the state of the icon.

[0032] S3: First, define a time-series cumulative matrix. The size of the time-series cumulative matrix is ​​the same as the size of the image captured by the high frame rate camera. It is used to count the number of frames where the same pixel position shows a continuous difference. Its update formula is as follows: ; In the formula: This represents the value in the i-th row and j-th column of the time-series cumulative matrix in the current frame; In this embodiment, when a certain pixel position hour, Add 1 to the value of the previous frame; when hour, Resetting to zero allows us to track the duration of each pixel's mutation region through this update method.

[0033] As one possible implementation of this embodiment, a timing threshold is preset in this embodiment. The timing threshold is set according to the change speed of the icons in the human-machine interface of the instrument control system and the system noise level. In this embodiment, the timing threshold is set to 3 frames. Combined with the sampling rate of 200fps of the high frame rate camera, the time corresponding to 3 frames is 15ms. This time point can filter instantaneous noise without missing real transient changes.

[0034] When the value of a connected region in the time-series cumulative matrix exceeds a preset time-series threshold, a valid symbol state change, i.e., a valid change event, is determined to have occurred in that region. In this embodiment, the connected region refers to a region composed of adjacent pixels with a value of 1. The complete symbol change region can be determined through connected region analysis. At the same time, in this embodiment, the system will immediately record the current moment as the time feature of the valid change event. This time feature is used for subsequent response time testing of the nuclear instrument control human-machine interface. Through image geometric calculation, the geometric centroid coordinates of the valid change connected region are solved. These coordinates are the precise location feature of the symbol state change, and this location feature will be used for the next coordinate mapping transformation.

[0035] In this embodiment, the visual persistence characteristic of the changes in the state of the symbols in the nuclear instrument control human-machine interface is utilized to ensure subsequent high-definition camera acquisition. The design of the symbols in the nuclear instrument control human-machine interface must meet the visual recognition needs of the operator. After the state changes, the symbols must remain in a stable state for a certain period of time. Therefore, even if there is a slight delay in the high-definition camera, it can be ensured that the symbols are still in a stable state after the change during acquisition, avoiding recognition errors caused by the symbol state changing again.

[0036] S4: Accurately map the changing areas captured by the high frame rate camera onto the field of view of the high-definition camera. The high-definition camera then acquires detailed images of this area and extracts the ROI (Region of Interest) for subsequent semantic analysis. Specifically: The spatial mapping matrix obtained in step S1 is retrieved from the memory. The geometric centroid coordinates of the effective change region obtained in step S3 are substituted into the coordinate transformation formula to calculate the mapped coordinates of the point in the high-definition camera image coordinate system. These mapped coordinates are the focal positions that the high-definition camera needs to focus on acquiring.

[0037] Furthermore, in this embodiment, to ensure that the images captured by the high-definition camera are consistent with the changing states captured by the high frame rate camera, this embodiment achieves spatiotemporal synchronization through hardware triggering and low-latency scheduling. That is, the high frame rate camera and the high-definition camera are connected through the GPIO hardware trigger interface. The GPIO interface can realize the synchronous acquisition command transmission with a microsecond-level latency, avoiding acquisition deviation caused by command transmission delay. Simultaneously, a real-time operating system (such as RT-Rhread) is used for low-latency task scheduling, strictly controlling the total processing latency of effective change judgment, coordinate mapping and command transmission to within 20ms.

[0038] In this embodiment, the visual persistence characteristic of the symbols in the nuclear instrument control human-machine interface (the duration of the changed state is greater than 100ms) is combined. Since the total processing delay (20ms) is much smaller than the duration of the symbol state, it is physically ensured that the image captured by the high-definition camera after the acquisition command is still in the stable state after the symbol change, thus avoiding acquisition failure or recognition error caused by delay.

[0039] In this embodiment, after the high-definition camera captures the current image of the nuclear instrument control human-machine interface, the corresponding ROI is captured centered on the mapped coordinates. There are two capture methods, which can be selected according to the actual test scenario: The first method: Based on the classic dimensions of the symbols in the design specifications for the human-machine interface of nuclear instrument control, a fixed-size ROI is extracted. In this embodiment, the size of the fixed ROI is set to 200×200 pixels.

[0040] The second method utilizes the bounding box of the icon in the previous frame's recognition result, combined with target motion prediction, to dynamically adjust the ROI size in the current frame. If the icon has slight motion (such as icon position shift caused by interface scrolling), the bounding box position of the icon in the current frame is predicted using a target motion prediction algorithm (such as Kalman filtering algorithm), thereby adjusting the size and position of the ROI to ensure that the ROI always completely covers the changed icon area, improving the accuracy of recognition.

[0041] S5: First, perform image preprocessing on the extracted ROI to remove image noise and enhance symbol details, laying the foundation for subsequent recognition; the specific steps are as follows: The color ROI image is converted to a grayscale image to reduce interference from the color channels. Then, an adaptive threshold binarization algorithm is used to convert the grayscale image into a black and white binary image to highlight the outline of the symbol and the text. Finally, a Gaussian filtering algorithm is used to remove Gaussian noise from the image to avoid recognition errors caused by noise.

[0042] Secondly, optical character recognition algorithms (such as OCR algorithms, with Tesseract OCR algorithm selected) are used to extract information such as equipment labels and status text within the ROI. For example, text information such as reactor temperature indicator lights, normal, and alarm is identified to clarify the associated equipment and current text status of the symbols.

[0043] At the same time, the current display content of the symbol is identified to clarify the real-time presentation status of the symbol. For example, for indicator light symbols, its current color (red, green, yellow) is identified; for button symbols, its current state (pressed, released) is identified; and for numerical display symbols, its current numerical information is identified. The above display content are the core features of the symbol state change.

[0044] Then, a shape feature matching algorithm is used to identify the type of symbols within the ROI. Since the symbols in the nuclear instrument control human-machine interface have fixed shape features (such as indicator lights being round, buttons being square, and alarm icons being triangles), a shape context algorithm is selected to extract the shape features of the symbols within the ROI and match them with preset symbol shape templates (such as graphic templates and square templates) to determine the specific type of the symbol (such as round indicator lights and square start buttons).

[0045] Finally, the optical character recognition results and shape feature matching results are semantically fused to comprehensively determine the identity of the changed symbol target and the current display content of the symbol. For example, if OCR recognizes a reactor temperature indicator light, shape matching recognizes a circular indicator light, and the display content is recognized as red, then the spatial semantic feature formed after fusion is: reactor temperature indicator light, current state is red. This spatial semantic feature will be used for subsequent collaborative matching and test judgment.

[0046] In this embodiment, there is no need to build a full-screen baseline map. The target icon can be located by targeted semantic analysis of the ROI of the changed area, which solves the pain point of having to rebuild the library when the cross-sectional layout changes in existing automated testing.

[0047] S6: The test results are determined by synergistically fusing temporal and spatial semantic features. Specifically: The high frame rate camera is responsible for providing the time characteristics of the effective change event (i.e., the change time recorded in step 3). This time characteristic is used for the response time test of the nuclear instrument control human-machine interface. The time characteristic is compared with the preset response time threshold in the test case to determine whether the response speed of the icon meets the requirements. For example, if the preset response time threshold is 50ms, the response time is qualified if the difference between the change time and the command trigger time is less than 50ms.

[0048] The high-definition camera is responsible for providing target identity confirmation and state content features after the icon changes (i.e., the spatial semantic features formed in step S5). The spatial semantic features are used to verify the functional correctness of the human-machine interface of the instrument control system. The spatial semantic features are compared with the expected results in the test cases to determine whether the icon has undergone the expected state change.

[0049] In this embodiment, temporal features and spatial semantic features are matched collaboratively. The specific matching steps are as follows: Confirm whether the change time corresponding to the time feature is within the preset instruction trigger time range in the test case; Confirm the identity of the symbols in the spatial semantic features and whether they are the preset test symbols in the test cases; Confirm whether the current display content of the symbols in the spatial semantic features is consistent with the expected state preset in the test case.

[0050] If all three of the above matches, the icon in the nuclear instrument control human-machine interface undergoes the expected state change at the preset time point, and the test is qualified. The system integrates the time features and spatial semantic features to generate a complete test record. The test record includes: change time, identity of the icon under test, icon type, state before change, state after change, response time, and test result (qualified).

[0051] If any of the above three matching items are inconsistent, the test is considered unqualified. The system will record the abnormal information and trigger an alarm to help testers troubleshoot the problem in a timely manner.

[0052] Example 2 This embodiment provides a dual-modal image acquisition system for a nuclear instrument control human-machine interface, used to implement the method described in Embodiment 1 above. The system includes a memory, a processor, and a computer program stored on the processor and capable of running on the processor. The processor calls the computer program to run the method described in Embodiment 1. As one possible implementation of this embodiment, the memory is a large-capacity solid-state drive with a capacity of not less than 1TB, used to store information such as space mapping matrix, acquired video streams, image data, test cases, and test records. It also supports high-speed read and write to ensure the real-time performance of data transmission and avoid test lag caused by storage delay.

[0053] As one possible implementation of this embodiment, the processor adopts an industrial-grade high-performance GPU (such as an Intel Core i7-12700H) with a main frequency of not less than 2.7 Hz and a core count of not less than 14. It also integrates a GPU (such as an NVIDIA RTX 3060) to accelerate computationally intensive tasks such as image processing, feature extraction, and semantic analysis, ensuring that the system can process video streams captured by high frame rate cameras in real time and complete operations such as effective change determination, coordinate mapping, and instruction transmission within 20 ms.

[0054] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface, characterized in that, Includes the following steps: S1: The field of view of two cameras covers the same display area of ​​the human-machine interface of the nuclear instrument control system. The two cameras are jointly calibrated to obtain the spatial mapping matrix between the coordinate systems of the two cameras. The two cameras are a high frame rate camera and a high-definition camera, respectively. S2: After starting the test, the high frame rate camera is used to collect the video stream of the nuclear instrument control human-machine interface in real time. By calculating the difference between two adjacent frames, the pixel change area in the image is detected. S3: Perform time-series tracking analysis on the detected pixel mutation region. When the duration of the mutation state in the pixel mutation region exceeds a preset time-series threshold, it is determined to be a valid change event. At the same time, record the time characteristics of the occurrence of the valid change event and the center position coordinates of the pixel mutation region. S4: Using the acquired spatial mapping matrix, the center position coordinates are transformed to the image coordinate system of the high-definition camera, and the high-definition camera is controlled to capture the current nuclear instrument control human-machine interface screen, and the region of interest corresponding to the center position coordinates is captured in the captured screen. S5: Perform real-time semantic analysis on the captured region of interest to identify the target identity and displayed content of the symbols within the region, forming spatial semantic features; S6: Perform collaborative matching between the temporal features of the effective change event and the extracted spatial semantic features to determine whether the symbol has undergone the expected state change at the time point corresponding to the recorded temporal features.

2. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The high frame rate camera and the high-definition camera are connected via a GPIO hardware trigger interface to synchronously acquire commands with a microsecond-level delay.

3. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The steps for obtaining the spatial mapping matrix are as follows: A standard calibration board is used to jointly calibrate a high frame rate camera and a high-definition camera. The homography matrix between the image coordinate systems of the two cameras is calculated by feature point matching as a spatial mapping matrix. The spatial mapping matrix is ​​a non-singular matrix with 8 degrees of freedom, which is obtained by solving at least 4 pairs of non-collinear corresponding feature points.

4. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The detection of pixel abrupt change regions in the image includes the following steps: The pixel difference between the current image and the previous image is calculated in real time to generate an absolute difference matrix; a preset pixel threshold is set, and the absolute difference matrix is ​​binarized according to the pixel threshold to generate a binarized mask image. The regions with a value of 1 in the binarized mask image are the pixel abrupt change regions; wherein the pixel threshold is set according to the image noise level.

5. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The time-series tracking analysis includes the following steps: Define a time-series cumulative matrix, update the time-series cumulative matrix in real time, and count the number of frames in which the same pixel position shows a continuous difference; When the value of a connected region in the time-series cumulative matrix exceeds a preset time-series threshold, it is determined to be a valid change event; at the same time, by utilizing the visual persistence characteristic of the state changes of the symbols in the nuclear instrumentation human-machine interface, it is ensured that when the high-definition camera captures the image, the image of the nuclear instrumentation human-machine interface is still in a stable state after the change.

6. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The steps for extracting the region of interest include: Based on the classic dimensions of symbols in the design specifications for the human-machine interface of nuclear instrumentation control, a fixed-size region of interest is extracted; The region of interest (ROI) of a fixed size is dynamically adjusted in the current frame based on the bounding box of the symbol in the recognition result of the previous frame and the target motion prediction.

7. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The real-time semantic analysis includes the following steps: First, the region of interest (ROI) is preprocessed. Then, an optical character recognition (OCR) algorithm is used to extract the device label and status text information within the ROI. Simultaneously, the current display content of the symbol is identified to determine its real-time presentation status. Next, a shape feature matching algorithm is used to identify the symbol type within the ROI. Finally, the OCR results and shape feature matching results are semantically fused to comprehensively determine the target identity of the changed symbol and its current display content.

8. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, The collaborative matching includes the following steps: The high frame rate camera is responsible for providing the temporal characteristics of effective change events, which are used for testing the response time of the nuclear instrument control human-machine interface. The high-definition camera is responsible for providing target identity confirmation and status content features after the icon changes, which is used to verify the functional correctness of the nuclear instrument control human-machine interface. After collaborative matching is completed, if the comparison is consistent, the temporal features and spatial semantic features are fused to generate a complete test record.

9. The dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface according to claim 1, characterized in that, By utilizing the low-latency task scheduling of the real-time operating system, the total processing latency of effective change determination, coordinate mapping, and instruction transmission is controlled within 20 milliseconds. By leveraging the visual persistence characteristic of the icon state changes in the nuclear instrument control human-machine interface, the state of the icon after the change is maintained for more than 100 milliseconds, ensuring that the icon remains in a stable state after the change when the high-definition camera is capturing data.

10. A dual-modal image acquisition and processing system for nuclear instrument control human-machine interface, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a dual-modal image acquisition and processing method for a nuclear instrument control human-machine interface as described in any one of claims 1 to 9.

Images