Special inspection equipment software starting state identification and information automatic input method and system

Abstract

The embodiment of the invention provides a special inspection equipment software starting state recognition and information automatic input method and system, and the method comprises the steps: obtaining a software interface screenshot of special inspection equipment software in a historical starting state, and constructing a relative position file and a standardized image library according to the screenshot; inputting a real-time desktop image of the special inspection equipment software into the image segmentation model, and processing to obtain a target mask with the maximum area; calculating the comprehensive similarity of the target mask relative to each image in the standardized image library, and when the maximum value of the comprehensive similarity is determined to be greater than a preset threshold value, determining that the special inspection equipment software is in a real-time starting state, otherwise, processing the next frame of image; converting the interactive relative coordinate points obtained from the relative position file into interactive absolute coordinate points by taking the target mask bounding box as a reference; and constructing an information input identification area based on each interactive absolute coordinate point, and carrying out OCR identification, option positioning and selection logic, text analysis and field mapping in the information input identification area.

Description

Technical Field

[0001] This application relates to the field of special equipment inspection and automation technology, and more specifically, to a method and system for identifying the software startup status and automatically recording information of special inspection equipment. Background Technology

[0002] When integrating data from medical special equipment with hospital information systems, existing technologies primarily rely on automated script solutions for "one-click entry" to achieve automatic patient information input. This solution requires pre-configuration of the graphical user interface (GUI) for each type of special equipment software, forming a configuration scheme. Currently, similar technologies used in implementing this function include manual identification and matching of configuration schemes, as well as keyword-based knowledge base retrieval. While these technologies can achieve basic data capture and input in scenarios where the equipment GUI layout is stable and field labels are standardized, they also have significant drawbacks, such as: 1) Heavy reliance on human experience: Implementation engineers need to manually analyze the equipment GUI structure and locate the control attributes of target fields one by one. New engineers need to determine configuration rules through repeated trial and error, which is time-consuming and prone to errors. 2) When equipment software upgrades or interface resolution adjustments cause window position shifts, the original configuration scheme based on fixed coordinates or simple text matching will become invalid, requiring manual repositioning and script updates, resulting in high maintenance costs.

[0003] In summary, to address the shortcomings of existing technologies that rely heavily on static GUI configuration, it is necessary to introduce computer vision-based intelligent configuration technology to solve the problem of inaccurate information input caused by changes in window position. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for identifying the startup status of special inspection equipment software and automatically entering information, in order to overcome the shortcomings of the prior art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A method for identifying the startup status of special inspection equipment software and automatically entering information, comprising the following steps:

[0006] S1. Obtain screenshots of the software interface of the special inspection equipment in its historical startup state. Based on the screenshots, perform interactive relative coordinate CSV conversion and image standardization storage processing to obtain a relative position file and a standardized image library.

[0007] S2. Obtain the real-time desktop image of the special inspection equipment software, input the real-time desktop image into the image segmentation model, and obtain the target mask with the largest area through automatic segmentation, mask generation and area filtering.

[0008] S3. Calculate the comprehensive similarity of the target mask relative to each image in the standardized image library. If the maximum value of the comprehensive similarity is greater than the preset threshold, determine that the special inspection equipment software is in real-time startup state and execute step S4. Otherwise, return to step S2 to process the next frame image.

[0009] S4. Using the target mask bounding box as a reference, convert the interactive relative coordinates obtained from the relative position file into interactive absolute coordinates.

[0010] S5. Construct an information input and recognition area based on the absolute coordinates of each interaction, and perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information input and recognition area to achieve automatic information input.

[0011] Furthermore, in step S1, the step of converting the interactive relative coordinates into a CSV file based on the screenshot of the software interface to obtain a relative position file includes:

[0012] (1) Select the reference point and the key interaction point for information entry from the screenshot of the software interface;

[0013] (2) Based on the reference point, perform relative coordinate transformation on each key interaction point to form a set of relative coordinate points;

[0014] (3) Store the set of relative coordinate points into a CSV file to obtain a relative position file.

[0015] Furthermore, in step S1, the image standardization and storage processing based on the software interface screenshot to obtain a standardized image library includes:

[0016] (1) Convert the obtained screenshots of each software interface into a standard image storage format to obtain a standard format image;

[0017] (2) Scale each standard format image to the preset image size. During the scaling process, use edge filling to perform center alignment to obtain the corresponding normalized image.

[0018] (3) Integrate, store and manage the normalized images to obtain a standardized image library.

[0019] Furthermore, in step S3, the comprehensive similarity of the target mask relative to each image in the standardized image library is calculated through the following steps:

[0020] S31. Based on the histogram bus distance metric, determine the color histogram similarity between the standardized image and the target mask;

[0021] S32. Based on the similarity of image feature elements, determine the structural similarity between the standardized image and the target mask;

[0022] S33. Based on the color histogram similarity and the structural similarity, a weighted fusion is performed to obtain the comprehensive similarity of the target mask relative to the standardized image.

[0023] Furthermore, in step S33, the weighted fusion based on the color histogram similarity and the structural similarity includes: combining color histogram similarity, structural similarity, histogram normalized entropy, saturation gate value, and gradient gate value, adaptively calculating the dynamic weights of color and structure through multi-parameter coordination with exponential and temperature control, and performing weighted fusion of the color histogram similarity and the structural similarity based on the dynamic weights.

[0024] Furthermore, the dynamic weights for color and structure are adaptively calculated using the following formula:

[0025] ;

[0026] in, and These represent the weights of color and structure, respectively. Represents the temperature constant. Indicates the similarity of color histograms. Indicates structural similarity. This represents the normalized entropy of the histogram. Indicates the saturation threshold value. This represents the gradient gating value.

[0027] Furthermore, in step S5, constructing the information input and recognition area based on each interaction absolute coordinate point includes:

[0028] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point;

[0029] (2) Select the leftmost horizontal point from the set of row points and use it as the reference point;

[0030] (3) Using the reference point as the starting reference point for constructing the rectangular area, the rectangular area coordinates are calculated and constructed by mathematical operations based on coordinate offset according to the predefined rectangular area width and height and horizontal interval, forming an information input recognition area for locating interactive elements to achieve automated information input.

[0031] Furthermore, in step S5, constructing the information input and recognition area based on each interaction absolute coordinate point includes:

[0032] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point;

[0033] (2) Based on the set of row points, when the number of points in the set is greater than a preset threshold, a rectangular area for identifying the content of the option is constructed to the right of any point in the same row.

[0034] Secondly, this application discloses a special inspection equipment software startup status recognition and automatic information entry system. The system includes a historical data preprocessing module, a real-time desktop image segmentation processing module, a special inspection equipment software startup status recognition module, an interactive absolute coordinate point conversion module, and an automatic information entry module, wherein:

[0035] The historical data preprocessing module is used to acquire screenshots of the software interface of the special inspection equipment software in the historical startup state, and perform interactive relative coordinate CSV conversion and image standardization storage processing based on the software interface screenshots to obtain relative position files and standardized image libraries.

[0036] The real-time desktop image segmentation processing module is used to acquire the real-time desktop image of the special inspection equipment software, input the real-time desktop image into the image segmentation model, and obtain the target mask with the largest area through automatic segmentation, mask generation and area filtering processing.

[0037] The special inspection equipment software startup status recognition module is used to calculate the comprehensive similarity of the target mask relative to each image in the standardized image library. When the maximum value of the comprehensive similarity is greater than a preset threshold, the special inspection equipment software is determined to be in real-time startup state, and the interactive absolute coordinate point conversion module is started. Otherwise, the real-time desktop image segmentation processing module is started to process the next frame image.

[0038] The interactive absolute coordinate point conversion module is used to convert interactive relative coordinate points obtained from the relative position file into interactive absolute coordinate points based on the target mask bounding box.

[0039] The automatic information entry module is used to construct an information entry and recognition area based on each interaction absolute coordinate point, and to perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information entry and recognition area to achieve automatic information entry.

[0040] Thirdly, this application discloses a readable storage medium, which includes a method program for identifying the startup status of special inspection equipment software and automatically recording information. When the method program for identifying the startup status of special inspection equipment software and automatically recording information is executed by a processor, it implements the steps of the method described in any of the preceding claims.

[0041] The beneficial effects of this invention are:

[0042] (1) By acquiring screenshots of the software interface in historical startup states, interactive relative coordinates are converted into CSV and images are standardized for storage, constructing a relative position file and a standardized image library. During real-time detection, the target mask with the largest area is obtained using an image segmentation model, and it is compared with the standardized image library to determine the software state. Then, based on the bounding box of the target mask, the interactive relative coordinate points in the relative position file are converted into interactive absolute coordinate points. This processing method ensures that no matter how the position of the software window changes on the actual desktop, the absolute position of the interactive element can be accurately determined based on the target mask, ensuring accurate positioning of subsequent operations;

[0043] (2) An information entry and recognition area is constructed based on the converted interactive absolute coordinates. Within this area, a series of automated operations, such as OCR recognition, option positioning and selection logic, text parsing and field mapping, are performed to achieve automatic information entry. The entire process is automated, avoiding positioning deviations caused by changes in window position during manual operation, reducing entry errors caused by human factors, and greatly improving the stability and accuracy of information entry. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a method for identifying the startup status of special inspection equipment software and automatically entering information, as disclosed in this invention.

[0045] Figure 2 This is a schematic diagram of the structure of a special inspection equipment software startup status recognition and automatic information entry system disclosed in this invention;

[0046] Figure 3 This is a schematic diagram of the structure of a readable storage medium disclosed in this invention. Detailed Implementation

[0047] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0048] like Figure 1 As shown, this application discloses a method for identifying the startup status of special inspection equipment software and automatically entering information, which includes the following steps:

[0049] Step S1: Obtain a screenshot of the software interface of the special inspection equipment in the historical startup state. Based on the screenshot of the software interface, perform interactive relative coordinate CSV conversion and image standardization storage processing to obtain a relative position file and a standardized image library.

[0050] Specifically, when performing interactive relative coordinate CSV conversion, this application associates the relative coordinates of interactive elements in the software interface with their corresponding element identifiers (such as the text content of buttons, the ID of input boxes, etc.) and writes them to a file in CSV format. Image standardization processing is specifically reflected in the unification of size and storage format to ensure the consistency and comparability of image data from different times and sources.

[0051] Step S2: Obtain the real-time desktop image of the special inspection equipment software, input the real-time desktop image into the image segmentation model, and obtain the target mask with the largest area after automatic segmentation, mask generation and area filtering.

[0052] Specifically, this application uses a large visual model (SAM) for image segmentation. The process begins by inputting the real-time desktop image of the special inspection equipment software into the SAM model. The model then quickly performs a global analysis of the image, identifying multiple objects and regions that may exist within it. Next, the SAM model uses its integrated segmentation algorithm to automatically generate an initial mask for each identified region. Then, by calculating and comparing the areas of these initial masks, the mask with the largest area is selected as the target mask.

[0053] Step S3: Calculate the overall similarity of the target mask relative to each image in the standardized image library. If the maximum value of the overall similarity is greater than a preset threshold, determine that the special inspection equipment software is in real-time startup state and execute step S4; otherwise, return to step S2 to process the next frame image.

[0054] Specifically, this application integrates color histogram similarity and structural similarity to perform multi-dimensional comprehensive matching between the segmentation results and the standardized image to determine whether the host computer software is open. Furthermore, to more accurately determine whether the host computer software is open, this application also employs a dynamic weighting method. Based on different scenarios and image characteristics, the weights of color histogram similarity and structural similarity in the comprehensive similarity calculation are automatically adjusted to improve the accuracy of determining the open status of the host computer software.

[0055] Step S4: Using the target mask bounding box as a reference, convert the interactive relative coordinates obtained from the relative position file into interactive absolute coordinates.

[0056] Specifically, this application uses the top-left corner of the target mask bounding box as the reference point. After obtaining the interactive relative coordinate points from the relative position file, it performs coordinate translation operations, that is, adds the horizontal and vertical coordinate values ​​corresponding to the top-left corner of the target mask bounding box to the interactive relative coordinates to complete the conversion of the interactive absolute coordinate points.

[0057] Step S5: Construct an information input and recognition area based on each interaction absolute coordinate point, and perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information input and recognition area to achieve automatic information input.

[0058] As can be seen from the above, the method for identifying the startup status of special inspection equipment software and automatically inputting information disclosed in this application has two aspects. First, by acquiring screenshots of the software interface in historical startup states, performing interactive relative coordinate CSV conversion and image standardization storage processing, a relative position file and a standardized image library are constructed. During real-time detection, the target mask with the largest area is obtained using an image segmentation model, and it is compared with the standardized image library to determine the software status. Then, using the target mask bounding box as a reference, the interactive relative coordinate points in the relative position file are converted into interactive absolute coordinate points. This processing method ensures that regardless of how the position of the software window changes on the actual desktop, the absolute position of the interactive element can be accurately determined based on the target mask, ensuring accurate positioning for subsequent operations. Second, an information input recognition area is constructed based on the converted interactive absolute coordinate points, and a series of automated operations such as OCR recognition, option positioning and selection logic, text parsing and field mapping are performed within this area to achieve automatic information input. The entire process is executed automatically, avoiding positioning deviations caused by changes in window position due to manual operation, reducing input errors caused by human factors, and greatly improving the stability and accuracy of information input.

[0059] In one embodiment, step S1, which involves converting the interactive relative coordinates into a CSV file based on the software interface screenshot to obtain a relative position file, includes:

[0060] (1) Select the reference point and the key interaction point for information entry from the screenshot of the software interface.

[0061] Specifically, this application will select a specific feature point in the software interface screenshot (which can be understood as a representative, easily identifiable and locatable feature point in the software interface), and align this point with the upper left corner of the software window interface. This point will be used as a reference point for subsequent coordinate calculations.

[0062] Furthermore, after determining the key locations (such as input boxes, buttons, drop-down boxes, etc.) in the software window interface where information needs to be entered, this application will select the center point or specific marker point of these key locations as the key interaction points for information entry.

[0063] (2) Based on the reference point, perform relative coordinate transformation on each key interaction point to form a set of relative coordinate points.

[0064] Specifically, for each key interaction point This application will also calculate its relative to a reference point. Offset: , All relative coordinates obtained Stored as a point set .

[0065] (3) Store the set of relative coordinate points into a CSV file to obtain a relative position file.

[0066] It should be noted that, considering the simple structure and fast parsing speed of CSV format, which can effectively improve data processing efficiency, this application stores the relative position file in CSV format to achieve efficient storage and fast reading of relative coordinate data, meeting the application scenarios with high real-time requirements.

[0067] In one embodiment, step S1, which involves performing image standardization and storage processing based on the software interface screenshot to obtain a standardized image library, includes:

[0068] (1) Convert the obtained screenshots of each software interface into a standard image storage format to obtain a standard format image.

[0069] Specifically, this application converts all images to the standard RGB color format and saves them in a lossless PNG format to ensure pixel consistency.

[0070] (2) Scale each standard format image to the preset image size. During the scaling process, use edge filling to perform center alignment to obtain the corresponding normalized image.

[0071] Specifically, considering that the acquired standard format images may have different sizes, in order to ensure accurate similarity calculation during subsequent matching, this application uniformly scales all standard format images to a fixed size. (The image size is not fixed and can be adjusted according to the detection accuracy requirements or hardware performance.)

[0072] In addition, in order to preserve the original shape and proportion of the image, this application also uses an edge filling method for center alignment during the scaling process (that is, after the image is scaled, the blank areas that may appear at the edge of the image are filled according to the original edge position of the icon, so that the icon is always in the center of the image) to avoid the icon being deformed due to stretching.

[0073] In one embodiment, this application uses bilinear interpolation (an image interpolation algorithm commonly used in geometric transformations such as image scaling and rotation) to scale each standard format image to a preset image size to ensure that the image retains clear details when it is reduced in size.

[0074] (3) Integrate, store and manage the normalized images to obtain a standardized image library.

[0075] Specifically, this application utilizes a database management system to integrate, store, and manage normalized images. Through database query and update functions, it achieves efficient management of the standardized image library, ensuring the security, integrity, and consistency of image data.

[0076] In one embodiment, step S3 involves calculating the overall similarity of the target mask relative to each image in the standardized image library using the following steps:

[0077] Step S31: Determine the color histogram similarity between the standardized image and the target mask based on the histogram bus distance metric.

[0078] Specifically, after obtaining the histogram of the segmented region corresponding to the target mask... and histograms of standardized images Subsequently, this application will be approved. Calculate the bus distance between the two (i.e., take the square root of the product of the corresponding bin values ​​of the two histograms and then sum them, where...). , This represents the value of the histogram of the segmented region and the normalized image at the i-th bin. Then, based on... The color histogram similarity is calculated using this formula.

[0079] Step S32: Determine the structural similarity between the standardized image and the target mask based on the similarity of image feature elements.

[0080] Specifically, this application will take the maximum mask bounding box of the segmented region and scale it according to the scaling rules of each image in the standardized image library. This is to ensure consistent size and feature representation for subsequent structural similarity calculations.

[0081] In one embodiment, this application calculates the structural similarity between the normalized image and the target mask using the following formula:

[0082] ;

[0083] in, and These represent the mean brightness values ​​of the segmented regions and the normalized image, respectively. and Let represent the contrast variance of the segmented region and the normalized image, respectively. Describing covariance, and This represents the stability constant.

[0084] Step S33: Perform weighted fusion based on the color histogram similarity and the structural similarity to obtain the comprehensive similarity of the target mask relative to the standardized image.

[0085] Specifically, this application will use color histogram similarity. and structural similarity The images are combined according to a certain proportion. Specifically, this application adaptively adjusts the weight values ​​based on the color and structural features of the images, and combines the color histogram similarity... Multiply by its corresponding weight Structural similarity Multiply by its corresponding weight The results are then added together to obtain the overall similarity of the target mask relative to the standardized image, thus taking into account both color and structural information.

[0086] In one embodiment, step S33, the weighted fusion based on the color histogram similarity and the structural similarity, includes: combining color histogram similarity, structural similarity, histogram normalized entropy, saturation gate value, and gradient gate value, adaptively calculating the dynamic weights of color and structure through multi-parameter coordination with exponential and temperature control, and performing weighted fusion of the color histogram similarity and the structural similarity based on the dynamic weights.

[0087] Specifically, this application comprehensively considers multiple features of color and structure, and uses exponential functions and gating values ​​to achieve adaptive weight allocation, thereby more reasonably integrating color histogram similarity and structural similarity to obtain a more accurate comprehensive similarity.

[0088] In one embodiment, the dynamic weights of color and structure are adaptively calculated using the following formula:

[0089] ;

[0090] in, and These represent the weights of color and structure, respectively. Represents the temperature constant. Indicates the similarity of color histograms. Indicates structural similarity. This represents the normalized entropy of the histogram. Indicates the saturation threshold value. This represents the gradient gating value.

[0091] It should be noted that, ,in, This represents the entropy of the histogram of the segmented regions. This represents the total number of bins in the histogram. The smaller the value, the more defined the color of the segmented region. ,in, This represents the average saturation of the segmented region. This represents the preset equilibrium constant. ,in, This represents the average gradient magnitude of the segmented region. This represents the preset equilibrium constant.

[0092] Specifically, molecules The main considerations were color similarity and color certainty (reflected in...) The influence of color similarity and saturation is also considered. When color similarity is high, color certainty is high, and saturation is appropriate, the index value will be larger, thus increasing the color weight. Increase. Denominator It is the sum of index values ​​related to color and structure, used to normalize the weights and ensure... Through the above calculations, the dynamic weights can adaptively adjust according to the color and structural features of the image. For example, when the image has rich and defined color information, the color weights... The value of the structure weight will be relatively large; while when the image structure features are obvious, the structure weight will be smaller. The value of will be relatively large. This dynamic adjustment mechanism makes the calculation of comprehensive similarity more consistent with the actual characteristics of the image, thus improving the accuracy of similarity assessment.

[0093] In one embodiment, step S5, which involves constructing an information input and recognition area based on each interaction absolute coordinate point, includes:

[0094] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point.

[0095] (2) Select the leftmost point in the horizontal direction from the set of row points and use it as the reference point.

[0096] Specifically, to ensure the accuracy and stability of the information entry and recognition area positioning, this application uses the leftmost horizontal point in the row point set as the reference point, and uses it as the core starting point for coordinate calculation, and constructs a rectangular area that meets the needs of the actual interaction scenario according to the predetermined width, height and interval parameters.

[0097] (3) Using the reference point as the starting reference point for constructing the rectangular area, the rectangular area coordinates are calculated and constructed by mathematical operations based on coordinate offset according to the predefined rectangular area width and height and horizontal interval, forming an information input recognition area for locating interactive elements to achieve automated information input.

[0098] Specifically, from the x-coordinate of the reference point Subtract the horizontal intervals to the left one by one and the width of the rectangular area Then, the x-coordinate of the left boundary of the rectangular region was determined. This ensures that the rectangle is to the left of the reference point and has an appropriate distance from it. Furthermore, this application also subtracts the horizontal spacing based on the reference point. The x-coordinate of the right boundary of the rectangular region was then determined. Furthermore, this application also uses the ordinate of the reference point... Offset upwards respectively and downward offset The distance was used to determine the ordinates of the upper and lower boundaries of the rectangular region. and This ensures that the rectangular area is symmetrically distributed vertically around the reference point. It should be noted that this area is primarily used to identify the question stem or attribute fields, providing a precise positioning foundation for subsequent operations such as accurate OCR recognition, accurate option location and selection, and efficient text information parsing and field mapping.

[0099] In one embodiment, step S5, which involves constructing an information input and recognition area based on each interaction absolute coordinate point, includes:

[0100] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point.

[0101] (2) Based on the set of row points, when the number of points in the set is greater than a preset threshold, a rectangular area for identifying the content of the option is constructed to the right of any point in the same row.

[0102] Specifically, when the number of points in a row exceeds a preset threshold, it indicates that the row may contain multiple selectable options. For any point in the same row ( , Construct a rectangular region to its right. The width and height of this rectangular region are set to... and Among them, height With the height of the left strip Keep it consistent, width The value range is between [28, 32] (the specific value can be adjusted according to the actual interface display effect and recognition accuracy). The coordinates of the four vertices of the rectangular area are defined as follows: ,in, , , , This coordinate calculation method allows for precise location and extent of a rectangular area, enabling subsequent identification of content within that area and the acquisition of brief options such as "Male / Female," "Yes / No," and "A / B / C."

[0103] Please refer to Figure 2 This application discloses a special inspection equipment software startup status recognition and automatic information entry system. The system includes a historical data preprocessing module, a real-time desktop image segmentation processing module, a special inspection equipment software startup status recognition module, an interactive absolute coordinate point conversion module, and an automatic information entry module, wherein:

[0104] The historical data preprocessing module is used to acquire screenshots of the software interface of the special inspection equipment software in the historical startup state, and perform interactive relative coordinate CSV conversion and image standardization storage processing based on the software interface screenshots to obtain relative position files and standardized image libraries.

[0105] The real-time desktop image segmentation processing module is used to acquire the real-time desktop image of the special inspection equipment software, input the real-time desktop image into the image segmentation model, and obtain the target mask with the largest area through automatic segmentation, mask generation and area filtering processing.

[0106] The special inspection equipment software startup status recognition module is used to calculate the comprehensive similarity of the target mask relative to each image in the standardized image library. When the maximum value of the comprehensive similarity is greater than a preset threshold, the special inspection equipment software is determined to be in real-time startup state, and the interactive absolute coordinate point conversion module is started. Otherwise, the real-time desktop image segmentation processing module is started to process the next frame image.

[0107] The interactive absolute coordinate point conversion module is used to convert interactive relative coordinate points obtained from the relative position file into interactive absolute coordinate points based on the target mask bounding box.

[0108] The automatic information entry module is used to construct an information entry and recognition area based on each interaction absolute coordinate point, and to perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information entry and recognition area to achieve automatic information entry.

[0109] In one embodiment, the above modules are also used to implement a method for identifying the startup status of special inspection equipment software and automatically recording information as described in any of the foregoing method embodiments, and this application does not limit this.

[0110] As can be seen from the above, the special inspection equipment software startup status recognition and automatic information entry system disclosed in this application, on the one hand, acquires screenshots of the software interface in historical startup states, performs interactive relative coordinate CSV conversion and image standardization storage processing, and constructs a relative position file and a standardized image library; during real-time detection, it uses an image segmentation model to obtain the target mask with the largest area and compares it with the standardized image library to determine the software state; then, using the target mask bounding box as a reference, it converts the interactive relative coordinate points in the relative position file into interactive absolute coordinate points; this processing method ensures that no matter how the position of the software window changes on the actual desktop, it can accurately determine the absolute position of the interactive element based on the target mask, ensuring accurate positioning for subsequent operations. On the other hand, it constructs an information entry recognition area based on the converted interactive absolute coordinate points, and performs a series of automated operations such as OCR recognition, option positioning and selection logic, text parsing and field mapping within this area to achieve automatic information entry; the entire process is executed automatically, avoiding positioning deviations caused by changes in window position due to manual operation, reducing entry errors caused by human factors, and greatly improving the stability and accuracy of information entry.

[0111] Please refer to Figure 3 This application discloses a readable storage medium, which includes a method program for identifying the startup status of special inspection equipment software and automatically recording information. When the method program for identifying the startup status of special inspection equipment software and automatically recording information is executed by a processor, it implements the steps of the method as described in any of the foregoing method embodiments.

[0112] As can be seen from the above, the readable storage medium disclosed in this application, on the one hand, acquires screenshots of the software interface in historical startup states, performs interactive relative coordinate CSV conversion and image standardization storage processing, and constructs a relative position file and a standardized image library; during real-time detection, it uses an image segmentation model to obtain the target mask with the largest area, and compares it with the standardized image library to determine the software state; then, using the target mask bounding box as a reference, it converts the interactive relative coordinate points in the relative position file into interactive absolute coordinate points; this processing method ensures that regardless of how the position of the software window changes on the actual desktop, it can accurately determine the absolute position of the interactive element based on the target mask, ensuring accurate positioning for subsequent operations. On the other hand, it constructs an information input recognition area based on the converted interactive absolute coordinate points, and performs a series of automated operations such as OCR recognition, option positioning and selection logic, text parsing and field mapping within this area to achieve automatic information input; the entire process is executed automatically, avoiding positioning deviations caused by changes in window position due to manual operation, reducing input errors caused by human factors, and greatly improving the stability and accuracy of information input.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying the startup status of special inspection equipment software and automatically entering information, characterized in that, Includes the following steps: S1. Obtain screenshots of the software interface of the special inspection equipment in its historical startup state. Based on the screenshots, perform interactive relative coordinate CSV conversion and image standardization storage processing to obtain a relative position file and a standardized image library. S2. Obtain the real-time desktop image of the special inspection equipment software, input the real-time desktop image into the image segmentation model, and obtain the target mask with the largest area through automatic segmentation, mask generation and area filtering. S3. Calculate the comprehensive similarity of the target mask relative to each image in the standardized image library. If the maximum value of the comprehensive similarity is greater than the preset threshold, determine that the special inspection equipment software is in real-time startup state and execute step S4. Otherwise, return to step S2 to process the next frame image. S4. Using the target mask bounding box as a reference, convert the interactive relative coordinates obtained from the relative position file into interactive absolute coordinates. S5. Construct an information input and recognition area based on the absolute coordinates of each interaction, and perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information input and recognition area to achieve automatic information input.

2. The method according to claim 1, characterized in that, In step S1, the step of converting the interactive relative coordinates into a CSV file based on the screenshot of the software interface to obtain a relative position file includes: (1) Select the reference point and the key interaction point for information entry from the screenshot of the software interface; (2) Based on the reference point, perform relative coordinate transformation on each key interaction point to form a set of relative coordinate points; (3) Store the set of relative coordinate points into a CSV file to obtain a relative position file.

3. The method according to claim 1, characterized in that, In step S1, the image standardization and storage processing based on the software interface screenshot to obtain a standardized image library includes: (1) Convert the obtained screenshots of each software interface into a standard image storage format to obtain a standard format image; (2) Scale each standard format image to the preset image size. During the scaling process, use edge filling to perform center alignment to obtain the corresponding normalized image. (3) Integrate, store and manage the normalized images to obtain a standardized image library.

4. The method according to claim 1, characterized in that, In step S3, the comprehensive similarity of the target mask relative to each image in the standardized image library is calculated through the following steps: S31. Based on the histogram bus distance metric, determine the color histogram similarity between the standardized image and the target mask; S32. Based on the similarity of image feature elements, determine the structural similarity between the standardized image and the target mask; S33. Based on the color histogram similarity and the structural similarity, a weighted fusion is performed to obtain the comprehensive similarity of the target mask relative to the standardized image.

5. The method according to claim 4, characterized in that, In step S33, the weighted fusion based on the color histogram similarity and the structural similarity includes: combining color histogram similarity, structural similarity, histogram normalized entropy, saturation gate value, and gradient gate value, adaptively calculating the dynamic weights of color and structure through multi-parameter coordination with exponential and temperature control, and performing weighted fusion of the color histogram similarity and the structural similarity based on the dynamic weights.

6. The method according to claim 5, characterized in that, The dynamic weights for color and structure are adaptively calculated using the following formula: ； in, and These represent the weights of color and structure, respectively. Represents the temperature constant. Indicates the similarity of color histograms. Indicates structural similarity. This represents the normalized entropy of the histogram. Indicates the saturation threshold value. This represents the gradient gating value.

7. The method according to claim 1, characterized in that, In step S5, constructing the information input and recognition area based on each interaction absolute coordinate point includes: (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point; (2) Select the leftmost horizontal point from the set of row points and use it as the reference point; (3) Using the reference point as the starting reference point for constructing the rectangular area, the rectangular area coordinates are calculated and constructed by mathematical operations based on coordinate offset according to the predefined rectangular area width and height and horizontal interval, forming an information input recognition area for locating interactive elements to achieve automated information input.

8. The method according to claim 1, characterized in that, In step S5, constructing the information input and recognition area based on each interaction absolute coordinate point includes: (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point; (2) Based on the set of row points, when the number of points in the set is greater than a preset threshold, a rectangular area for identifying the content of the option is constructed to the right of any point in the same row.

9. A system for identifying the startup status of special inspection equipment software and automatically entering information, characterized in that, The system includes a historical data preprocessing module, a real-time desktop image segmentation processing module, a special inspection equipment software startup status recognition module, an interactive absolute coordinate point conversion module, and an automatic information entry module, wherein: The historical data preprocessing module is used to acquire screenshots of the software interface of the special inspection equipment software in the historical startup state, and perform interactive relative coordinate CSV conversion and image standardization storage processing based on the software interface screenshots to obtain relative position files and standardized image libraries. The real-time desktop image segmentation processing module is used to acquire the real-time desktop image of the special inspection equipment software, input the real-time desktop image into the image segmentation model, and obtain the target mask with the largest area through automatic segmentation, mask generation and area filtering processing. The special inspection equipment software startup status recognition module is used to calculate the comprehensive similarity of the target mask relative to each image in the standardized image library. When the maximum value of the comprehensive similarity is greater than a preset threshold, the special inspection equipment software is determined to be in real-time startup state, and the interactive absolute coordinate point conversion module is started. Otherwise, the real-time desktop image segmentation processing module is started to process the next frame image. The interactive absolute coordinate point conversion module is used to convert interactive relative coordinate points obtained from the relative position file into interactive absolute coordinate points based on the target mask bounding box. The automatic information entry module is used to construct an information entry and recognition area based on each interaction absolute coordinate point, and to perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information entry and recognition area to achieve automatic information entry.

10. A readable storage medium, characterized in that, The readable storage medium includes a special inspection equipment software startup status recognition and information automatic entry method program. When the special inspection equipment software startup status recognition and information automatic entry method program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images