Self-adaptive special inspection information automatic input method and system

Abstract

The embodiment of the invention provides a self-adaptive special inspection information automatic input method and system, and the method comprises the steps: obtaining a full-screen window image and a full-screen added fixed information screenshot of special inspection equipment software in a historical starting state, and constructing an information input position file and a standardized image library according to the full-screen window image and the full-screen added fixed information screenshot; acquiring a real-time desktop image of the special inspection equipment software, and performing horizontal / vertical line segment analysis, directional screening and alignment optimization based on the real-time desktop image to obtain a target line segment set; performing candidate rectangular frame generation, screening, evaluation and full-screen window form judgment on the basis of the target line segment set so as to determine full-screen window related information meeting requirements; performing bounding box positioning and coordinate conversion according to the full-screen window related information to obtain an absolute coordinate point set; and constructing an information input identification area based on the absolute coordinate point set, and performing OCR identification, option positioning and selection logic, text analysis and field mapping in the information input identification area to realize automatic information input.

Description

Technical Field

[0001] This application relates to the field of special equipment inspection and automation technology, and more specifically, to an adaptive method and system for automatically entering special inspection information. Background Technology

[0002] When integrating data between medical special equipment and hospital information systems, existing technologies primarily rely on automated script solutions for "one-click entry" to achieve automatic patient information entry. This solution requires pre-configuration of the graphical user interface (GUI) for each type of special equipment software, forming a configuration scheme. Currently, similar technologies used in implementing this function include manual identification and matching of configuration schemes, as well as keyword-based knowledge base retrieval. While these technologies can achieve basic data capture and entry in scenarios with stable equipment GUI layouts and standardized field labels, they also have significant drawbacks, such as: 1) When the host computer is in full-screen mode, the information entered is divided into window and fixed-position full-screen entry. A single segmentation method cannot accurately segment certain types, leading to either automatic information entry or incorrect information entry in different situations; 2) Frequent manual input and intervention are required, resulting in a heavy workload and low efficiency for operators.

[0003] In summary, to address the issue that a single method cannot simultaneously input information in both windowed and fixed states when the host computer software is in full-screen mode during the use of special examination equipment, it is necessary to propose an adaptive method for determining whether the information being added is in a windowed or fixed state. This would enable automatic and accurate input of patient information across different device interface layouts and in full-screen mode. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an adaptive method and system for automatically entering special inspection information, which addresses the shortcomings of the prior art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: an adaptive method for automatically entering special inspection information, the method comprising:

[0006] S1. Obtain full-screen window images and full-screen screenshots of fixed information added to the special inspection equipment software in its historical startup state, and use these to construct information entry location files and a standardized image library;

[0007] S2. Obtain the real-time desktop image of the special inspection equipment software, and perform horizontal / vertical line segment analysis, orientation filtering, and alignment optimization based on the real-time desktop image to obtain the target line segment set;

[0008] S3. Based on the target line segment set, generate, filter, evaluate, and determine the full-screen window format of candidate rectangles to identify relevant information of the full-screen window that meets the requirements.

[0009] S4. Based on the full-screen window information, perform bounding box positioning and coordinate transformation to obtain the absolute coordinate point set;

[0010] S5. Construct an information input and recognition area based on the absolute coordinate point set, and perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information input and recognition area to achieve automatic information input.

[0011] Furthermore, in step S1, based on the full-screen window image and the full-screen screenshot with added fixed information, an information entry location file is constructed, including:

[0012] (1) Select a reference point and the key interaction point for corresponding information input from the full-screen window image, and select a fixed information position point from the full-screen screenshot of adding fixed information;

[0013] (2) Based on the reference point, perform relative coordinate transformation on each key interaction point to form a set of relative coordinate points;

[0014] (3) Store the set of relative coordinate points and the fixed information location points into a CSV file to obtain the information entry location file.

[0015] Furthermore, in step S1, image standardization and storage processing is performed based on the full-screen window image and the full-screen screenshot with added fixed information to obtain a standardized image library, including:

[0016] (1) Convert the acquired full-screen window images and full-screen screenshots with added fixed information into a standard image storage format to obtain a standard format image;

[0017] (2) Scale each standard format image to the preset image size. During the scaling process, use edge filling to perform center alignment to obtain the corresponding normalized image.

[0018] (3) Integrate, store and manage the normalized images to obtain a standardized image library.

[0019] Furthermore, in step S2, the horizontal / vertical line segment analysis and directional filtering based on the real-time desktop image includes:

[0020] S21. Perform edge detection based on the real-time desktop image to obtain an edge binary map;

[0021] S22. Use a line detection algorithm to extract lines from the edge binary map to obtain an initial set of line segments;

[0022] S23. Based on the direction angle and length of each line segment in the set, by traversing all angle scales, the total length of the line segments that meet the preset angle tolerance judgment conditions under each angle scale is counted, and a weighted histogram is constructed accordingly.

[0023] S24. Based on the weighted histogram, perform peak detection and angle selection to determine the near-horizontal and near-vertical directions;

[0024] S25. Based on the near-horizontal and near-vertical directions, select approximate oriented line segments that satisfy these two directions from the initial line segment set, and perform adjacent line segment identification and merging, as well as length filtering operations based on each approximate oriented line segment to obtain a line segment set.

[0025] Furthermore, in step S2, alignment optimization is performed based on the selected line segment set to obtain the target line segment set, including:

[0026] S26. Based on the edge binary map, count the number of edge pixels along the horizontal and vertical directions to obtain the statistical results of the number of edge pixels in each row and column in the horizontal and vertical directions;

[0027] S27. In order to locate the position of the edge concentration, smoothing and peak detection operations are performed based on the above statistical results to obtain the set of horizontal position peaks and the set of vertical position peaks.

[0028] S28. In order to align the line segments more accurately with the edge concentration position, each line segment in the oriented filtered line segment set is snapped to the corresponding horizontal and vertical lines in the horizontal position peak set and the vertical position peak set, to obtain the target line segment set after snapping and alignment.

[0029] Furthermore, in step S3, the generation, filtering, evaluation, and full-screen window format determination of candidate rectangles based on the target line segment set, to determine the relevant information of the full-screen window that meets the requirements, includes:

[0030] S31. Select multiple horizontal and vertical lines from the target line segment set and combine them in pairs to form multiple candidate rectangles.

[0031] S32. Filter the multiple candidate rectangles formed according to the preset geometric filtering conditions to obtain a set of target rectangles that meet the geometric filtering conditions.

[0032] S33. For each target rectangle in the set, a weighted calculation is performed based on the ring ratio, the overall smoothness of the window interface, and the edge-fitting ratio to obtain the overall window evaluation score.

[0033] S34. When the difference between the comprehensive window evaluation score of the corresponding target rectangle and the corresponding edge-fitting ratio is greater than a preset threshold, it is determined whether the full-screen window format information corresponding to the rectangle is added.

[0034] S35. When the maximum value of the comprehensive window evaluation score is less than the preset threshold, determine the situation of information being filled in the current fixed position of the full screen.

[0035] Furthermore, in step S4, the step of performing bounding box positioning and coordinate transformation based on the full-screen window information to obtain a set of absolute coordinate points includes:

[0036] S41. Based on the full-screen window related information, when it is determined that the current situation corresponds to the addition of full-screen window form information, the SAM model is used to segment the real-time desktop image, and the bounding box image that can cover the target information corresponding to the obtained maximum mask is determined.

[0037] S42. Based on the full-screen window information, when it is determined that the current situation corresponds to the information being filled in at a fixed position on the full screen, the real-time desktop image is cropped into multiple sub-region images, and the region splitting and merging algorithm is used to perform region integration and boundary determination processing on the multiple sub-region images to obtain a bounding box-enclosed region image that can cover the target information.

[0038] S43. Calculate the intersection-union ratio (IUR) of the bounding box region image relative to each image in the standardized image library. If the maximum IUR value is greater than a preset threshold, determine that the special inspection equipment software is in real-time startup state and execute subsequent steps; otherwise, process the next frame of real-time desktop image.

[0039] S44. When determining to use the SAM model for image segmentation, the corresponding interactive relative coordinates obtained from the information entry location file are converted into absolute coordinates based on the obtained maximum mask bounding box. When determining to use the region splitting and merging algorithm to determine the bounding box, the corresponding interactive relative coordinates are directly converted into absolute coordinates to obtain a set of absolute coordinates.

[0040] Furthermore, in step S5, constructing the information input and recognition area based on the absolute coordinate point set includes:

[0041] (1) Construct a set of row points with the same ordinate based on the absolute coordinates of each interactive point in the set;

[0042] (2) Select the leftmost horizontal point from the set of row points and use it as the reference point;

[0043] (3) Using the reference point as the starting reference point for constructing the rectangular area, the rectangular area coordinates are calculated and constructed by mathematical operations based on coordinate offset according to the predefined rectangular area width and height and horizontal interval, forming an information input recognition area for locating interactive elements to achieve automated information input.

[0044] Furthermore, in step S5, constructing the information input and recognition area based on the absolute coordinate point set includes:

[0045] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point;

[0046] (2) Based on the set of row points, when the number of points in the set is greater than a preset threshold, a rectangular area for identifying the content of the option is constructed to the right of any point in the same row.

[0047] Secondly, this application discloses an adaptive automatic information entry system for special inspections, the system comprising a historical data preprocessing module, a real-time image line segment analysis module, a full-screen window information determination module, a coordinate positioning and transformation module, and an automatic information entry module, wherein:

[0048] The historical data preprocessing module is used to acquire full-screen window images and full-screen screenshots with added fixed information of the special inspection equipment software in the historical startup state, and to construct information entry location files and standardized image libraries based on these images.

[0049] The real-time image line segment analysis module is used to acquire the real-time desktop image of the special inspection equipment software, and perform line segment analysis, orientation filtering and alignment optimization based on the real-time desktop image to obtain the target line segment set.

[0050] The full-screen window information determination module is used to generate, filter, evaluate, and determine the full-screen window form based on the target line segment set, so as to determine the relevant information of the full-screen window that meets the requirements.

[0051] The coordinate positioning and transformation module is used to perform bounding box positioning and coordinate transformation based on the full-screen window information to obtain a set of absolute coordinate points.

[0052] The automatic information entry module is used to construct an information entry and recognition area based on the absolute coordinate point set, and to perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information entry and recognition area to achieve automatic information entry.

[0053] The beneficial effects of this invention are as follows: By preprocessing historical data to construct information entry location files and a standardized image library, a basic reference standard is provided for subsequent processing. Line segment analysis and other operations are performed on real-time desktop images to obtain a set of target line segments, and further, relevant information about the full-screen window is determined, enabling precise understanding of the software interface layout features. Based on this, bounding box positioning and coordinate transformation are performed to obtain a set of absolute coordinate points, providing an accurate basis for information entry positioning. Finally, an information entry recognition area is constructed, and automatic information entry is achieved. This series of steps comprehensively utilizes multiple technical means, solving the problem that traditional single methods cannot effectively distinguish between windowed and fixed forms of added information when the host computer software is in full-screen mode during the use of special inspection equipment. This improves the accuracy and efficiency of information entry and enhances the convenience and intelligence of the special inspection equipment software. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating an adaptive method for automatically entering special inspection information disclosed in this invention.

[0055] Figure 2 This is a schematic diagram of the structure of an adaptive special inspection information automatic entry system disclosed in this invention;

[0056] Figure 3 This is a schematic diagram of the structure of a readable storage medium disclosed in this invention. Detailed Implementation

[0057] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0058] Please refer to Figure 1 This application discloses an adaptive method for automatically entering special inspection information, comprising the following steps:

[0059] Step S1: Obtain full-screen window images and full-screen screenshots of added fixed information from the special inspection equipment software in its historical startup state, and use these to construct information entry location files and a standardized image library.

[0060] Step S2: Obtain the real-time desktop image of the special inspection equipment software, and perform horizontal / vertical line segment analysis, orientation filtering, and alignment optimization based on the real-time desktop image to obtain the target line segment set.

[0061] Step S3: Based on the target line segment set, candidate rectangles are generated, filtered, evaluated, and full-screen window format is determined to identify the relevant information of the full-screen window that meets the requirements.

[0062] Step S4: Based on the full-screen window information, perform bounding box positioning and coordinate transformation to obtain a set of absolute coordinate points.

[0063] Step S5: Construct an information input and recognition area based on the absolute coordinate point set, and perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information input and recognition area to achieve automatic information input.

[0064] As can be seen from the above, the adaptive automatic information entry method for special inspection disclosed in this application constructs an information entry location file and a standardized image library through historical data preprocessing, providing a basic reference standard for subsequent processing. Line segment analysis and other operations are performed on real-time desktop images to obtain a set of target line segments, and further, relevant information of the full-screen window is determined, enabling precise understanding of the layout characteristics of the software interface. Based on this, bounding box positioning and coordinate transformation are performed to obtain a set of absolute coordinate points, providing an accurate basis for information entry positioning. Finally, an information entry recognition area is constructed, and automatic information entry is achieved. This series of steps comprehensively utilizes multiple technical means, solving the problem that traditional single methods cannot effectively distinguish between windowed and fixed forms of added information when the host computer software is in full-screen mode during the use of special inspection equipment. This improves the accuracy and efficiency of information entry and enhances the convenience and intelligence of the special inspection equipment software.

[0065] In one embodiment, step S1 involves constructing an information entry location file based on the full-screen window image and the full-screen screenshot with added fixed information, including:

[0066] (1) Select a reference point and the key interaction point for corresponding information input from the full-screen window image, and select the fixed information position point from the full-screen screenshot of adding fixed information.

[0067] Specifically, for full-screen window images, this application selects the top-left corner of the image window as the reference point, and denotes it as... Then, select the key interaction points (such as input boxes, buttons, drop-down lists, etc.) in the software interface corresponding to information entry, and record the position of each point in the screen coordinate system. To ensure that key information remains in a fixed position throughout subsequent processes and is easily extracted and analyzed, this application selects fixed information location points from full-screen screenshots with added fixed information.

[0068] (2) Based on the reference point, perform relative coordinate transformation on each key interaction point to form a set of relative coordinate points.

[0069] Specifically, for each key interaction point This application will also calculate its relative to a reference point. Offset: , All relative coordinate points obtained It is stored as a set of relative coordinate points with reference to the reference point.

[0070] (3) Store the set of relative coordinate points and the fixed information location points into a CSV file to obtain the information entry location file.

[0071] It should be noted that, considering the simple structure and fast parsing speed of CSV format, which can effectively improve data processing efficiency, this application stores the information in a location file in CSV format to achieve efficient storage and fast retrieval of relative coordinate data, meeting the needs of application scenarios with high real-time requirements.

[0072] In one embodiment, step S1 involves performing image standardization and storage processing based on the full-screen window image and the full-screen screenshot with added fixed information to obtain a standardized image library, including:

[0073] (1) Convert the obtained full-screen window images and full-screen screenshots with added fixed information into a standard image storage format to obtain a standard format image.

[0074] Specifically, this application converts all images to the standard RGB color format and saves them in a lossless PNG format to ensure pixel consistency.

[0075] (2) Scale each standard format image to the preset image size. During the scaling process, use edge filling to perform center alignment to obtain the corresponding normalized image.

[0076] Specifically, considering that the acquired standard format images may have different sizes, in order to ensure accurate similarity calculation during subsequent matching, this application uniformly scales all standard format images to a fixed size. (The image size is not fixed and can be adjusted according to the detection accuracy requirements or hardware performance.)

[0077] In addition, in order to preserve the original shape and proportion of the image, this application also uses an edge filling method for center alignment during the scaling process (that is, after the image is scaled, the blank areas that may appear at the edge of the image are filled according to the original edge position of the icon, so that the icon is always in the center of the image) to avoid the icon being deformed due to stretching.

[0078] In one embodiment, this application uses bilinear interpolation (an image interpolation algorithm commonly used in geometric transformations such as image scaling and rotation) to scale each standard format image to a preset image size to ensure that the image retains clear details when it is reduced in size.

[0079] (3) Integrate, store and manage the normalized images to obtain a standardized image library.

[0080] Specifically, this application utilizes a database management system to integrate, store, and manage normalized images. Through database query and update functions, it achieves efficient management of the standardized image library, ensuring the security, integrity, and consistency of image data.

[0081] In one embodiment, step S2, which involves performing horizontal / vertical line segment analysis and directional filtering based on the real-time desktop image, includes:

[0082] Step S21: Perform edge detection based on the real-time desktop image to obtain an edge binary map.

[0083] Specifically, when acquiring a real-time desktop image, this application first converts it into a grayscale image, and then uses the Canny edge detection algorithm to perform edge exploration and binarization processing on this grayscale image to obtain an edge binary image.

[0084] Step S22: Use a line detection algorithm to extract lines from the edge binary map to obtain an initial set of line segments.

[0085] Specifically, this application uses the LSD (Line Segment Detector) algorithm to detect line segments based on image grayscale gradient information. This algorithm assumes that the grayscale changes of pixels in local areas of the image have a certain directionality. By calculating the gradient direction and magnitude of each pixel, pixels with similar gradient directions are merged into a line support region, thereby extracting the line segments.

[0086] Step S23: Based on the direction angle and length of each line segment in the set, by traversing all angle scales, the total length of the line segments that meet the preset angle tolerance judgment conditions under each angle scale is counted, and a weighted histogram is constructed accordingly.

[0087] Specifically, the first step is to clarify how the angle scale is divided. For example, we can use 1 degree as a scale interval to define a series of angle scales from 0 degrees to 360 degrees. Next, a preset angle tolerance is established. And begin iterating through these angle scales; where, for each angle scale... Iterate through each line segment in the set and obtain the direction angle of the line segment. and line segment length Determine whether it satisfies This is the tolerance condition for the angle. If it is met, then the line segment... Accumulated to the current angle scale In the corresponding statistical values, after traversing and counting all angle scales, a weighted histogram is constructed with the angle scales as the horizontal axis and the sum of the line segment lengths obtained from the statistics as the vertical axis. .

[0088] Step S24: Based on the weighted histogram, peak detection and angle selection are performed to determine the near-horizontal and near-vertical directions.

[0089] Specifically, this application is based on Peak detection and angle selection are performed, among which, Indicates the near-horizontal direction. Indicates a near-vertical direction.

[0090] Step S25: Based on the near horizontal and near vertical directions, select approximate oriented line segments that satisfy these two directions from the initial line segment set, and perform adjacent line segment identification and merging, as well as length filtering operations based on each approximate oriented line segment to obtain a line segment set.

[0091] Specifically, firstly, based on the near-horizontal direction determined in the previous steps... and near vertical direction Traverse the initial set of line segments Each line segment in and obtain its orientation angle. Through calculation and its value is compared with the preset angle tolerance. Compare the values, and if it is determined that the value is less than... If the line segment satisfies the requirement of being nearly horizontal or nearly vertical, then the line segment will be retained, resulting in a set of approximately oriented line segments that satisfy the requirement of being nearly horizontal or nearly vertical. .

[0092] Next, adjacent line segments are identified and merged. Specifically, for the set... For each line segment in the diagram, this application searches for adjacent line segments within a certain search range. Specifically, this application determines whether two line segments are in the same direction, that is, it judges the difference in their direction angles. Is it less than the preset directional difference threshold? And, to determine whether they are collinear, that is, by calculating the maximum normal distance between the two line segments relative to the same common straight line. Check if the selected value is less than the preset normal distance threshold. And, determine whether the gap between two line segments is small enough, that is, calculate the length of the interval gap after the two line segments are projected along the average direction. Check if the selected value is less than the preset gap threshold. If the requirements of being in the same direction, collinear, and having a small gap are met simultaneously, then the two line segments are merged into a longer line segment, and the line segment set is updated.

[0093] Finally, this application also performs a length filtering operation, by traversing the merged set of line segments and filtering each line segment. Calculate its length If there is Greater than the preset length threshold If the line segment is considered valid as a component of the target line segment set, it will be retained and the final line segment set will be obtained.

[0094] In one embodiment, step S2 involves alignment optimization based on the directional filtered set of line segments to obtain a target set of line segments, including:

[0095] Step S26: Based on the edge binary map, count the number of edge pixels along the horizontal and vertical directions to obtain the statistical results of the number of edge pixels in each row and column in the horizontal and vertical directions.

[0096] Specifically, the process begins by traversing the edge binary image. In the horizontal direction, starting from the first row of the image, each pixel is scanned row by row. If a pixel is an edge pixel (based on the characteristics of edge binary images, it is typically represented by a specific value such as 1 to indicate an edge pixel and 0 to indicate a non-edge pixel), the edge pixel count for that row is incremented by 1. This process is repeated for all rows, thus obtaining the edge pixel count for each row in the horizontal direction. The same procedure applies to the vertical direction, and this application does not impose further limitations on this.

[0097] Step S27: In order to locate the position of the edge concentration, smoothing and peak detection operations are performed based on the above statistical results to obtain the set of horizontal position peaks and the set of vertical position peaks.

[0098] Specifically, this application uses a peak detection algorithm to find the peak positions in the horizontal and vertical directions from the smoothed data, and collects the row and column positions corresponding to these peaks to form corresponding horizontal position peak sets and vertical position peak sets.

[0099] Step S28: In order to align the line segments more accurately with the edge concentration position, each line segment in the oriented filtered line segment set is snapped to the corresponding horizontal and vertical lines in the horizontal position peak set and the vertical position peak set, to obtain the target line segment set after snapping and alignment.

[0100] Specifically, this application iterates through each line segment in the selected line segment set. For each line segment, based on its directional characteristics (horizontal or vertical), it performs matching operations with both the horizontal and vertical position peak sets.

[0101] (1) If the line segment is horizontal, find the horizontal line closest to it in the set of horizontal position peaks, and calculate the difference between the coordinates of the line segment endpoint and the coordinates of the horizontal line, according to the given adsorption rule ( ,in, This represents a line segment in the near-horizontal direction after targeted filtering. This represents a specific horizontal line segment in the set of horizontal position peaks (when the difference between the two is less than 10 pixels, it is moved and merged), which snaps the horizontal line segment to the nearest horizontal line, so that the line segment is aligned with the horizontal line in the set of horizontal position peaks.

[0102] (2) If the line segment is vertical, then find the closest vertical line in the set of vertical position peaks, according to the adsorption rule ( ,in, This represents a line segment in the near-vertical direction after targeted filtering. This represents a specific vertical line segment in the set of vertical position peaks (when the difference between the two is less than 10 pixels, a move and merge operation is performed), snapping the vertical line segment to the nearest vertical line, aligning the line segment with the vertical line in the set of vertical position peaks.

[0103] After completing the snap-in alignment operation for all line segments, the processed line segments are integrated together to form a set of snap-in aligned target line segments.

[0104] In one embodiment, step S3, which involves generating, filtering, evaluating, and determining the full-screen window format based on the target line segment set to identify eligible full-screen window information, includes:

[0105] Step S31: Select multiple horizontal and vertical lines from the target line segment set and combine them in pairs to form multiple candidate rectangles.

[0106] Step S32: Filter the multiple candidate rectangles formed according to the preset geometric filtering conditions to obtain a set of target rectangles that meet the geometric filtering conditions.

[0107] Based on steps S31 to S32, it should be noted that this application will randomly select two horizontal lines and two vertical lines from the target line segment set to form a candidate rectangle R, and perform the following geometric screening on the candidate rectangle R:

[0108] (1) Angle constraint: ;

[0109] (2) Aspect Ratio Constraint: ;

[0110] (3) Area constraint: ;

[0111] in, , This represents the unit direction vector of the horizontal and vertical lines. This represents the transpose of the unit normal vector. The cosine of the angle between the horizontal and vertical lines. , This represents the length of the line segment between the horizontal and vertical lines after satisfying the angular constraints. This represents the area of ​​a rectangular frame that satisfies the aspect ratio constraint. , This indicates the image width and height.

[0112] Step S33: For each target rectangle in the set, a weighted calculation is performed based on the ring ratio, the overall smoothness of the window interface, and the edge-fitting ratio to obtain the overall window evaluation score.

[0113] Specifically, the ring-zone ratio The calculation formula is as follows: , , ,in, , , , This represents the mean / standard deviation of the pixels in the narrow rings inside / outside the bounding box. This represents a function that cuts x to the interval [0,1]. A robust score representing the brightness contrast of the inner / outer narrow rings. , Indicates by azimuth angle (in, G represents the vector average of the grayscale image corresponding to the real-time desktop image on the inner / outer narrow rings. This represents the fraction indicating the opposite direction of the gradients on the inner and outer sides. This represents the pre-assigned weights, and .

[0114] Specifically, the overall smoothness of the window interface The calculation formula is as follows: , , ,in, This represents the overall visibility score of window controls. This indicates the title bar smoothness score. This represents the pre-assigned weights, and , This represents the visibility score of the minimized page control in the upper right corner area of ​​the rectangle. This represents the visibility score of the full-screen control in the upper right corner area of ​​the rectangle. This represents the visibility score of the close page control in the upper right corner of the rectangle. This represents the standard deviation of the grayscale value of the top stripe area within the rectangle. This represents the standard deviation of gray levels within the entire rectangle. This represents the minimum constant used to avoid division by zero.

[0115] Specifically, edge ratio The calculation formula is as follows: ,in, This represents the distance from the four sides of the rectangle to the corresponding four sides of the entire image. Indicates the edge-fitting threshold. This indicates an indicator function. The function takes the value 1 if the condition within the square brackets is true, and 0 otherwise.

[0116] Finally, after obtaining the various ring-zone ratios Overall smoothness of the window interface and edge ratio After that, through A weighted calculation is performed to obtain the overall window evaluation score. It should be noted that the weight allocation of each indicator (ring-band ratio, overall smoothness of window interface, and edge-fitting ratio) is determined based on actual needs and experimental verification, and this application does not impose any restrictions on it.

[0117] Step S34: When the difference between the comprehensive window evaluation score of the corresponding target rectangle and the corresponding edge-fitting ratio is greater than a preset threshold, it is determined that the full-screen window format information corresponding to the rectangle has been added.

[0118] Specifically, when the difference between the overall window rating score and the edge-fitting ratio is greater than a preset threshold, it indicates that the rectangle has significant differences in layout characteristics compared to a regular non-full-screen window, and is more in line with the visual presentation and layout characteristics of a full-screen window. Therefore, it is determined that the rectangle corresponds to the situation where full-screen window form information is added. This determination helps to accurately distinguish the full-screen window type in window recognition and related processing, so as to carry out targeted operations and optimizations.

[0119] Step S35: When the maximum value of the comprehensive window evaluation scores is less than the preset threshold, determine the current situation of filling information in the corresponding fixed position of the full screen.

[0120] Specifically, when the maximum value of each comprehensive window evaluation score is less than a preset threshold, it means that the overall performance of all rectangles has not reached a relatively high level, i.e., there are no rectangles that clearly conform to the typical characteristics of a full-screen window. In this case, based on the actual scenario and business logic, the current situation of filling information in the corresponding fixed position of the full screen is determined.

[0121] In one embodiment, step S4, which involves performing bounding box positioning and coordinate transformation based on the full-screen window information to obtain a set of absolute coordinate points, includes:

[0122] Step S41: Based on the full-screen window related information, when it is determined that the current situation corresponds to the addition of full-screen window form information, the SAM model is used to segment the real-time desktop image, and the bounding box image that can cover the target information corresponding to the obtained maximum mask is determined.

[0123] Specifically, considering that full-screen window-style information addition often results in the target information being relatively prominent and occupying a significant area in the image, this application inputs the real-time desktop image into a pre-trained SAM (Segment Anything Model) when determining that the current situation involves full-screen window-style information addition. The SAM model, leveraging its powerful semantic segmentation capabilities, performs fine segmentation of different objects and regions in the image to generate multiple masks. These segmented masks are then sorted by area, and the mask with the largest area is selected. Finally, based on the edge information of this largest mask, a bounding box that completely covers the target information is determined.

[0124] Step S42: Based on the full-screen window information, when it is determined that the current situation corresponds to the information being filled in at a fixed position on the full screen, the real-time desktop image is cropped into multiple sub-region images, and the multiple sub-region images are integrated and boundary determined using a region splitting and merging algorithm to obtain a bounding box-enclosed region image that can cover the target information.

[0125] Specifically, this application will crop the real-time desktop image into 8 sub-region images (it should be noted that since the position of the information filled in at the fixed position of the full screen is not the same, some are on the left, some are on the right, etc., 8 parts are used to ensure that the ROI area is determined within these 8 parts):

[0126] ;

[0127] ;

[0128] in, Indicates the number of parts to be divided. Indicates the height of the top region of the image. Indicates the height of the bottom region of the image. Indicates the width of the right side of the image. This indicates the width of the left side of the image.

[0129] Specifically, the region splitting and merging algorithm begins with the cropped sub-region image. If the current region does not meet the preset homogeneity criterion (this criterion can be set based on the characteristics of the target information, such as color similarity, texture complexity, etc.), then the region is split into four equal sub-regions. Next, the splitting operation is recursively applied to the newly generated sub-regions until all sub-regions meet the homogeneity criterion. Then, the merging phase begins, checking whether adjacent sub-regions meet the merging condition, i.e., they are sufficiently similar in features. If they do, the adjacent sub-regions are merged into a larger region. Through continuous splitting and merging operations, the region division is gradually optimized, and finally, the bounding boxes that can encompass the target information are selected from the merged regions to form the region image.

[0130] Step S43: Calculate the intersection-union ratio (IUR) of the bounding box region image relative to each image in the standardized image library. If the maximum IUR value is greater than a preset threshold, determine that the special inspection equipment software is in real-time startup state and execute subsequent steps; otherwise, process the next frame of real-time desktop image.

[0131] Step S44: When determining to use the SAM model for image segmentation, the corresponding interactive relative coordinates obtained from the information entry location file are converted into absolute coordinates based on the obtained maximum mask bounding box. When determining to use the region splitting and merging algorithm to determine the bounding box, the corresponding interactive relative coordinates obtained are directly converted into absolute coordinates to obtain a set of absolute coordinates.

[0132] Based on steps S43 to S44, it should be noted that before calculating the intersection-union ratio, the bounding box-enclosed region image and each image in the standardized image library need to be grayscaled and processed by GLAHE (contrast-limited adaptive histogram equalization), Otsu adaptive threshold binarization, morphological denoising, and preservation of the largest connected component.

[0133] Specifically, grayscale conversion and GLAHE processing enhance the contrast between the foreground and background of the image, resulting in an enhanced grayscale image that provides a clearer image foundation for subsequent processing. Next, Otsu adaptive thresholding binarization is performed. This method automatically determines the optimal threshold based on the image's grayscale characteristics, converting the grayscale image into a binary image, making the target information in the image more prominent. Then, morphological denoising is performed, using morphological operations (such as erosion and dilation) to remove small noise points or spurs in the binary image, making the target area more regular. Finally, the largest connected component is preserved by removing some scattered small regions, retaining only the largest connected components, further highlighting the target information.

[0134] Furthermore, for each image in the standardized image library, this application obtains the bounding box-enclosed region image, the overlapping region area, and the union region area of ​​that image, and then calculates the Intersection over Union (IoU) using the formula: IoU = (overlapping region area / union region area). Next, all calculated IoU values ​​are recorded and compared, and the maximum value is identified. This maximum value is compared with a pre-set threshold, which is determined based on the image similarity requirements in the actual scenario, for example, it can be set to 0.7. Specifically, if the obtained maximum IoU value is greater than the pre-set threshold, the bounding box-enclosed region image is considered to be highly matched with an image in the standardized image library, and the special inspection equipment software is in real-time startup mode; otherwise, the match is considered to have failed, the frame is discarded, and the system waits for the next frame to continue detection.

[0135] In one embodiment, step S5, constructing the information entry and recognition area based on the absolute coordinate point set, includes:

[0136] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point.

[0137] (2) Select the leftmost point in the horizontal direction from the set of row points and use it as the reference point.

[0138] Specifically, to ensure the accuracy and stability of the information entry and recognition area positioning, this application uses the point with the smallest horizontal coordinate in the row point set as the reference point, and uses it as the core starting point for coordinate calculation, and constructs a rectangular area that meets the needs of the actual interaction scenario according to the predetermined width, height and interval parameters.

[0139] (3) Using the reference point as the starting reference point for constructing the rectangular area, the rectangular area coordinates are calculated and constructed by mathematical operations based on coordinate offset according to the predefined rectangular area width and height and horizontal interval, forming an information input recognition area for locating interactive elements to achieve automated information input.

[0140] Specifically, from the x-coordinate of the reference point Subtract the horizontal intervals to the left one by one and the width of the rectangular area Then, the x-coordinate of the left boundary of the rectangular region was determined. This ensures that the rectangle is to the left of the reference point and has an appropriate distance from the point. Furthermore, this application also uses the reference point... Based on, subtract the horizontal interval The x-coordinate of the right boundary of the rectangular region was then determined. Furthermore, this application also uses the ordinate of the reference point... Offset upwards respectively and downward offset The distance was used to determine the ordinates of the upper and lower boundaries of the rectangular region. and This makes the rectangular area symmetrically distributed in the vertical direction with the reference point as the center.

[0141] It should be noted that, and According to the Line spacing estimation between rows and adjacent rows and the horizontal distance from the leftmost point of the current row to the smallest x-coordinate in the row set. Adaptive adjustment is performed according to the following formula: , ,in, This represents the Sigmoid function, with a range of [0,1]. Indicates the height mapping scale, Indicates the width mapping scale. This indicates the reference value for the midpoint of the line spacing. This represents the reference value at the midpoint. Finally, the obtained... , It can adaptively construct OCR rectangles. This rectangle area is mainly used to recognize the question stem or attribute fields, and then fill in relevant information or select from drop-down lists based on the recognized content.

[0142] In one embodiment, step S5, constructing the information entry and recognition area based on the absolute coordinate point set, includes:

[0143] (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point.

[0144] (2) Based on the set of row points, when the number of points in the set is greater than a preset threshold, a rectangular area for identifying the content of the option is constructed to the right of any point in the same row.

[0145] Specifically, if there is only one point in the row set, it is skipped; if multiple points are determined, the right-side recognition region is constructed. Specifically, for the points in the row set... The right-side recognition area is defined as: ,in, , , and Representing the i-th point in the r-th row respectively The OCR recognition frame width and height can also be adaptively adjusted to match the long strip on the left. and The content in this area corresponds to short options such as "Male / Female", "Yes / No", and "A / B / C".

[0146] Please refer to Figure 2 This application discloses an adaptive automatic information entry system for special inspections. The system includes a historical data preprocessing module, a real-time image line segment analysis module, a full-screen window information determination module, a coordinate positioning and transformation module, and an automatic information entry module, wherein:

[0147] The historical data preprocessing module is used to acquire full-screen window images and full-screen screenshots with added fixed information of the special inspection equipment software in its historical startup state, and to construct information entry location files and standardized image libraries based on these images.

[0148] The real-time image line segment analysis module is used to acquire the real-time desktop image of the special inspection equipment software, and perform line segment analysis, orientation filtering and alignment optimization based on the real-time desktop image to obtain the target line segment set.

[0149] The full-screen window information determination module is used to generate, filter, evaluate, and determine the full-screen window form based on the target line segment set, so as to determine the relevant information of the full-screen window that meets the requirements.

[0150] The coordinate positioning and transformation module is used to perform bounding box positioning and coordinate transformation based on the relevant information of the full-screen window to obtain a set of absolute coordinate points.

[0151] The automatic information entry module is used to construct an information entry and recognition area based on the absolute coordinate point set, and to perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information entry and recognition area to achieve automatic information entry.

[0152] In one embodiment, the above modules are also used to implement an adaptive method for automatically recording special inspection information as described in any of the foregoing method embodiments, and this application does not limit this method.

[0153] As can be seen from the above, the adaptive automatic information entry system for special inspection disclosed in this application constructs an information entry location file and a standardized image library through historical data preprocessing, providing a basic reference standard for subsequent processing. Line segment analysis and other operations are performed on real-time desktop images to obtain a set of target line segments, and further, relevant information of the full-screen window is determined, enabling precise understanding of the layout characteristics of the software interface. Based on this, bounding box positioning and coordinate transformation are performed to obtain a set of absolute coordinate points, providing an accurate basis for information entry positioning. Finally, an information entry recognition area is constructed, and automatic information entry is achieved. This series of steps comprehensively utilizes multiple technical means, solving the problem that traditional single methods cannot effectively distinguish between windowed and fixed forms of added information when the host computer software is in full-screen mode during the use of special inspection equipment. This improves the accuracy and efficiency of information entry and enhances the convenience and intelligence of the special inspection equipment software.

[0154] Please refer to Figure 3 This application discloses a readable storage medium, which includes an adaptive method program for automatically recording special inspection information. When the adaptive method program for automatically recording special inspection information is executed by a processor, it implements the steps of the method described in any of the preceding claims.

[0155] As can be seen from the above, the readable storage medium disclosed in this application constructs an information entry location file and a standardized image library through historical data preprocessing, providing a basic reference standard for subsequent processing. Line segment analysis and other operations are performed on real-time desktop images to obtain a set of target line segments, and further, relevant information of the full-screen window is determined, enabling precise understanding of the layout features of the software interface. Based on this, bounding box positioning and coordinate transformation are performed to obtain a set of absolute coordinate points, providing an accurate basis for information entry positioning. Finally, an information entry recognition area is constructed, and automatic information entry is achieved. This series of steps comprehensively utilizes multiple technical means, solving the problem that traditional single methods cannot effectively distinguish between windowed and fixed forms of added information when the host computer software is in full-screen mode during the use of special inspection equipment. This improves the accuracy and efficiency of information entry and enhances the convenience and intelligence of the special inspection equipment software.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive method for automatically entering special inspection information, characterized in that, Includes the following steps: S1. Obtain full-screen window images and full-screen screenshots of fixed information added to the special inspection equipment software in its historical startup state, and use these to construct information entry location files and a standardized image library; S2. Obtain the real-time desktop image of the special inspection equipment software, and perform horizontal / vertical line segment analysis, orientation filtering, and alignment optimization based on the real-time desktop image to obtain the target line segment set; S3. Based on the target line segment set, generate, filter, evaluate, and determine the full-screen window format of candidate rectangles to identify relevant information of the full-screen window that meets the requirements. S4. Based on the full-screen window information, perform bounding box positioning and coordinate transformation to obtain the absolute coordinate point set; S5. Construct an information input and recognition area based on the absolute coordinate point set, and perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information input and recognition area to achieve automatic information input.

2. The method according to claim 1, characterized in that, In step S1, based on the full-screen window image and the full-screen screenshot with added fixed information, an information entry location file is constructed, including: (1) Select a reference point and the key interaction point for corresponding information input from the full-screen window image, and select a fixed information position point from the full-screen screenshot of adding fixed information; (2) Based on the reference point, perform relative coordinate transformation on each key interaction point to form a set of relative coordinate points; (3) Store the set of relative coordinate points and the fixed information location points into a CSV file to obtain the information entry location file.

3. The method according to claim 1, characterized in that, In step S1, image standardization and storage processing is performed based on the full-screen window image and the full-screen screenshot with added fixed information to obtain a standardized image library, including: (1) Convert the acquired full-screen window images and full-screen screenshots with added fixed information into a standard image storage format to obtain a standard format image; (2) Scale each standard format image to the preset image size. During the scaling process, use edge filling to perform center alignment to obtain the corresponding normalized image. (3) Integrate, store and manage the normalized images to obtain a standardized image library.

4. The method according to claim 1, characterized in that, In step S2, the analysis and directional filtering of horizontal / vertical line segments based on the real-time desktop image includes: S21. Perform edge detection based on the real-time desktop image to obtain an edge binary map; S22. Use a line detection algorithm to extract lines from the edge binary map to obtain an initial set of line segments; S23. Based on the direction angle and length of each line segment in the set, by traversing all angle scales, the total length of the line segments that meet the preset angle tolerance judgment conditions under each angle scale is counted, and a weighted histogram is constructed accordingly. S24. Based on the weighted histogram, perform peak detection and angle selection to determine the near-horizontal and near-vertical directions; S25. Based on the near-horizontal and near-vertical directions, select approximate oriented line segments that satisfy these two directions from the initial line segment set, and perform adjacent line segment identification and merging, as well as length filtering operations based on each approximate oriented line segment to obtain a line segment set.

5. The method according to claim 4, characterized in that, In step S2, alignment optimization is performed based on the selected line segment set to obtain the target line segment set, including: S26. Based on the edge binary map, count the number of edge pixels along the horizontal and vertical directions to obtain the statistical results of the number of edge pixels in each row and column in the horizontal and vertical directions; S27. In order to locate the position of the edge concentration, smoothing and peak detection operations are performed based on the above statistical results to obtain the set of horizontal position peaks and the set of vertical position peaks. S28. In order to align the line segments more accurately with the edge concentration position, each line segment in the oriented filtered line segment set is snapped to the corresponding horizontal and vertical lines in the horizontal position peak set and the vertical position peak set, to obtain the target line segment set after snapping and alignment.

6. The method according to claim 1, characterized in that, In step S3, the process of generating, filtering, evaluating, and determining the full-screen window format based on the target line segment set to identify qualified full-screen window information includes: S31. Select multiple horizontal and vertical lines from the target line segment set and combine them in pairs to form multiple candidate rectangles. S32. Filter the multiple candidate rectangles formed according to the preset geometric filtering conditions to obtain a set of target rectangles that meet the geometric filtering conditions. S33. For each target rectangle in the set, a weighted calculation is performed based on the ring ratio, the overall smoothness of the window interface, and the edge-fitting ratio to obtain the overall window evaluation score. S34. When the difference between the comprehensive window evaluation score of the corresponding target rectangle and the corresponding edge-fitting ratio is greater than a preset threshold, it is determined whether the full-screen window format information corresponding to the rectangle is added. S35. When the maximum value of the comprehensive window evaluation score is less than the preset threshold, determine the situation of information being filled in the current fixed position of the full screen.

7. The method according to claim 1, characterized in that, In step S4, the step of performing bounding box positioning and coordinate transformation based on the full-screen window information to obtain an absolute coordinate point set includes: S41. Based on the full-screen window related information, when it is determined that the current situation corresponds to the addition of full-screen window form information, the SAM model is used to segment the real-time desktop image, and the bounding box image that can cover the target information corresponding to the obtained maximum mask is determined. S42. Based on the full-screen window information, when it is determined that the current situation corresponds to the information being filled in at a fixed position on the full screen, the real-time desktop image is cropped into multiple sub-region images, and the region splitting and merging algorithm is used to perform region integration and boundary determination processing on the multiple sub-region images to obtain a bounding box-enclosed region image that can cover the target information. S43. Calculate the intersection-union ratio (IUR) of the bounding box region image relative to each image in the standardized image library. If the maximum IUR value is greater than a preset threshold, determine that the special inspection equipment software is in real-time startup state and execute subsequent steps; otherwise, process the next frame of real-time desktop image. S44. When determining to use the SAM model for image segmentation, the corresponding interactive relative coordinates obtained from the information entry location file are converted into absolute coordinates based on the obtained maximum mask bounding box. When determining to use the region splitting and merging algorithm to determine the bounding box, the corresponding interactive relative coordinates are directly converted into absolute coordinates to obtain a set of absolute coordinates.

8. The method according to claim 1, characterized in that, In step S5, constructing the information input and recognition area based on the absolute coordinate point set includes: (1) Construct a set of row points with the same ordinate based on the absolute coordinates of each interactive point in the set; (2) Select the leftmost horizontal point from the set of row points and use it as the reference point; (3) Using the reference point as the starting reference point for constructing the rectangular area, the rectangular area coordinates are calculated and constructed by mathematical operations based on coordinate offset according to the predefined rectangular area width and height and horizontal interval, forming an information input recognition area for locating interactive elements to achieve automated information input.

9. The method according to any one of claims 1-8, characterized in that, In step S5, constructing the information input and recognition area based on the absolute coordinate point set includes: (1) Construct a set of row points with the same ordinate based on each interaction absolute coordinate point; (2) Based on the set of row points, when the number of points in the set is greater than a preset threshold, a rectangular area for identifying the content of the option is constructed to the right of any point in the same row.

10. An adaptive automatic special inspection information entry system, characterized in that, The system includes a historical data preprocessing module, a real-time image line segment analysis module, a full-screen window information determination module, a coordinate positioning and transformation module, and an automatic information entry module, wherein: The historical data preprocessing module is used to acquire full-screen window images and full-screen screenshots with added fixed information of the special inspection equipment software in the historical startup state, and to construct information entry location files and standardized image libraries based on these images. The real-time image line segment analysis module is used to acquire the real-time desktop image of the special inspection equipment software, and perform line segment analysis, orientation filtering and alignment optimization based on the real-time desktop image to obtain the target line segment set. The full-screen window information determination module is used to generate, filter, evaluate, and determine the full-screen window form based on the target line segment set, so as to determine the relevant information of the full-screen window that meets the requirements. The coordinate positioning and transformation module is used to perform bounding box positioning and coordinate transformation based on the full-screen window information to obtain a set of absolute coordinate points. The automatic information entry module is used to construct an information entry and recognition area based on the absolute coordinate point set, and to perform OCR recognition, option positioning and selection logic, text parsing and field mapping within the information entry and recognition area to achieve automatic information entry.

Images