Visual large model special inspection equipment real-time identification method and system based on prior prompt

Abstract

The embodiment of the invention provides a visual large model special inspection equipment real-time identification method and system based on prior prompt, and the method comprises the steps: carrying out the reference point selection based on a historical desktop image of special inspection equipment software, and obtaining a comparison file which records the position information of a reference point; storing the icon images matched and associated according to the reference point to obtain a standardized image library; carrying out Otsu threshold-driven binarization segmentation and connected domain analysis on the basis of a real-time desktop image of the special inspection equipment software, and selecting candidate points from each connected domain; performing reference point difference verification and deviation calibration on each candidate point based on the comparison file to obtain a target candidate point set; driving a visual large model SAM to perform zero sample segmentation based on the target candidate point set to obtain a target mask set; and respectively calculating the comprehensive similarity of each target mask in the set relative to each icon image in the standardized image library, and determining the icon category identifier corresponding to the maximum value of the comprehensive similarity as a real-time identification result of the special inspection equipment software.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of special equipment detection and automation, in particular to a visual large model special inspection equipment real-time identification method and system based on prior prompt. BACKGROUND

[0002] When a medical information system and equipment are integrated, in order to realize automatic input of patient information, the existing technologies relied on include:

[0003] 1) Manual recognition and manual matching configuration scheme: this scheme needs to observe the interface, desktop icon, window title, etc. of the equipment software by naked eye, and combine with experience to judge the brand and model of the equipment. Then, a possible matching scheme is searched and selected from a knowledge base containing hundreds of configuration schemes to load and debug. This scheme can complete the matching of the configuration scheme to a certain extent, and provide basic operation guidance for information input, but it seriously depends on the "experience" of the implementation engineer, and a lot of time is needed to try and troubleshoot for new engineers or rare equipment; in addition, due to the high similarity of many equipment software interfaces, or the multiple versions of the software of the same model by the manufacturer, it is easy to confuse and cause the selection of the wrong configuration scheme, leading to information input errors.

[0004] 2) Keyword-based knowledge base retrieval: the existing configuration knowledge base mainly relies on manual "brand", "model" and other text tags for retrieval, which requires the implementation engineer to accurately input the keywords to find the target configuration. However, when the equipment information is not clear or the label is not standardized, the retrieval process becomes very difficult. In addition, due to the lack of extraction and analysis capabilities of the visual features of the equipment software interface in the existing data management method, the interface data generated by the host computer system of each equipment and the configuration scheme data are stored independently, lacking a unified association identifier and integration mechanism, resulting in information islands, so that the visual information (what does the software look like) and the configuration scheme (how to operate) are disconnected, and cannot be directly associated, resulting in low utilization efficiency of the knowledge base.

[0005] In recent years, the visual large model Segment Anything Model (SAM) has made a breakthrough development. The visual large model SAM is a prompt-based segmentation model, which is pre-trained on a large-scale dataset and has strong image segmentation capability and generalization performance. The visual large model SAM can automatically segment objects in an image according to the input prompt information (such as points, boxes, etc.), and generate the corresponding mask.

[0006] Compared with traditional computer vision methods, the visual large model SAM has obvious advantages. It does not need to perform complex feature engineering and can automatically learn high-level semantic information in images, and has good adaptability to different types and styles of images. In the software identification scene of the special inspection equipment, SAM can accurately segment each element in the software interface, such as icons and windows, to provide accurate image area information for subsequent software identification.

[0007] However, as the number of desktop icons increases, the SAM visual large model automatic segmentation is prone to missed detection, and the segmentation stability is insufficient. In addition, the SAM visual large model automatic segmentation is slow, and it is difficult to realize synchronous display in the special inspection equipment. Therefore, considering that the reference points selected from the historical desktop image have stability and representativeness, and can provide reliable guidance for segmentation, and the special inspection equipment has high requirements for real-time performance and accuracy, in order to improve the accuracy and robustness of the visual large model SAM in the software identification of the special inspection equipment, it is necessary to introduce a point prompt optimization strategy assisted by a comparison file and a parallel computing acceleration method to improve the speed of segmentation and matching and enhance the stability of the segmentation result. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a visual large model special inspection equipment real-time identification method and system based on prior prompt to solve the above problems of the prior art.

[0009] The technical solution of the present application to solve the above technical problem is as follows: a visual large model special inspection equipment real-time identification method based on prior prompt, comprising the following steps:

[0010] S1, obtaining a historical desktop image of the special inspection equipment software, selecting reference points for assisting icon image segmentation based on the historical desktop image, and obtaining a comparison file recording reference point position information;

[0011] S2, standardizing and storing each icon image associated with the reference points to obtain a standardized image library;

[0012] S3, obtaining a real-time desktop image of the special inspection equipment software, performing Otsu threshold-driven binary segmentation and connected component analysis based on the real-time desktop image, and selecting candidate points from each connected component;

[0013] S4, verifying the difference between each candidate point and the reference point based on the comparison file, and calibrating the deviation to obtain a target candidate point set;

[0014] S5, inputting the target candidate point set as a point prompt to drive the visual large model SAM to perform zero-shot segmentation to obtain a corresponding target mask set;

[0015] S6, respectively calculate the comprehensive similarity of each target mask in the set relative to each icon image in the standardized image library, and determine the icon category corresponding to the maximum comprehensive similarity as the real-time recognition result of the special detection equipment software.

[0016] Further, in step S1, the reference point selection for assisting icon image segmentation based on the historical desktop image includes:

[0017] S11, select an initial prompt point from the historical desktop image, drive the visual large model SAM to perform single-point prompt segmentation according to the initial prompt point, obtain the corresponding initial segmentation mask, and synchronize the initial prompt point and the initial segmentation mask for visual display;

[0018] S12, in the display process, manually adjust the position of the initial prompt point in an interactive manner to form a new interactive prompt point, wherein the visual large model SAM generates a new segmentation mask based on the new interactive prompt point and the initial segmentation mask, and continuously iterates and updates until the interactive prompt point is located at the center position of the segmentation mask, and the prompt point coordinates at this time are recorded.

[0019] S13, arrange the recorded prompt point coordinates that meet the condition of being located at the center position of the segmentation mask into a point set to ensure that the point set can completely cover all icon positions in the left half of the image region, and finally form a comparison file recording the reference point position information.

[0020] Further, in step S2, the standardized storage processing of each icon image associated with the reference point matching is performed to obtain a standardized image library, which includes:

[0021] S21, convert each icon image associated with the reference point matching into a standard image storage format to obtain a standard format image;

[0022] S22, scale each standard format image to a preset image size, and use edge padding to perform center alignment during the scaling process to obtain a corresponding normalized image;

[0023] S23, integrate and store each normalized image for management to obtain a standardized image library.

[0024] Further, in step S3, the Otsu threshold driven binary segmentation and connected component analysis based on the real-time desktop image, and the selection of candidate points from each connected component, include:

[0025] S31, crop the real-time desktop image according to a preset cropping ratio to obtain a core region image containing software icons;

[0026] S32. Based on the Otsu thresholding method, the core region image is converted into a binary mask by adaptively selecting the optimal threshold.

[0027] S33. Perform connected component analysis based on the binary mask to obtain multiple candidate regions;

[0028] S34. For each candidate region, extract the markers with strong centrality and use them as candidate points.

[0029] Furthermore, in step S4, the benchmark difference verification and deviation calibration are performed on each candidate point based on the comparison file to obtain the target candidate point set:

[0030] S41. For each candidate point, select the target reference point with the smallest spatial position difference from the candidate point from the comparison file;

[0031] S42. When the spatial difference between the current candidate point and its matching target reference point is less than a preset threshold, replace the current candidate point with the target reference point.

[0032] S43. After filtering out the valid prompts that have been replaced or do not need to be replaced, the target prompt set is obtained.

[0033] Furthermore, in step S6, the comprehensive similarity of the target mask relative to each icon image in the standardized image library is calculated through the following steps:

[0034] S61. Based on the histogram bus distance metric, determine the color histogram similarity between the icon image and the target mask;

[0035] S62. Based on the similarity of image feature elements, determine the overall structural similarity between the icon image and the target mask;

[0036] S63. Based on the color histogram similarity and the overall structural similarity, a weighted fusion is performed to obtain the comprehensive similarity between the target mask and the icon image.

[0037] Furthermore, in step S62, determining the overall structural similarity between the icon image and the target mask based on the similarity of image feature elements includes:

[0038] S621. Under a local Gaussian window, calculate the local mean brightness, local contrast variance, and covariance of the icon image and the target mask, respectively.

[0039] S622, based on the local brightness mean, local contrast variance and covariance of the two, by combining the comprehensive operation of brightness feature correlation, contrast feature difference and structure feature coupling, the local structure similarity of the icon image and the target mask is calculated;

[0040] S623, based on the obtained local structure similarity, the global average calculation is carried out to obtain the overall structure similarity of the icon image and the target mask.

[0041] Further, in step S622, the local structure similarity of the icon image and the target mask is calculated based on the following formula:

[0042] ;

[0043] Wherein, and respectively represent the local brightness mean of the target mask and the icon image under the local Gaussian window; and respectively represent the local contrast variance of the target mask and the icon image under the local Gaussian window; Covariance of the target mask and the icon image under the local Gaussian window is represented; and represent the stability constant for avoiding numerical instability when the denominator is close to zero.

[0044] Secondly, the application discloses a visual large model feature detection equipment real-time identification system based on prior prompt, the system comprises a reference point selection module, an icon image standardization processing module, a candidate point selection module, a candidate point calibration module, a SAM segmentation module and an icon recognition determination module, wherein:

[0045] The reference point selection module is used for acquiring a historical desktop image of the feature detection equipment software, selecting reference points for assisting icon image segmentation based on the historical desktop image, and obtaining a comparison file recording reference point position information;

[0046] The icon image standardization processing module performs standardization storage processing on each icon image associated with the reference point matching, and obtains a standardized image library;

[0047] The candidate point selection module is used for acquiring a real-time desktop image of the feature detection equipment software, performing Otsu threshold driven binary segmentation and connected domain analysis based on the real-time desktop image, and selecting candidate points from each connected domain;

[0048] The candidate point calibration module is configured to perform reference point difference verification and deviation calibration on each candidate point based on the comparison file, to obtain a target candidate point set;

[0049] The SAM segmentation module is configured to input the target candidate point set as a point prompt, drive the visual large model SAM to perform zero-shot segmentation, and obtain a corresponding target mask set;

[0050] The icon recognition and determination module is configured to calculate the comprehensive similarity of each target mask in the set with respect to each icon image in the standardized image library, and determine the icon category corresponding to the maximum comprehensive similarity as the real-time recognition result of the special inspection equipment software.

[0051] In a third aspect, the present application discloses a readable storage medium, which comprises a visual large model special inspection equipment real-time identification method program based on prior prompt. When the visual large model special inspection equipment real-time identification method program based on prior prompt is executed by a processor, the steps of the method according to any one of the preceding aspects are implemented.

[0052] The present application has the following beneficial effects: by obtaining historical desktop images and selecting reference points to form a comparison file, a reliable reference standard is provided for accurate segmentation and recognition of subsequent icon images; by standardizing the storage of associated icons and establishing a standardized image library, subsequent comparison is facilitated. After obtaining a real-time desktop image, candidate points are selected by using Otsu threshold-driven binary segmentation and connected component analysis, which can effectively extract potential target points. Reference point difference verification and deviation calibration based on the comparison file can improve the accuracy of candidate points and obtain a more reliable target candidate point set. Inputting the target candidate point set as a point prompt into the visual large model SAM for zero-shot segmentation can efficiently obtain a target mask set. Finally, the comprehensive similarity is calculated and the real-time recognition result is determined, which realizes accurate and efficient recognition of the icons of the special inspection equipment software, and helps to improve the operation convenience and management efficiency of the special inspection equipment software. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 FIG. 1 is a flowchart of a visual large model special inspection equipment real-time identification method based on prior prompt disclosed by the present application;

[0054] Figure 2 FIG. 2 is a structural diagram of a visual large model special inspection equipment real-time identification system disclosed by the present application;

[0055] Figure 3 FIG. 3 is a structural diagram of a readable storage medium disclosed by the present application. DETAILED DESCRIPTION

[0056] The principles and characteristics of the present application are described below in conjunction with the accompanying drawings, and the examples are used to explain the present application and are not intended to limit the scope of the present application.

[0057] As shown in Figure 1 The present application discloses a visual large model special inspection equipment real-time identification method based on prior prompt, which comprises the following steps:

[0058] Step S1, obtain the historical desktop image of the special inspection equipment software, select the reference point for assisting icon image segmentation based on the historical desktop image, and obtain the comparison file recording the reference point position information.

[0059] Step S2, perform standardized storage processing on each icon image associated according to the reference point matching, and obtain a standardized image library.

[0060] Step S3, obtain the real-time desktop image of the special inspection equipment software, perform Otsu threshold driven binary segmentation and connected component analysis based on the real-time desktop image, and select candidate points from each connected component.

[0061] Step S4, perform reference point difference verification and deviation calibration on each candidate point based on the comparison file, and obtain a target candidate point set.

[0062] Step S5, input the target candidate point set as a point prompt, drive the visual large model SAM to perform zero sample segmentation, and obtain a corresponding target mask set.

[0063] Step S6, calculate the comprehensive similarity of each target mask in the set with respect to each icon image in the standardized image library, and determine the icon category label corresponding to the maximum comprehensive similarity as the real-time identification result of the special inspection equipment software.

[0064] As can be seen from the above, the visual large model special inspection equipment real-time identification method disclosed by the present application provides a reliable reference basis for the accurate segmentation and identification of subsequent icon images by obtaining historical desktop images and selecting reference points to form a comparison file. The associated icons are standardized and stored to establish a standardized image library for subsequent comparison. After obtaining the real-time desktop image, the candidate points are selected by using Otsu threshold driven binary segmentation and connected component analysis, which can effectively extract potential target points. The reference point difference verification and deviation calibration based on the comparison file can improve the accuracy of the candidate points and obtain a more reliable target candidate point set. Inputting the target candidate point set as a point prompt into the visual large model SAM for zero sample segmentation can efficiently obtain a target mask set. Finally, the comprehensive similarity is calculated and the real-time identification result is determined to achieve accurate and efficient identification of the special inspection equipment software icons, which helps to improve the operation convenience and management efficiency of the special inspection equipment software.

[0065] In one of the embodiments, in step S1, the reference point selection for assisting icon image segmentation based on the historical desktop image is performed to obtain a comparison file recording reference point position information, which includes:

[0066] In step S11, an initial prompt point is selected from the historical desktop image, a visual large model SAM is driven to perform single-point prompt segmentation according to the initial prompt point to obtain an initial segmentation mask, and the initial prompt point and the initial segmentation mask are synchronously visualized.

[0067] Specifically, an operator selects a representative position as an initial prompt point on the image based on intuitive observation of icon distribution in the historical desktop image. The initial prompt point should be as close as possible to the expected icon center position to improve the accuracy of subsequent segmentation. Then, the coordinate information of the initial prompt point is input into the visual large model SAM, and the SAM performs single-point prompt segmentation on the historical desktop image according to its built-in powerful segmentation algorithm. Finally, the initial prompt point is displayed on the historical desktop image with a clear marker (such as a point of a specific color), and the initial segmentation mask is displayed in a semi-transparent mask or other easily recognizable form in the corresponding area of the image, realizing synchronous visualization of the two and facilitating the operator to intuitively see the relationship between the prompt point and the segmentation result, thereby providing a clear visual reference for subsequent interactive adjustment.

[0068] In step S12, the position of the initial prompt point is manually adjusted in an interactive manner during the display process to form a new interactive prompt point. The visual large model SAM generates a new segmentation mask based on the new interactive prompt point on the basis of the initial segmentation mask, and iteratively updates until the interactive prompt point is located at the center position of the segmentation mask, and the prompt point coordinates at this time are recorded.

[0069] Specifically, in the visualization interface of the initial prompt point and the initial segmentation mask, the operator can flexibly move the position of the initial prompt point through interactive operations such as mouse clicking. After each position adjustment, the point becomes a new interactive prompt point. The visual large model SAM receives the coordinate information of the new interactive prompt point in real time, and quickly generates a new segmentation mask with the new interactive prompt point as a new guide point. The operator can judge again whether the interactive prompt point is located at the center position of the segmentation mask according to the new visualization result. If not, the interactive adjustment is continued, and the above process is repeated until the interactive prompt point is located at the center position of the segmentation mask.

[0070] In step S13, the recorded prompt point coordinates that meet the condition of being located at the center position of the segmentation mask are arranged into a point set to ensure that the point set can completely cover all icon positions in the left half of the image, and finally a comparison file recording reference point position information is formed.

[0071] Specifically, the application first collects and organizes the recorded prompt point coordinates that meet the condition of being located in the center position of the segmentation mask, integrates these scattered coordinate data into an ordered point set. Then, through real-time visualization technology, the point set is displayed on the corresponding position of the historical desktop image, and the operator can intuitively observe the coverage of the point set on the icon position in the left half of the image. Finally, the organized point set is saved in a specific data format (such as a table form containing coordinate information), forming a comparison file of recorded reference point position information, providing reliable reference data for subsequent special inspection equipment software icon recognition and other operations.

[0072] In one embodiment, in step S2, the standardized storage processing of each icon image associated with the reference point matching is performed to obtain a standardized image library, including:

[0073] Step S21, converting each icon image associated with the reference point matching into a standard image storage format to obtain a standard format image.

[0074] Specifically, the application converts all images into a standard RGB color format, and saves the format uniformly as lossless PNG to ensure pixel consistency.

[0075] Step S22, scaling each standard format image to a preset image size, and using edge padding method for center alignment operation during scaling to obtain the corresponding normalized image.

[0076] Specifically, considering that the sizes of the obtained standard format images may not be the same, in order to ensure accurate calculation of similarity during subsequent matching, the application uniformly scales each standard format image to a fixed size (the image size is not fixed and can be adjusted according to detection accuracy requirements or hardware performance).

[0077] In addition, in order to preserve the original shape and proportion of the image, during scaling, the application also uses edge padding method for center alignment operation (i.e. after scaling the image, the blank area that may appear on the edge of the image is filled according to the position relationship of the original edge of the icon, so that the icon is always in the center position of the image), to avoid icon deformation caused by stretching.

[0078] In one embodiment, the application uses bilinear interpolation method (a commonly used image interpolation algorithm in image scaling, rotation and other geometric transformations) to scale each standard format image to a preset image size, to ensure that the image remains clear in detail when scaled.

[0079] Step S23, integrating and storing each normalized image for management to obtain a standardized image library.

[0080] Specifically, this application utilizes a database management system to integrate, store, and manage normalized images. Through database query and update functions, it achieves efficient management of the standardized image library, ensuring the security, integrity, and consistency of image data.

[0081] In one embodiment, step S3, which involves performing Otsu threshold-driven binarization segmentation and connected component analysis based on the real-time desktop image, and selecting candidate points from each connected component, includes:

[0082] Step S31: Crop the real-time desktop image according to a preset cropping ratio to obtain a core area image containing software icons.

[0083] Specifically, this application first obtains the width W and height H of the real-time desktop image; then, based on... Based on the area ratio, the width of the cutting area is determined as... And the height is consistent with H; finally, array slicing is used to extract the core area image containing the software icon according to the following formula: ,in, I represents the cropped core area image, and I represents the real-time desktop image.

[0084] Step S32: Based on the Otsu thresholding method, the core region image is converted into a binary mask by adaptively selecting the optimal threshold.

[0085] Specifically, to automatically obtain candidate points from the cropped area, this application employs the Otsu thresholding method for adaptive binarization. This method automatically selects the optimal threshold (a grayscale value that best separates the foreground and background) by maximizing the inter-class difference between the foreground and background, thereby avoiding the instability caused by manually setting the threshold. After obtaining the optimal threshold... Then, the core region image Convert to a binary mask in the following form. : .

[0086] It should be noted that for core region images Each pixel in This application will compare its grayscale value with the optimal threshold. Compare; if its grayscale value is greater than Then in the corresponding binary mask The value at this position is assigned to 1; if its grayscale value is less than or equal to... Then in The value at this position is assigned to 0, thus completing the conversion from the core region image to a binary mask.

[0087] Step S33, based on the binary mask, a connected component analysis is performed to obtain a plurality of candidate regions.

[0088] Specifically, the main purpose of the connected component analysis is to gather the pixels having a connected relationship in the binary mask together to form independent regions. In actual operation, each pixel in the binary mask is traversed, and the connectedness between the pixels is determined according to a preset connected rule, and the mutually connected pixels are combined into a connected component. Each connected component formed in this way represents a region having similar features, and these connected components are the plurality of candidate regions obtained.

[0089] Step S34, for each candidate region, a landmark point having strong centrality is extracted therefrom as a candidate point.

[0090] Specifically, for each candidate region, the present application converts it into a distance image through distance transformation, wherein each pixel value of the distance image represents the shortest distance from the point to the background pixel. In the obtained distance image, by traversing all the pixel points in the distance image, the pixel point having the largest distance value is found, which is taken as the landmark point having strong centrality, and is taken as the candidate point.

[0091] In one embodiment, in step S4, based on the contrast file, the reference point difference verification and deviation calibration are performed on each candidate point to obtain a target candidate point set:

[0092] Step S41, for each candidate point, a target reference point having the smallest spatial position difference with the candidate point is filtered out from the contrast file.

[0093] Specifically, let the reference point set be , and the candidate point set be . For each candidate point , the present application calculates the Euclidean distance of the candidate point relative to each reference point according to the nearest neighbor matching principle by traversing the reference point set . Then, the reference point having the smallest Euclidean distance with the candidate point is found by using the formula , and the reference point is the target reference point having the smallest spatial position difference with the candidate point .

[0094] Step S42, when the spatial position difference value between the current candidate point and the target reference point matched therewith is smaller than a preset threshold, the current candidate point is replaced based on the target reference point.

[0095] Specifically, the preset threshold is a fixed value set in advance according to the requirement of the point position accuracy in actual application. When the Euclidean distance (i.e. the spatial position difference value) between the calculated current candidate point and the target reference point matched therewith is less than the preset threshold, it indicates that the spatial positions of the two are very close, and the target reference point can be considered to better represent the position within the accuracy allowable range. At this time, the application will replace the coordinate information of the current candidate point with the coordinate information of the target reference point to improve the accuracy of subsequent processing.

[0096] In one of the embodiments, to avoid different candidate points being mapped to the same reference point, after the target reference point is screened out, a deduplication process is still needed to ensure the accuracy and uniqueness of the final screening result.

[0097] Step S43, after the effective prompt points that have been replaced or do not need to be replaced are screened out, the target prompt point set is obtained.

[0098] In one of the embodiments, in step S6, the comprehensive similarity of the target mask relative to each icon image in the standardized image library is calculated by the following steps:

[0099] Step S61, based on the histogram chi-square distance measurement, the color histogram similarity of the icon image and the target mask is determined.

[0100] Specifically, after obtaining the segmentation region histogram corresponding to the target mask , and the histogram of the icon image , the application will calculate the chi-square distance of the two (i.e. the sum of the square roots of the product of the bin values of the two histograms, wherein represents the value of the segmentation region, icon image histogram at the i-th bin). Then, the color histogram similarity is calculated according to , .

[0101] Step S62, based on the similarity of the image feature elements, the overall structural similarity of the icon image and the target mask is determined.

[0102] Specifically, the application will take the maximum mask bounding box of the segmentation region, and scale it to according to the scaling rules of each image in the icon image library, so as to have a consistent size and feature expression for subsequent structural similarity calculation.

[0103] Further, in order to improve the efficiency and accuracy of the structural similarity calculation, the application will also convert the icon image and the segmentation candidate region into a grayscale image and normalize it to , so as to eliminate the interference of color information and unify the data range.​

[0104] Step S63, based on the color histogram similarity and the overall structure similarity, weighted fusion is performed to obtain the comprehensive similarity of the target mask with respect to the icon image.

[0105] Specifically, the color histogram similarity and the overall structure similarity are combined in a certain proportion. Specifically, the color histogram similarity is multiplied by its corresponding weight , the structure similarity is multiplied by its corresponding weight , and the results are added to obtain the comprehensive similarity of the target mask with respect to the standardized image, so as to comprehensively consider the color and structure information.

[0106] In one embodiment, in step S62, the similarity degree based on the image feature elements is used to determine the overall structure similarity of the icon image and the target mask, including:

[0107] Step S621, under the local Gaussian window, the local

[0108] brightness mean, local contrast variance, and the covariance of the icon image and the target mask are calculated respectively.

[0109] Specifically, under the given local Gaussian window , the local brightness mean , of the icon image and the target mask is calculated according to the following formula: , It should be noted that by performing convolution operation with the Gaussian window , the image can be smoothed and the local brightness mean at each position i can be calculated. This operation helps to reduce the influence of noise and highlight the local features of the image, so as to more accurately analyze the brightness distribution of the icon image and the target mask.

[0110] Specifically, the calculation formula of the local contrast variance includes: , It should be noted that the convolution operation of the Gaussian window and the square of the image , is first performed to obtain: , The purpose of this step is to make a weighted average of the square of the image pixel value in a local area, and the weight of the weighting is determined by the Gaussian window, which makes the closer the pixel to the center pixel, the greater the weight in the calculation. Then subtract the square of the local area brightness average 、 The square of the local contrast variance is obtained. The local contrast variance reflects the degree of brightness change of the image in the local area. The larger the variance value, the greater the difference in pixel brightness in the local area, and the higher the contrast. Conversely, the smaller the variance value, the more uniform the pixel brightness in the local area, and the lower the contrast.

[0111] Specifically, the calculation formula of the covariance of the two includes: It should be noted that the covariance is used to measure the correlation of the brightness change of the icon image and the target mask in the local area. If the covariance is positive, it indicates that the brightness change trend of the two in the local area is the same, that is, when the icon image brightness increases, the target mask brightness also tends to increase. If the covariance is negative, it indicates that the brightness change trend of the two is opposite. If the covariance is zero, it indicates that there is no linear correlation between the brightness change of the two in the local area.

[0112] Step S622, based on the local brightness average, the local contrast variance and the covariance of the two, the local structure similarity of the icon image and the target mask is calculated by combining the comprehensive operation of the brightness feature correlation, the contrast feature difference and the structure feature coupling.

[0113] The specific calculation formula can be referred to in the subsequent content, which will not be described here.

[0114] Step S623, based on the obtained local structure similarity, the global average calculation is performed to obtain the overall structure similarity of the icon image and the target mask.

[0115] Specifically, after obtaining the local structure similarity of the icon image and the target mask in each local area in step S622, since these local structure similarities reflect the structure similarity degree of the icon image and the target mask in different local positions, in order to evaluate the structure similarity of the two as a whole, global average calculation is needed. Specifically, first, the total number of local areas divided by the icon image is determined, which is N. Then, the values of all local structure similarities are added, that is, the local structure similarity values calculated for each local area are added to obtain the sum S. Then, the sum S is divided by the total number of local areas N to obtain the overall structure similarity of the icon image and the target mask.

[0116] In one embodiment, in step S622, the local structure similarity of the icon image and the target mask is calculated based on the following formula:

[0117] ;

[0118] wherein, and denote the local mean intensity of the target mask and the icon image under a local Gaussian window, respectively; and denote the local contrast variance of the target mask and the icon image under a local Gaussian window, respectively; denotes the covariance of the target mask and the icon image under a local Gaussian window; and denote the stabilization constant for avoiding numerical instability when the denominator is close to zero.

[0119] It should be noted that the formula measures the local structural similarity from three key dimensions of intensity, contrast and structure. In the numerator part, the correlation degree of the local mean intensity of the target mask and the icon image is embodied, the closer the mean value, the greater the value, indicating that the similarity in the intensity dimension is higher; and the coupling relationship between the two in the structure is reflected, the greater the covariance, the stronger the correlation of the structure change. In the denominator part, the intensity dimension is normalized based on the formula , and the contrast dimension is normalized based on , so that the similarity can be measured within a reasonable range.

[0120] Please refer to Figure 2 , the visual large model feature detection equipment real-time identification system disclosed in the present application comprises a reference point selection module, an icon image standardization processing module, a candidate point selection module, a candidate point calibration module, a SAM segmentation module and an icon recognition determination module, wherein:

[0121] The reference point selection module is used for acquiring a historical desktop image of the feature detection equipment software, performing reference point selection for assisting icon image segmentation based on the historical desktop image, and obtaining a comparison file recording reference point position information.

[0122] The icon image standardization processing module performs standardized storage processing on each icon image associated according to the reference point matching, and obtains a standardized image library.

[0123] The candidate point selection module is configured to acquire a real-time desktop image of the special inspection equipment software, perform Otsu threshold value driven binary segmentation and connected domain analysis based on the real-time desktop image, and select candidate points from each connected domain.

[0124] The candidate point calibration module is configured to perform reference point difference verification and deviation calibration on each candidate point based on the comparison file, to obtain a target candidate point set.

[0125] The SAM segmentation module is configured to input the target candidate point set as a point prompt, drive a visual large model SAM to perform zero-shot segmentation, and obtain a corresponding target mask set.

[0126] The icon recognition and determination module is configured to calculate a comprehensive similarity of each target mask in the set relative to each icon image in the standardized image library, and determine an icon category corresponding to a maximum comprehensive similarity as a real-time recognition result of the special inspection equipment software.

[0127] In one of the embodiments, each of the above modules is also configured to implement a visual large model special inspection equipment real-time recognition method based on prior prompt, as described in any of the preceding method embodiments, which is not limited in the present application.

[0128] As can be seen from the above, the visual large model special inspection equipment real-time recognition system based on prior prompt disclosed in the present application provides a reliable reference for the accurate segmentation and recognition of subsequent icon images by acquiring historical desktop images and selecting reference points to form a comparison file. The standardized storage of associated icons establishes a standardized image library, which is convenient for subsequent comparison. After acquiring a real-time desktop image, candidate points are selected by using Otsu threshold value driven binary segmentation and connected domain analysis, which can effectively extract potential target points. The reference point difference verification and deviation calibration based on the comparison file can improve the accuracy of the candidate points and obtain a more reliable target candidate point set. Inputting the target candidate point set as a point prompt into a visual large model SAM for zero-shot segmentation can efficiently obtain a target mask set. Finally, the comprehensive similarity is calculated and the real-time recognition result is determined, which realizes accurate and efficient recognition of the icons of the special inspection equipment software, and helps to improve the operation convenience and management efficiency of the special inspection equipment software.

[0129] For reference Figure 3 The present application discloses a readable storage medium, wherein a visual large model special inspection equipment real-time recognition method program based on prior prompt is included in the readable storage medium, and the visual large model special inspection equipment real-time recognition method program based on prior prompt is executed by a processor to implement the steps of the method as described in any of the preceding embodiments.

[0130] From the above, the application discloses a readable storage medium, which forms a comparison file by acquiring a historical desktop image and selecting a reference point, thereby providing a reliable reference for the accurate segmentation and identification of subsequent icon images. The associated icons are standardized and stored to establish a standardized image library for subsequent comparison. After acquiring a real-time desktop image, the Otsu threshold-driven binary segmentation and connected component analysis are used to select candidate points, which can effectively extract potential target points. The reference point difference verification and deviation calibration based on the comparison file can improve the accuracy of the candidate points and obtain a more reliable target candidate point set. The point prompt is input into the visual large model SAM for zero-shot segmentation, which can efficiently obtain the target mask set. Finally, the comprehensive similarity is calculated and the real-time recognition result is determined, thereby realizing the accurate and efficient identification of the special inspection equipment software icon and helping to improve the operation convenience and management efficiency of the special inspection equipment software.

[0131] The above description is only the preferred embodiment of the application, and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A priori prompt-based visual large model feature detection device real-time identification method, characterized in that, The method comprises the following steps: S1, obtaining a historical desktop image of the special inspection equipment software, performing reference point selection for assisting icon image segmentation based on the historical desktop image, and obtaining a comparison file recording reference point position information; S2, performing standardized storage processing on each icon image associated according to the reference point matching, and obtaining a standardized image library; S3, obtaining a real-time desktop image of the special inspection equipment software, performing Otsu threshold value driven binary segmentation and connected domain analysis based on the real-time desktop image, and selecting candidate points from each connected domain; S4, performing reference point difference verification and deviation calibration on each candidate point based on the comparison file, and obtaining a target candidate point set; S5, taking the target candidate point set as a point prompt input, driving a visual large model SAM to perform zero sample segmentation, and obtaining a corresponding target mask set; S6, calculating the comprehensive similarity of each target mask in the set with respect to each icon image in the standardized image library, and determining the icon category label corresponding to the maximum comprehensive similarity as the real-time recognition result of the special inspection equipment software.

2. The method of claim 1, wherein, In step S1, the reference point selection for assisting icon image segmentation based on the historical desktop image to obtain the comparison file recording the reference point position information comprises: S11, selecting an initial prompt point from the historical desktop image, driving the visual large model SAM to perform single-point prompt segmentation according to the initial prompt point to obtain an initial segmentation mask, and synchronously visualizing the initial prompt point and the initial segmentation mask; S12, in the display process, manually adjusting the position of the initial prompt point in an interactive manner to form a new interactive prompt point, wherein the visual large model SAM generates a new segmentation mask based on the new interactive prompt point on the basis of the initial segmentation mask, and iteratively updates until the generated interactive prompt point is located at the center position of the segmentation mask, and records the prompt point coordinates at this time; S13, arranging the recorded prompt point coordinates that meet the condition of being located at the center position of the segmentation mask into a point set to ensure that the point set can completely cover all icon positions in the left half region of the image, and finally forming the comparison file recording the reference point position information.

3. The method of claim 1, wherein, In step S2, the standardized storage processing on each icon image associated according to the reference point matching to obtain the standardized image library comprises: S21, converting each icon image associated according to the reference point matching into a standard image storage format to obtain a standard format image; S22, scaling each standard format image to a preset image size, and using edge padding to perform center alignment during the scaling process to obtain a corresponding normalized image; S23, integrating and storing each normalized image to obtain the standardized image library.

4. The method of claim 1, wherein, In step S3, the Otsu threshold value driven binary segmentation and connected domain analysis based on the real-time desktop image, and the selection of candidate points from each connected domain comprise: S31, cropping the real-time desktop image according to a preset cropping ratio to obtain a core region image containing software icons; S32, convert the core region image into a binary mask based on the Otsu threshold method and an optimal threshold value selected adaptively; S33, perform connected component analysis based on the binary mask to obtain a plurality of candidate regions; S34, for each candidate region, extract a mark point with strong centrality therefrom as a candidate point.

5. The method of claim 1, wherein, In step S4, the candidate points are verified based on the reference file to obtain a target candidate point set: S41, for each candidate point, filter a target reference point with the smallest spatial position difference from the candidate point from the reference file; S42, when the spatial position difference between the current candidate point and the target reference point matched therewith is smaller than a preset threshold, replace the current candidate point with the target reference point; S43, after filtering the effective prompt points that are replaced or do not need to be replaced, obtain a target prompt point set.

6. The method of claim 1, wherein, In step S6, the comprehensive similarity of the target mask relative to each icon image in the standardized image library is calculated by the following steps: S61, determine the color histogram similarity of the icon image and the target mask based on the histogram bus distance measurement; S62, determine the overall structural similarity of the icon image and the target mask based on the similarity of the image feature elements; S63, perform weighted fusion based on the color histogram similarity and the overall structural similarity to obtain the comprehensive similarity of the target mask relative to the icon image.

7. The method of claim 6, wherein, In step S62, the overall structural similarity of the icon image and the target mask based on the similarity of the image feature elements includes: S621, calculate the local brightness mean value, local contrast variance and covariance of the icon image and the target mask under a local Gaussian window, respectively; S622, based on the local brightness mean value, local contrast variance and covariance, calculate the local structural similarity of the icon image and the target mask by combining the comprehensive operation of brightness feature correlation, contrast feature difference and structural feature coupling; S623, perform global average calculation based on the obtained local structural similarity to obtain the overall structural similarity of the icon image and the target mask. In step S622, the local structural similarity of the icon image and the target mask is calculated based on the following formula:

8. The method of claim 7, wherein, The system comprises a reference point selection module, an icon image standardization processing module, a candidate point selection module, a candidate point calibration module, a SAM segmentation module and an icon recognition and determination module, wherein: ； wherein, and denote the local mean intensity of the target mask and the icon image under a local Gaussian window, respectively; and denote the local contrast variance of the target mask and the icon image under a local Gaussian window, respectively; denotes the covariance of the target mask and the icon image under a local Gaussian window; and denote a stabilization constant used to avoid numerical instability when the denominator is close to zero.

9. A priori prompt-based visual large model special inspection equipment real-time identification system, characterized in that, The reference point selection module is used to acquire a historical desktop image of the special inspection equipment software, perform reference point selection for assisting icon image segmentation based on the historical desktop image, and obtain a reference file recording reference point position information; The icon image standardization processing module is used to perform standardized storage processing on each icon image associated based on the reference point matching and correlation, and obtain a standardized image library; The candidate point selection module is used to acquire a real-time desktop image of the special inspection equipment software, perform Otsu threshold driven binarization segmentation and connected component analysis based on the real-time desktop image, and select candidate points from each connected component; ​ The candidate point calibration module is configured to perform reference point difference verification and deviation calibration on each candidate point based on the contrast file to obtain a target candidate point set; The SAM segmentation module is configured to input the target candidate point set as a point prompt, drive the visual large model SAM to perform zero-shot segmentation, and obtain a corresponding target mask set; The icon recognition and determination module is configured to calculate the comprehensive similarity of each target mask in the set with respect to each icon image in the standardized image library, and determine the icon category corresponding to the maximum comprehensive similarity as the real-time recognition result of the special inspection equipment software.

10. A readable storage medium, characterized by, The readable storage medium comprises a visual large model special inspection equipment real-time recognition method program based on prior prompt, and the visual large model special inspection equipment real-time recognition method program based on prior prompt, when executed by the processor, implements the steps of the method in any one of claims 1 to 8.

Images