A tool rack status detection method and system based on visual recognition

Abstract

The present invention discloses a tool rack status detection method and system based on visual recognition, comprising: marking tools in an original image to obtain marking data; photographing the tool rack using a camera to obtain photo data byte[] and converting it into a Mat matrix imageSrc; performing HSV spatial processing and binarization on the Mat matrix imageSrc to obtain a Mat matrix mask of the engraved background; performing matrix cropping on the mask matrix to obtain a Mask matrix imageOut after HSV spatial processing of the identified area; calculating the number of non-zero characters in the Mask matrix imageOut to obtain a background area maskCount of the imageOutput matrix; step S6, obtaining a proportion score using an area ratio algorithm; step S7, grouping the Score values according to the tool label label; and step S8, calculating the tool rack status DetectionResult after grouping. The present invention can obtain the in-place, out-of-place, and misplaced status of tools, thereby improving the efficiency of tool borrowing and returning management and tool inventory. At the same time, the algorithm is simple and saves application costs.

Description

Technical Field

[0001] The present invention relates to a tool rack status detection method, and in particular to a tool rack status detection method and system based on visual recognition. Background Art

[0002] Currently, there are several ways to detect the status of tool racks:

[0003] Hardware sensor-based solution: Sensors are installed to obtain tool rack position information and determine the tool rack status. However, this solution requires distance sensors (such as ultrasonic sensors, laser sensors, or gravity sensors) to be installed at specific locations on the tool rack. These sensors can measure the distance between the tool and the sensor in real time and detect the tool's tilt angle or direction, thereby completing the tool rack status detection. However, the additional hardware deployment increases system complexity and cost, and is subject to limitations such as installation space and wiring.

[0004] Deep neural network-based visual recognition solutions: These utilize the powerful feature learning capabilities of deep learning models to analyze and process camera-collected images to identify rack positions. However, these solutions require massive amounts of labeled data to train the models. Data collection and labeling consume significant human resources, and training is prone to overfitting, impacting practical application effectiveness.

[0005] Solutions based on traditional visual recognition algorithms: These solutions extract image features based on traditional image processing and analysis algorithms to determine the rack status. However, these solutions are susceptible to factors such as lighting and noise. Lighting changes, angles, or other interference factors can degrade image quality, affecting the reliability of feature extraction and rack status determination. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a tool rack status detection method and system based on visual recognition that can obtain the in-place, out-of-place, and misplaced status of tools, thereby improving the efficiency of tool borrowing and returning management and tool inventory, while having a simple algorithm and saving application costs.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0008] A tool rack status detection method based on visual recognition comprises the following steps: step S1, marking the tools in the original image to obtain marking data, wherein the marking data includes a preset left margin left, a top margin top, a width width, a height height, a Score upper limit thresholdUpper, a Score lower limit thresholdLower and a tool label label; step S2, taking a photo of the tool rack through a camera, obtaining the photo data byte[] and converting it into a Mat matrix imageSrc; step S3, performing HSV spatial processing and binarization processing on the Mat matrix imageSrc to obtain a mold background Mat matrix mask; step S4, performing matrix cropping on the mask matrix to obtain a Mask matrix imageOut of the recognition area after HSV spatial processing; step S5, calculating the number of non-zeros in the Mask matrix imageOut to obtain imageOutp The background area maskCount of the ut matrix; step S6, using the area ratio algorithm, and obtaining the proportional score using the following formula: Score=maskCount / (imageOutput.width*imageOutput.height); step S7, grouping the Score scores according to the tool labels label; step S8, calculating the tool rack status DetectionResult after grouping: if the Score score is greater than or equal to the Score score upper limit thresholdUpper, the tool rack status is out of position; if the Score score is greater than or equal to the Score score lower limit thresholdLower and the Score score is less than the Score score upper limit thresholdUpper, the tool rack status is misplaced; if the Score score is less than the Score score lower limit thresholdLower, the tool rack status is in position.

[0009] Preferably, in the step S1, marking basis DetectionTargetInfo data is constructed according to the marking data.

[0010] Preferably, in step S3, upper and lower limits of HSV space processing thresholds are set, and then HSV space processing is performed on the Mat matrix imageSrc.

[0011] Preferably, in step S3, the HSV space includes hue, saturation and brightness.

[0012] Preferably, in step S4, the mask matrix is cropped according to the DetectionTargetInfo data through the marking.

[0013] Preferably, it also includes: step S9, recalculating the tool rack position status DetectionResult after the label grouping: if all the DetectionResult in the label grouping are in place, it means that the tool is in place; if one of the DetectionResult in the label grouping is misplaced, it means that the tool is misplaced; other situations indicate that the tool is out of place.

[0014] Preferably, the method further includes: step S10, printing out the parameters and calculation results.

[0015] Preferably, in step S10, the parameters and calculation results include the label label, the location status result of the label label, and the label calculation result DetectionResult in the group.

[0016] Preferably, the result DetectionResult includes the marking basis DetectionTargetInfo, Score score and Mask matrix imageOutput after HSV processing.

[0017] A tool rack status detection system based on visual recognition, the system is used to execute the above-mentioned method.

[0018] The tool rack status detection method based on visual recognition disclosed in this invention uses a camera to take a picture, performs HSV spatial processing on the picture, extracts the background features of the tool engraving, then binarizes these features and performs an algorithmic calculation with the marking data to obtain a score (Score) representing the area ratio of the background within the marking range. This score is then logically calculated by combining the score with the marking threshold to determine the tool rack status. Compared to existing technologies, this method not only detects the in-place, out-of-place, and misplaced status of tools, but also improves the efficiency of tool borrowing and returning management and tool inventory. Furthermore, the algorithm is simple, does not rely on complex hardware, and has a lower application cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The tool stand position map obtained after marking the original image;

[0020] Figure 2 The picture taken by the camera;

[0021] Figure 3 This is the image processed in HSV space;

[0022] Figure 4 The image is obtained after matrix cropping of the mask matrix;

[0023] Figure 5 This is the system interface for outputting the proportional score Score Figure 1 ;

[0024] Figure 6 This is the system interface for outputting the proportional score Score Figure 2 . DETAILED DESCRIPTION

[0025] The present invention will be described in more detail below with reference to the accompanying drawings and embodiments.

[0026] The present invention discloses a tool rack status detection method based on visual recognition, which comprises the following steps:

[0027] Step S1, such as Figure 1 As shown, the tool in the original image is marked to obtain marking data, which includes a preset left margin, a top margin, a width, a height, an upper limit of the score thresholdUpper, a lower limit of the score thresholdLower, and a tool label label;

[0028] Step S2, take a photo of the tool holder through the camera, obtain the photo data byte[] and convert it into Mat matrix imageSrc;

[0029] Step S3, see Figure 2 and Figure 3 , performing HSV spatial processing and binarization processing on the Mat matrix imageSrc to obtain the engraving background Mat matrix mask;

[0030] Step S4, see Figure 4 , performing matrix cropping on the engraving background Mat matrix mask to obtain the Mask matrix imageOut of the recognition area after HSV space processing;

[0031] Step S5, calculating the number of non-zero values in the Mask matrix imageOut to obtain the background area maskCount of the Mask matrix imageOut;

[0032] Step S6, see Figure 5 and Figure 6 , using the area ratio algorithm, use the following formula to obtain the proportion score:

[0033] Score=maskCount / (imageOutput.width*imageOutput.height);

[0034] In the above formula, imageOutput.width refers to the width of the image corresponding to the Mask matrix imageOut, and imageOutput.height refers to the height of the image corresponding to the Mask matrix imageOut;

[0035] Step S7, grouping the proportion scores according to the tool labels;

[0036] Step S8, calculate the tool rack status DetectionResult after grouping:

[0037] If the proportional score Score is greater than or equal to the Score upper limit thresholdUpper, the tool rack position status is out of position;

[0038] If the proportional score Score is greater than or equal to the Score lower limit thresholdLower and the proportional score Score is less than the Score upper limit thresholdUpper, the tool rack status is misplaced;

[0039] If the proportional score Score is less than the Score lower limit thresholdLower, the tool rack position status is in place.

[0040] In this method, a camera captures the image, performs HSV spatial processing on the image, extracts the background features of the tool engraving, and then binarizes these features. This is then combined with the marking data to calculate a score (Score) representing the area ratio of the background within the marking range. This score is then logically calculated by combining it with the marking threshold to determine the tool placement status. Compared to existing technologies, this method not only captures the status of tools in place, out of place, and misplaced, but also improves the efficiency of tool borrowing and return management and inventory. Furthermore, the algorithm is simple, does not rely on complex hardware, and has a lower application cost.

[0041] Furthermore, in step S1, the marking basis DetectionTargetInfo data is constructed according to the marking data, specifically including the preset left margin left, top margin top, width width, height height, Score upper limit thresholdUpper, Score lower limit thresholdLower and tool label label.

[0042] In step S3 of this embodiment, the upper and lower limits of the HSV space processing threshold are set, and then the Mat matrix imageSrc is processed in the HSV space. Specifically, in step S3, the HSV space includes hue, saturation and brightness.

[0043] For the identification area, in the step S4, the mask matrix is cropped according to the DetectionTargetInfo data through the marking.

[0044] As a preferred embodiment, this embodiment further includes:

[0045] Step S9, recalculate the tool rack status DetectionResult after label grouping:

[0046] If all the tool rack position status DetectionResult in the label group are in place, it means that the tool is in place;

[0047] If one of the tool rack status DetectionResult in the label group is misplaced, it means that the tool is misplaced;

[0048] Other situations indicate that the tool is out of position.

[0049] On this basis, this embodiment further includes: step S10, printing out the parameters and calculation results.

[0050] In step S10 above, the parameters and calculation results include the label, the label's position status, and the label calculation result within the group (tool position status DetectionResult). Specifically, the label calculation result (tool position status DetectionResult) includes the labeling basis DetectionTargetInfo, the score, and the mask matrix imageOutput after HSV processing.

[0051] This embodiment also proposes a tool rack status detection system based on visual recognition, which is used to execute the above-mentioned method.

[0052] A specific embodiment is provided below.

[0053] Example 1

[0054] This embodiment describes a tool status detection method based on visual recognition, applied to tool rack status detection scenarios within aviation industry tool management systems. It uses visual computing to determine the tool's in-place, out-of-place, or misplaced status. This algorithm is also applicable to scenarios with engraved dies where the background and the identified object are distinguishable in HSV space. It can be used in various warehouse rack recognition and detection scenarios.

[0055] In the specific application process, a camera is used to take a photo, and the EaeroQuealToolsStateDetection algorithm is used to calculate the status of the tool in place, out of place, and misplaced. This can be used for subsequent business applications such as tool borrowing and returning, and tool inventory. The EaeroQuealToolsStateDetection algorithm process is described as follows:

[0056] Step 1: Mark the image to obtain the tool rack information that needs to be identified, such as Figure 1 As shown;

[0057] Among them, the marking data mainly includes the tool holder status ( Figure 1 The left margin left, top margin top, width width, height height, Score upper limit thresholdUpper, Score lower limit thresholdLower and tool label of the tool outer box are used to construct the DetectionTargetInfo data for labeling;

[0058] Step 2: Use the camera to take a picture of the tool holder, obtain the original data byte[] of the picture and convert it into Mat matrix imageSrc to prepare for the subsequent processing of the algorithm;

[0059] Step 3, set the upper and lower limits of HSV space processing threshold, perform HSV space processing on Mat matrix imageSrc, and binarize it, and finally obtain the engraving background Mat matrix mask, where the engraving background refers to Figure 3 The white area in the image. The HSV space mainly includes hue, saturation, and brightness. Lighting has a significant impact on the RGB color gamut, but the algorithm uses a relatively low weight for RGB hue when processing the HSV space, so it is less affected by lighting. For the original image, please see Figure 2 , see the processing result diagram Figure 3 ;

[0060] Step 4: By marking the DetectionTargetInfo data, the mask matrix is cropped to obtain the HSV processed Mask matrix imageOut of the recognition area, as shown in the following example: Figure 4 ;

[0061] Step 5: Since the binarization process was performed in step 3, the data in the current imageOut matrix is only 0 and 1. Then calculate the number of non-zero values in the imageOutput matrix to obtain the background area maskCount of the imageOutput matrix.

[0062] Step 6, such as Figure 5 As shown, the proportion score Score is obtained by the area ratio algorithm. Specifically, the proportion score Score is obtained by using maskCount / (imageOutput.width*imageOutput.height);

[0063] Step 7: Group the score values by label to calculate the subsequent shelf status;

[0064] Step 8: Calculate the tool rack status DetectionResult after label grouping:

[0065] a. If the Score value is greater than or equal to thresholdUpper, the tool rack status is off;

[0066] b. If the Score value is greater than or equal to thresholdLower and the Score value is less than thresholdUpper, the tool rack status is misplaced;

[0067] c. In other cases, the tool is in place.

[0068] Step 9: Secondarily calculate the tool rack status DetectionResult after label grouping:

[0069] a. If all DetectionResults in the label group are in place, the tool is in place;

[0070] b. If one of the DetectionResult in the label branch is misplaced, the tool is misplaced;

[0071] c. Other situations are when the tool is out of place.

[0072] Step 10: Output the results: Print out the relevant parameters and calculation results, including the label, the label's rack status, and the label calculation result DetectionResult within the group. DetectionResult includes the labeling basis DetectionTargetInfo, Score, and the Mask matrix imageOutput after HSV processing.

[0073] Compared with the existing technology, the present invention can take pictures of the tool holder through a camera based on the above method, and use an algorithm with efficient noise reduction capability and the ability to accurately process marking data to achieve accurate and reliable tool holder status detection, which better meets application requirements.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements or improvements made within the technical scope of the present invention should be included in the scope of protection of the present invention.

Claims

1. A tool rack status detection method based on visual recognition, characterized in that: The steps include: Step S1: Mark the tool in the original image to obtain marking data, where the marking data includes a preset left margin, a top margin, a width, a height, an upper limit of the score thresholdUpper, a lower limit of the score thresholdLower, and a tool label label; Step S2, take a photo of the tool holder through the camera, obtain the photo data byte[] and convert it into Mat matrix imageSrc; Step S3, performing HSV spatial processing and binarization processing on the Mat matrix imageSrc to obtain the engraving background Mat matrix mask; Step S4, performing matrix cropping on the engraving background Mat matrix mask to obtain the Mask matrix imageOut of the recognition area after HSV space processing; Step S5, calculating the number of non-zero values in the Mask matrix imageOut to obtain the background area maskCount of the Mask matrix imageOut; Step S6: Use the area ratio algorithm to obtain the proportion score using the following formula: Score=maskCount / (imageOutput.width*imageOutput.height); Step S7, grouping the proportion scores according to the tool labels; Step S8, calculate the tool rack status DetectionResult after grouping: If the proportional score Score is greater than or equal to the Score upper limit thresholdUpper, the tool rack position status is out of position; If the proportional score Score is greater than or equal to the Score lower limit thresholdLower and the proportional score Score is less than the Score upper limit thresholdUpper, the tool rack status is misplaced; If the proportional score Score is less than the Score lower limit thresholdLower, the tool rack position status is in place.

2. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: In the step S1, marking basis DetectionTargetInfo data is constructed according to the marking data.

3. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: In the step S3, the upper and lower limits of the HSV space processing threshold are set, and then the HSV space processing is performed on the Mat matrix imageSrc.

4. The tool rack status detection method based on visual recognition according to claim 3, characterized in that: In step S3, the HSV space includes hue, saturation and brightness.

5. The tool rack status detection method based on visual recognition according to claim 2, characterized in that: In the step S4, the mask matrix is cropped according to the DetectionTargetInfo data through the marking.

6. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: Also includes: Step S9, recalculate the tool rack status DetectionResult after label grouping: If all the tool rack position status DetectionResult in the label group are in place, it means that the tool is in place; If one of the tool rack status DetectionResult in the label group is misplaced, it means that the tool is misplaced; Other situations indicate that the tool is out of position.

7. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: Also includes: Step S10: Print out the parameters and calculation results.

8. The tool rack status detection method based on visual recognition according to claim 7, characterized in that: In the step S10, the parameters and calculation results include the label label, the position status result of the label label, and the tool position status DetectionResult.

9. The tool rack status detection method based on visual recognition according to claim 8, characterized in that: The tool holder position status DetectionResult includes the marking basis DetectionTargetInfo, Score score and HSV processed Mask matrix imageOutput.

Images