Tool rack position state detection method and system based on visual identification

Abstract

The invention discloses a tool rack position state detection method and system based on visual identification, and the method comprises the steps: marking a tool in an original picture, and obtaining the marking data; the method comprises the following steps: photographing a tool rack through a camera to obtain photo data byte [], and converting the photo data byte [] into a Mat matrix imageSrc; hSV space processing and binarization processing are carried out on the Mat matrix imageSrc, and a die sinking background Mat matrix mask is obtained; matrix cutting is carried out on the mask matrix, and a Mask matrix imageOut after HSV space processing of the identification area is obtained; the number of non-zero in the imageOut of the Mask matrix is calculated, and the background area maskCount of the imageOut matrix is obtained; s6, acquiring a proportion score by adopting an area proportion algorithm; step S7, the Score scores are grouped according to the tool labels (labels); and S8, the state Desection Result of the tool rack positions after grouping is calculated. According to the method, the in-place, off-place and misplacement states of the tool can be obtained, the tool borrowing and returning management and tool checking efficiency can be improved, meanwhile, the algorithm is simple, and the application cost is saved.

Description

Technical Field

[0001] The present invention relates to a tool rack status detection method, and in particular to a tool rack status detection method and system based on visual recognition. Background Art

[0002] At present, there are several ways to detect the status of tool racks:

[0003] Solution based on hardware sensors: Install sensors to obtain tool rack information and then determine the tool rack status. However, this solution requires installing distance sensors (such as ultrasonic sensors or laser sensors / gravity sensors) at specific locations on the tool rack. These sensors can measure the distance between the tool and the sensor / detect the tilt angle or direction of the tool in real time, thereby completing the detection of the tool rack status. However, due to the additional deployment of hardware, the system complexity and cost increase, and is limited by installation space and wiring.

[0004] Visual recognition solution based on deep neural network: Utilize the powerful feature learning ability of deep learning model to analyze and process the images collected by the camera to identify the rack status. This solution requires a large amount of labeled data to train the model. Data collection and labeling consume a lot of manpower and other resources, and training is prone to overfitting, which affects the actual application effect.

[0005] Based on traditional visual recognition algorithm solution: Based on traditional image processing and analysis algorithms, image features are extracted to determine the rack status. However, such solutions are easily affected by factors such as lighting and noise. Lighting changes, different angles or other interference factors will reduce image quality and affect the reliability of feature extraction and rack status judgment. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a tool rack status detection method and system based on visual recognition, which can obtain the in-place, out-of-place and misplaced status of tools, improve the efficiency of tool borrowing and returning management and tool inventory, and at the same time has a simple algorithm and saves application costs.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0008] A tool rack state detection method based on visual recognition comprises the following steps: step S1, marking the tools in the original image to obtain marking data, wherein the marking data comprises a preset left margin left, a top margin top, a width width, a height height, a Score upper limit thresholdUpper, a Score lower limit thresholdLower and a tool label label; step S2, taking a photo of the tool rack through a camera, obtaining the photo data byte[] and converting it into a Mat matrix imageSrc; step S3, performing HSV spatial processing and binarization processing on the Mat matrix imageSrc to obtain a mold engraving background Mat matrix mask; step S4, performing matrix cropping on the mask matrix to obtain a Mask matrix imageOut of the recognition area after the HSV spatial processing; step S5, calculating the number of non-zeros in the Mask matrix imageOut to obtain imageOutp The background area maskCount of the ut matrix; step S6, using the area ratio algorithm, and using the following formula to obtain the proportion score: Score=maskCount / (imageOutput.width*imageOutput.height); step S7, grouping the Score scores according to the tool labels label; step S8, calculating the tool rack status DetectionResult after grouping: if the Score score is greater than or equal to the Score score upper limit thresholdUpper, the tool rack status is out of position; if the Score score is greater than or equal to the Score score lower limit thresholdLower and the Score score is less than the Score score upper limit thresholdUpper, the tool rack status is misplaced; if the Score score is less than the Score score lower limit thresholdLower, the tool rack status is in position.

[0009] Preferably, in the step S1, marking basis DetectionTargetInfo data is constructed according to the marking data.

[0010] Preferably, in step S3, upper and lower limits of HSV space processing thresholds are set, and then HSV space processing is performed on the Mat matrix imageSrc.

[0011] Preferably, in step S3, the HSV space includes hue, saturation and brightness.

[0012] Preferably, in the step S4, the mask matrix is ​​cropped according to the DetectionTargetInfo data through the marking.

[0013] Preferably, it also includes: step S9, recalculating the tool rack position status DetectionResult after the label grouping: if all the DetectionResult in the label grouping are in place, it means that the tool is in place; if one of the DetectionResult in the label grouping is misplaced, it means that the tool is misplaced; other situations indicate that the tool is out of place.

[0014] Preferably, the method further comprises: step S10, printing out the parameters and calculation results.

[0015] Preferably, in step S10, the parameters and calculation results include the label label, the position status result of the label label and the label calculation result DetectionResult in the group.

[0016] Preferably, the result DetectionResult includes the labeled basis DetectionTargetInfo, Score score and Mask matrix imageOutput after HSV processing.

[0017] A tool rack status detection system based on visual recognition, the system is used to execute the above method.

[0018] In the tool rack status detection method based on visual recognition disclosed in the present invention, a camera is used to take a picture, and the picture is processed by HSV space to extract the background features of the tool engraving, and then the background features are binarized, and then the algorithm is calculated with the marking data to obtain the area ratio score of the background within the marking range, and then the Score is logically calculated with the marking threshold to obtain the tool rack status. Compared with the prior art, the present invention can not only obtain the status of the tool in place, out of place, and misplaced, but also improve the efficiency of tool borrowing and returning management and tool inventory. At the same time, the algorithm of the present invention is simple, does not rely on complex hardware, and has a lower application cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The tool rack position map obtained after marking the original image;

[0020] Figure 2 The picture taken by the camera;

[0021] Figure 3 This is the image processed in HSV space;

[0022] Figure 4 The image is obtained after matrix cropping of the mask matrix;

[0023] Figure 5 The system interface for outputting the proportional score Score Figure 1 ;

[0024] Figure 6 The system interface for outputting the proportional score Score Figure 2 . DETAILED DESCRIPTION

[0025] The present invention is described in more detail below with reference to the accompanying drawings and embodiments.

[0026] The present invention discloses a tool rack status detection method based on visual recognition, which comprises the following steps:

[0027] Step S1, such as Figure 1 As shown, the tool in the original image is marked to obtain marking data, wherein the marking data includes a preset left margin left, a top margin top, a width width, a height height, a Score upper limit thresholdUpper, a Score lower limit thresholdLower, and a tool label label;

[0028] Step S2, taking a photo of the tool stand through a camera, obtaining the photo data byte[] and converting it into a Mat matrix imageSrc;

[0029] Step S3, see Figure 2 and Figure 3 , performing HSV space processing and binarization processing on the Mat matrix imageSrc to obtain the engraving background Mat matrix mask;

[0030] Step S4, see Figure 4 , performing matrix cropping on the engraving background Mat matrix mask to obtain the Mask matrix imageOut of the recognition area after HSV space processing;

[0031] Step S5, calculating the number of non-zero values ​​in the Mask matrix imageOut to obtain the background area maskCount of the Mask matrix imageOut;

[0032] Step S6, see Figure 5 and Figure 6 , using the area ratio algorithm, use the following formula to obtain the proportion score:

[0033] Score=maskCount / (imageOutput.width*imageOutput.height);

[0034] In the above formula, imageOutput.width refers to the width of the image corresponding to the Mask matrix imageOut, and imageOutput.height refers to the height of the image corresponding to the Mask matrix imageOut;

[0035] Step S7, grouping the proportion scores Score according to the tool labels label;

[0036] Step S8, calculate the tool rack status DetectionResult after grouping:

[0037] If the proportion score Score is greater than or equal to the Score upper limit thresholdUpper, the tool rack position status is out of position;

[0038] If the proportion score Score is greater than or equal to the Score lower limit thresholdLower and the proportion score Score is less than the Score upper limit thresholdUpper, the tool rack status is misplaced;

[0039] If the proportion score Score is less than the Score lower limit thresholdLower, the tool rack position status is in place.

[0040] In the above method, a camera is used to take a picture, and the picture is processed by HSV space to extract the background features of the tool engraving, and then the background features are binarized, and then the algorithm is calculated with the marking data to obtain the area ratio score of the background within the marking range, and then the Score is logically calculated with the marking threshold to obtain the tool rack status. Compared with the prior art, the present invention can not only obtain the status of the tool in place, out of place, and misplaced, but also improve the efficiency of tool borrowing and returning management and tool inventory. At the same time, the algorithm of the present invention is simple, does not rely on complex hardware, and has a lower application cost.

[0041] Furthermore, in the step S1, the marking basis DetectionTargetInfo data is constructed according to the marking data, which specifically includes a preset left margin left, a top margin top, a width width, a height height, a Score upper limit thresholdUpper, a Score lower limit thresholdLower, and a tool label label.

[0042] In step S3 of this embodiment, the upper and lower limits of the HSV space processing threshold are set, and then the Mat matrix imageSrc is processed in the HSV space. Specifically, in step S3, the HSV space includes hue, saturation and brightness.

[0043] For the identification area, in the step S4, the mask matrix is ​​cropped according to the DetectionTargetInfo data through the marking.

[0044] As a preferred embodiment, this embodiment also includes:

[0045] Step S9, recalculate the tool rack status DetectionResult after label grouping:

[0046] If all the tool rack position status DetectionResult in the label group are in place, it means that the tool is in place;

[0047] If one of the tool rack status DetectionResult in the label group is misplaced, it means that the tool is misplaced;

[0048] Other situations indicate that the tool is out of position.

[0049] On this basis, this embodiment further includes: Step S10, printing out the parameters and calculation results.

[0050] In the above step S10, the parameters and calculation results include the label label, the position status result of the label label, and the label calculation result in the group (tool position status DetectionResult). Specifically, the label calculation result (tool position status DetectionResult) includes the labeling basis DetectionTargetInfo, Score score, and Mask matrix imageOutput after HSV processing.

[0051] This embodiment also proposes a tool rack status detection system based on visual recognition, which is used to execute the above-mentioned method.

[0052] A specific embodiment is provided below.

[0053] Embodiment 1

[0054] This embodiment is a tool status detection method based on visual recognition, which is applied to the detection scenario of tool rack status in the tool management system of the aviation industry. The tool's in-place, out-of-place, and misplaced status can be obtained through visual calculation. The algorithm is also applicable to the scenario with engraving and the background of the engraving and the identified object can be distinguished in HSV space. It can be used in the business of various storage rack recognition and detection scenarios.

[0055] In the specific application process, the camera takes photos to obtain photos, and the EaeroQuealToolsStateDetection algorithm is used to calculate the photos to obtain the status of the tools in place, out of place, and misplaced, which can be used for subsequent business applications such as tool borrowing and returning, tool inventory, etc. Among them, the EaeroQuealToolsStateDetection algorithm process is described as follows:

[0056] Step 1: Mark the image to obtain the tool rack information that needs to be identified, such as Figure 1 As shown;

[0057] Among them, the marking data mainly includes the tool rack status ( Figure 1 The left margin left, top margin top, width width, height height, Score upper limit thresholdUpper, Score lower limit thresholdLower and tool label label of the tool outer box in the middle are used to construct the DetectionTargetInfo data based on which the labeling is performed;

[0058] Step 2: Use the camera to take a photo of the tool stand, obtain the original photo data byte[] and convert it into Mat matrix imageSrc to prepare for the subsequent processing of the algorithm;

[0059] Step 3, set the upper and lower limits of the HSV space processing threshold, perform HSV space processing on the Mat matrix imageSrc, and binarize it, and finally obtain the engraving background Mat matrix mask, where the engraving background refers to Figure 3 The white area in the HSV space mainly includes hue, saturation, and brightness. The main influence of light on the RGB color gamut is obvious, and the algorithm has a relatively low weight for RGB hue when processing in the HSV space, so it is less affected by light; please refer to the original image Figure 2 , see the processing result diagram Figure 3 ;

[0060] Step 4: By marking the DetectionTargetInfo data, the mask matrix is ​​cropped to obtain the HSV processed Mask matrix imageOut of the recognition area, as shown in Figure 4 ;

[0061] Step 5: Since the binarization process was performed in step 3, the data in the current imageOut matrix only contains 0 and 1. Then, the number of non-zero values ​​in the imageOutput matrix is ​​calculated to obtain the background area maskCount of the imageOutput matrix.

[0062] Step 6, such as Figure 5 As shown, the proportion score Score is obtained by the area ratio algorithm. Specifically, the proportion score Score is obtained by using maskCount / (imageOutput.width*imageOutput.height);

[0063] Step 7: Group the score values ​​according to the label label to calculate the subsequent shelf status;

[0064] Step 8: Calculate the tool rack status DetectionResult after label grouping:

[0065] a. If the Score value is greater than or equal to thresholdUpper, the tool rack position status is off;

[0066] b. If the Score value is greater than or equal to thresholdLower and the Score value is less than thresholdUpper, the tool rack status is misplaced;

[0067] c. In other cases, the tool is in place.

[0068] Step 9, recalculate the tool rack status DetectionResult after label grouping:

[0069] a. If all DetectionResult in the label group are in place, the tool is in place;

[0070] b. If one of the DetectionResult in the label branch is misplaced, the tool is misplaced;

[0071] c. In other cases, the tool is out of place.

[0072] Step 10, output results: print out the relevant parameters and calculation results, including the label label, the label label's rack status result, and the label calculation result DetectionResult in the group. DetectionResult includes the labeling basis DetectionTargetInfo, Score, and the Mask matrix imageOutput after HSV processing.

[0073] Compared with the prior art, the present invention can take pictures of the tool holder through a camera based on the above method, and use an algorithm with high-efficiency noise reduction capability and accurate processing of marking data to achieve accurate and reliable tool holder status detection, which better meets application requirements.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the technical scope of the present invention should be included in the scope of protection of the present invention.

Claims

1. A tool rack status detection method based on visual recognition, characterized in that: The steps include: Step S1, marking the tools in the original image to obtain marking data, wherein the marking data includes a preset left margin left, a top margin top, a width width, a height height, a Score upper limit thresholdUpper, a Score lower limit thresholdLower, and a tool label label; Step S2, taking a photo of the tool stand through a camera, obtaining the photo data byte[] and converting it into a Mat matrix imageSrc; Step S3, performing HSV space processing and binarization processing on the Mat matrix imageSrc to obtain a mold engraving background Mat matrix mask; Step S4, performing matrix cropping on the engraving background Mat matrix mask to obtain the Mask matrix imageOut of the recognition area after HSV space processing; Step S5, calculating the number of non-zero values ​​in the Mask matrix imageOut to obtain the background area maskCount of the Mask matrix imageOut; Step S6, using the area ratio algorithm, obtain the proportion score Score using the following formula: Score=maskCount / (imageOutput.width*imageOutput.height); Step S7, grouping the proportion scores Score according to the tool labels label; Step S8, calculate the tool rack status DetectionResult after grouping: If the proportion score Score is greater than or equal to the Score upper limit thresholdUpper, the tool rack position status is out of position; If the proportion score Score is greater than or equal to the Score lower limit thresholdLower and the proportion score Score is less than the Score upper limit thresholdUpper, the tool rack status is misplaced; If the proportion score Score is less than the Score lower limit thresholdLower, the tool rack position status is in place.

2. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: In the step S1, the marking basis DetectionTargetInfo data is constructed according to the marking data.

3. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: In the step S3, the upper and lower limits of the HSV space processing threshold are set, and then the Mat matrix imageSrc is subjected to HSV space processing.

4. The tool rack status detection method based on visual recognition according to claim 3, characterized in that: In step S3, the HSV space includes hue, saturation and brightness.

5. The tool rack status detection method based on visual recognition according to claim 2, characterized in that: In the step S4, the mask matrix is ​​cropped according to the DetectionTargetInfo data through the marking.

6. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: Also includes: Step S9, recalculate the tool rack status DetectionResult after label grouping: If all the tool rack position status DetectionResult in the label group are in place, it means that the tool is in place; If one of the tool rack status DetectionResult in the label group is misplaced, it means that the tool is misplaced; Other situations indicate that the tool is out of position.

7. The tool rack status detection method based on visual recognition according to claim 1, characterized in that: Also includes: Step S10, printing out the parameters and calculation results.

8. The tool rack status detection method based on visual recognition according to claim 7, characterized in that: In the step S10, the parameters and calculation results include the label label, the position status result of the label label and the tool position status DetectionResult.

9. The tool rack status detection method based on visual recognition according to claim 8, characterized in that: The tool stand status DetectionResult includes the marking basis DetectionTargetInfo, Score score and Mask matrix imageOutput after HSV processing.

10. A tool rack status detection system based on visual recognition, characterized in that: The system is used to execute the method according to any one of claims 1 to 9.

Images