A method and computer program product and computing device for borehole location and sequence detection recognition

Abstract

This invention provides a method, computer program, and computing device for detecting and identifying drilling positions and sequences. The method includes: acquiring and decoding a multimedia stream from a camera device to obtain current color image data; performing mold recognition and segmentation on the current color image data to obtain the current mold type and mold outline region; determining whether the current mold is correctly placed based on the mold outline region; performing drilling segmentation on the current color image data to obtain current operation data, the current operation data including the center point coordinates of each drill hole and the drilling sequence during the current operation; and determining whether the drilling positions and drilling sequences are correct based on the current operation data. According to the technical solution of this invention, the correctness of the drilling position and drilling sequence can be automatically detected during drilling operations, and a real-time alarm can be triggered when an error in the drilling position or drilling sequence occurs.

Description

Technical Field

[0001] This invention relates to the field of video surveillance technology, specifically to a method, computer program product, and computing device for detecting and identifying the location and sequence of boreholes. Background Technology

[0002] With the development of Industry 4.0, more and more manufacturing companies are adopting automation and intelligent technologies to improve production efficiency, reduce human error, lower costs, and improve product quality. This includes technologies for precise control of every stage of the manufacturing process, especially for drilling operations that require high precision.

[0003] In drilling operations, images captured by cameras can be processed and analyzed to identify important information such as mold type, drilling location, and sequence. The application of computer vision technology enables machines to understand image and video data, achieving automated inspection and analysis. By combining machine learning and deep learning techniques with modern image processing algorithms, such as edge detection, feature extraction, and template matching, key inspection information such as mold contours and drilling center points can be effectively identified from complex industrial scenes.

[0004] In the manufacturing industry, ensuring that parts are machined in accordance with design specifications is crucial. Incorrect drilling locations or sequences can lead to product quality issues and even safety hazards.

[0005] Therefore, a technical solution is needed that can automatically detect the correctness of the drilling position and drilling sequence during drilling operations, and provide real-time alarm prompts when errors occur in the drilling position or drilling sequence. Summary of the Invention

[0006] The present invention aims to provide a method, computer program product and computing device for detecting and identifying drilling position and sequence, which can automatically detect the correctness of drilling position and drilling sequence during drilling operations, and provide real-time alarm prompts when drilling position or drilling sequence errors occur.

[0007] According to one aspect of the present invention, a method for detecting and identifying borehole location and sequence is provided, comprising:

[0008] Acquire and decode the multimedia stream from the camera device to obtain the current color image data;

[0009] The current color image data is segmented for mold recognition to obtain the current mold type and mold outline region;

[0010] Determine whether the current mold is placed correctly based on the mold outline area;

[0011] The current color image data is segmented by drilling to obtain current operation data, which includes the center point coordinates of each drill hole and the drilling sequence during the current operation.

[0012] Based on the current operation data, determine whether the drilling position and drilling sequence are correct.

[0013] According to some embodiments, determining whether the drilling position and drilling sequence are correct based on the current operation data includes: matching the current operation data with the operation template; if there is a deviation, an alarm prompt is generated.

[0014] According to some embodiments, before acquiring and decoding the multimedia stream from the camera device to obtain the current color image data, the method further includes:

[0015] Deploy the aforementioned camera device for monitoring the workshop work area;

[0016] Set up different types of mold placement block areas, and generate a polygonal region for each mold placement block area;

[0017] Mold calibration is performed on different types of molds to obtain an operation template for each type of mold. The operation template includes the coordinates of all drilling center points and the drilling sequence number for each type of mold.

[0018] According to some embodiments, determining whether the current mold is correctly placed based on the mold outline area includes:

[0019] Calculate the intersection-union ratio of the mold outline region and the polygonal region of the corresponding type of mold;

[0020] If the crossover ratio is greater than the first threshold, it indicates that the mold is placed correctly; otherwise, it indicates that the mold is placed abnormally, until the mold is placed correctly.

[0021] According to some embodiments, the current operation data is matched with the operation template. If a deviation exists, an alarm is generated, including:

[0022] The current operation data is matched with the operation template to obtain the drilling coordinate deviation and verify the drilling sequence;

[0023] An alarm will be generated if the borehole coordinate deviation exceeds the tolerance requirement or the borehole sequence is inconsistent.

[0024] According to some embodiments, borehole coordinate deviations are obtained, including:

[0025] The borehole coordinate deviation is the Euclidean distance between the current borehole center coordinate point and the corresponding borehole center coordinate point in the operation template.

[0026] According to some embodiments, the drilling sequence is verified, including:

[0027] After the current drilling operation is completed, calculate the Euclidean distance between the current drilling hole and the center coordinates of any previous drilling hole, subtract the absolute value of the Euclidean distance between the corresponding two drilling center coordinates in the corresponding operation template, divide by the Euclidean distance between the corresponding two drilling center coordinates in the corresponding operation template, and obtain the sequence verification value.

[0028] If the sequence verification value is greater than the second threshold, then the drilling sequence is determined to be inconsistent.

[0029] According to some embodiments, different types of mold placement block areas are set, and a polygonal region is generated for each mold placement block area, including:

[0030] Acquire frame images from the camera device;

[0031] Different types of mold placement block areas are defined in the frame image, and the key point coordinates on the boundary of each mold placement block area are recorded;

[0032] The polygonal region of the mold placement block area is constructed using the coordinates of the key points.

[0033] According to some embodiments, mold calibration is performed on different types of molds to obtain an operation template for each type of mold, including:

[0034] Obtain frame images of different types of molds placed in the polygonal regions of the corresponding mold types, and use them as mold calibration maps;

[0035] By using an open-source computer vision library, the pixel coordinates of the center point of the drill hole in each mold calibration diagram and the order of drilling requirements are obtained, and operation templates for the corresponding mold types are generated.

[0036] According to another aspect of the present invention, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any of the preceding claims.

[0037] According to another aspect of the present invention, a computing device is provided, comprising:

[0038] Processor; and

[0039] A memory storing a computer program that, when executed by the processor, causes the processor to perform the method described in any of the preceding methods.

[0040] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any of the preceding claims.

[0041] According to an embodiment of the present invention, the mold type and mold outline area are obtained through mold identification and segmentation. Then, the correctness of the placement position and occlusion are judged to ensure that the relative position of the mold is correct. Subsequently, during the drilling operation, the position and sequence of each new hole are judged, which can more accurately identify drilling position and sequence errors. When drilling deviations occur, the operation is stopped in time to prevent a large number of defective products due to equipment problems.

[0042] According to some embodiments, the design scheme of the present invention utilizes monitoring and identification methods for operational supervision, replacing traditional manual supervision and significantly reducing supervision costs. The use of a single camera device reduces equipment costs, making this method suitable for more small and medium-sized enterprises and better promoting the automation of supervision in the manufacturing industry.

[0043] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0045] Figure 1 A flowchart illustrating a method for detecting and identifying borehole location and sequence according to an example embodiment is shown.

[0046] Figure 2 This diagram illustrates the device setup flowchart prior to identification detection according to an example embodiment.

[0047] Figure 3 This diagram illustrates a method for detecting and identifying borehole location and sequence according to an example embodiment.

[0048] Figure 4 A schematic diagram illustrating a method for generating a polygonal region of a mold placement block area according to an example embodiment.

[0049] Figure 5 This diagram illustrates a method for generating an operation template according to an example embodiment.

[0050] Figure 6 A block diagram of a computing device according to an exemplary embodiment is shown. Detailed Implementation

[0051] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0052] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0053] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0054] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0055] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0056] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0057] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing the present invention, and therefore cannot be used to limit the scope of protection of the present invention.

[0058] With advancements in network technology and computing power, an increasing number of manufacturing companies are adopting automation and intelligent technologies to improve production efficiency, reduce human error, lower costs, and enhance product quality. This includes technologies for precise control of every stage of the manufacturing process, particularly for drilling operations that require high accuracy.

[0059] In drilling operations, images captured by cameras can be processed and analyzed to identify crucial information such as mold type, drilling location, and sequence. The application of computer vision technology enables machines to understand image and video data, achieving automated inspection and analysis. By combining machine learning and deep learning techniques with modern image processing algorithms, such as edge detection, feature extraction, and template matching, key inspection information such as mold contours and drilling center points can be effectively identified from complex industrial scenes. In the manufacturing industry, ensuring that parts are processed according to design specifications is crucial. Incorrect drilling locations or sequences can lead to product quality issues and even safety hazards.

[0060] To address this, the present invention proposes a method, computer program, and computing device for detecting and identifying drilling positions and sequences. This method can automatically detect the correctness of drilling positions and sequences during drilling operations, and provide real-time alarms when errors in drilling position or sequence occur. According to an embodiment, the mold type and mold outline area are obtained through mold identification and segmentation. Correct placement and occlusion checks are used to ensure the relative position of the mold is correct. During the drilling operation, position and sequence checks are performed on each new hole, enabling more accurate identification of drilling position and sequence errors. Operations are stopped promptly when drilling deviations occur, preventing a large number of defective products due to equipment problems.

[0061] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention.

[0062] Figure 1 A flowchart illustrating a method for detecting and identifying borehole location and sequence according to an example embodiment is shown.

[0063] See Figure 1 A method for detecting and identifying the location and sequence of boreholes, according to an example embodiment, includes:

[0064] In S101, the multimedia stream from the camera device is acquired and decoded to obtain the current color image data.

[0065] According to some embodiments, firstly, a monocular camera with a resolution of 1080p or higher is installed and deployed, facing the workshop work area. The camera continuously captures real-time video streams during the production process and decodes these streams to obtain color image data, for example, using Python combined with an open-source computer vision library (OpenCV). The decoded video stream consists of a series of consecutive image frames. Each image frame is a separate color image data.

[0066] According to some embodiments, video stream acquisition can be performed using a software development kit (SDK) provided by the camera manufacturer.

[0067] In S103, the current color image data is segmented for mold recognition to obtain the current mold type and mold outline region.

[0068] According to some embodiments, before image processing, simple image processing is usually required, such as converting the image to grayscale, i.e., converting a color image to a grayscale image, to simplify subsequent processing. Image filtering is also usually required, using Gaussian filtering or other filters to remove noise and smooth the image. After that, image feature extraction is performed. Specifically, methods such as Canny edge detection and Sobel operator can be used to detect the edges of the mold, and Hough transform or contour detection can be used to identify the basic shape of the mold.

[0069] According to some embodiments, semantic segmentation can be used for mold identification and segmentation. Algorithms such as DeepLab, Mask2Former, and SegFormer can effectively capture multi-scale contextual information in the image, thereby improving segmentation accuracy. Alternatively, template matching technology can be used to identify the mold type. This involves matching a predefined template image with the current image and automatically learning features from the image using a learning model (e.g., Support Vector Machine, Random Forest) or deep learning methods for mold classification and identification.

[0070] According to some embodiments, the mold contour detection can be achieved using contour detection functions in libraries such as OpenCV, and the mold contour can be fitted by using methods such as minimum bounding rectangle, minimum bounding circle or ellipse to obtain the precise position and size of the mold.

[0071] In S105, it is determined whether the current mold is placed correctly based on the mold outline area.

[0072] According to some embodiments, before determining whether the current mold is correctly placed based on the mold outline area, the correct placement position of each mold needs to be set. Typically, this setting needs to be done before monitoring and detection, by setting the correct placement position of all molds in the monitoring area to obtain the correct placement area for each mold.

[0073] According to some embodiments, after obtaining the correct placement area of ​​the mold, the correctness of mold placement is determined by calculating the intersection-union ratio (IUR) between the mold outline area and the correct placement area. If the IUR is greater than a certain threshold, the mold placement is considered correct. For specific scenarios, a mold orientation determination can be added, calculating the principal axis direction of the mold outline to determine if it meets expectations. When both the mold position and orientation meet the requirements, the mold is considered correctly placed.

[0074] According to some embodiments, the bounding rectangle of the mold contour region is generated and occlusion classification is performed. For each mold contour region, functions provided by computer vision libraries (such as OpenCV), such as cv2.minAreaRect() or cv2.boundingRect(), can be used to generate its minimum bounding rectangle, which is the rectangle that minimizes the bounding contour.

[0075] In some embodiments, occlusion classification is based on the position and size of the circumscribed rectangle, combined with the position of image boundaries or other objects, to determine whether the mold is occluded. Occlusion is categorized into occlusion and non-occlusion. When occlusion is detected, an alarm is triggered until the mold is determined to be non-occluded, at which point the next step can be performed. Occlusion classification helps operators determine whether the mold needs to be repositioned or whether the next processing operation can continue. This judgment can reduce drilling position errors and deviations caused by mold occlusion, ensuring the pass rate of mold components.

[0076] In S107, the current color image data is segmented by drilling to obtain current operation data, which includes the center point coordinates of each hole and the drilling sequence during the current operation.

[0077] According to some embodiments, borehole segmentation can employ image segmentation techniques, such as algorithms like DeepLab, Mask2Former, and SegFormer, to mark the borehole region in an independent binary image or mask image for each borehole. For each segmented borehole region, functions from an image processing library (such as OpenCV) are used to calculate the centroid or center point coordinates, obtaining a list of center point coordinates for each completed borehole in the current operation. By analyzing the changes in borehole coordinates obtained from the image data of each preceding and following frame in the video stream—that is, the coordinates of newly added points—the order of all borehole coordinates is determined; in other words, the order of completed boreholes is determined by the temporal order of their appearance.

[0078] In S109, based on the current operation data, it is determined whether the drilling position and drilling sequence are correct.

[0079] According to some embodiments, determining whether the drilling position and drilling sequence are correct based on the current operation data includes: matching the current operation data with the operation template; if there is a deviation, an alarm prompt is generated.

[0080] According to some embodiments, a predefined operation template is loaded, which contains data for correct operation, including the coordinates of all drill center points and drill sequence numbers for each type of mold. The drill positions and sequences in the current operation data are compared with the data in the template to check for positional and sequence deviations. The deviation between the actual position of each drill hole and the corresponding position in the template is calculated, and the drill sequence in the current operation data is compared with the sequence in the template. If a deviation exceeds the tolerance requirement, an alarm is triggered.

[0081] Figure 2 This diagram illustrates the device setup flowchart prior to identification detection according to an example embodiment.

[0082] See Figure 2 According to the method described in the example embodiment, before acquiring and decoding the multimedia stream from the camera device to obtain the current color image data, the method further includes:

[0083] In S201, the camera device is deployed for monitoring the workshop work area.

[0084] According to some embodiments, to meet the requirements of subsequent recognition accuracy and the actual workshop scene, a camera device with a resolution of 1080p or higher suitable for the workshop environment is selected and installed in a position that can clearly cover the entire work area. The appropriate camera device is selected based on factors such as workshop lighting conditions and working environment. In special industrial scenarios, considering the possibility of dust, temperature changes, etc., industrial-grade cameras with higher protection levels can be selected to ensure that the selected camera has sufficient resolution and frame rate to capture clear images. The optimal installation position of the camera device is determined according to the workshop layout, so that the camera is directly facing the mold placement area to ensure that the camera can cover all important work areas.

[0085] According to some embodiments, the camera is configured as necessary, such as adjusting parameters like resolution, frame rate, and exposure time. The camera's IP address and other network settings are configured to enable it to function properly within the local area network. After installation, a preliminary image quality test is performed, and adjustments are made based on the test results to optimize image quality.

[0086] In S203, different types of mold placement block areas are set, and a polygonal area is generated for each mold placement block area.

[0087] According to some embodiments, in actual working scenarios, the mold to be processed needs to be manually or automatically placed in a specific position on the corresponding machining lathe or equipment before being processed. Therefore, it is necessary to set the correct placement area for the mold.

[0088] According to some embodiments, since each mold may have different sizes and shapes, it is necessary to place different types of molds into their corresponding correct placement areas. Then, using the OpenCV library, frame images are displayed, and a series of point coordinates for the placement areas of different types of molds are obtained. Based on these point coordinates, polygonal regions can be generated, each with only one unique number, to facilitate subsequent identification of mold placement blocks based on mold type. Specifically, real-time video streams need to be acquired and decoded to obtain color image data. The color image data is preprocessed, such as denoising and contrast enhancement, to improve image quality. Then, edge detection methods are applied to highlight the mold outline, and image segmentation techniques (such as semantic segmentation and instance segmentation) are used to identify the position and shape of the mold. The segmented image generates a mask for each mold. Key point coordinates of the mold are extracted from the mask; these key points can be corner points, center points, etc. In some special scenarios, training samples can also be created by manual annotation to train the model to recognize these key points. Finally, based on these extracted key point coordinates, polygonal regions surrounding the mold are generated and numbered for subsequent identification.

[0089] In S205, mold calibration is performed for different types of molds to obtain an operation template for each type of mold. The operation template includes the coordinates of all drilling center points and the drilling sequence number for each type of mold.

[0090] According to some embodiments, different types of molds are placed within corresponding polygonal areas. The positions of the molds are captured by a camera device, and the coordinates of the center points of all drill holes on each mold and their drilling sequence numbers are determined.

[0091] According to some embodiments, feature detection algorithms (such as Harris corner detection, SIFT, etc.) can be used to find the center point of the drill hole, or contour detection algorithms (such as the findContours function in OpenCV) can be applied to find the drill hole contour and extract the center point from it. The drilling order is determined according to the mold design drawing or process requirements. An operation template is created, and the detailed information of each mold is recorded on the operation template, including: the coordinates of all drill hole center points of each type of mold and the drilling sequence number. Specifically, the mold is manually placed in the corresponding polygonal area and completely matches the polygonal area. Then, the frame image is displayed using the OpenCV library. The mouse is placed at the drill hole center point position for manual setting. Then, the pixel coordinates of the center point are obtained using the OpenCV library. According to the drilling requirements, the pixel coordinates of all drill hole center points are obtained respectively, and all drill hole coordinates are saved in a txt file in order. The fields from left to right are mold type, first drill hole coordinate, second drill hole coordinate, third drill hole coordinate, and so on.

[0092] Figure 3 This diagram illustrates a method for detecting and identifying borehole location and sequence according to an example embodiment.

[0093] See Figure 3 The diagram illustrates a method for detecting and identifying the location and sequence of drill holes according to an example embodiment. As shown in the diagram, the method requires first deploying a camera device to monitor the workshop work area, setting up different types of mold placement blocks, and generating a polygonal area for each mold placement block. Mold calibration is performed on different types of molds to obtain an operation template for each type of mold. The operation template includes the coordinates of all drill hole center points and the drill hole sequence number for each type of mold.

[0094] Then, the multimedia stream from the camera device is acquired and decoded to obtain the current color image data. Mold detection, recognition, and segmentation are performed to obtain the current mold type and mold outline region. Based on the mold outline region, it is determined whether the current mold is correctly placed. A bounding rectangle of the mold outline region is generated, and occlusion is classified. If occlusion exists, an alarm is triggered until the mold is unobstructed. Drilling segmentation is performed on the current color image data to obtain the current operation data. Finally, the current operation data is matched with the operation template. If a deviation exists, an alarm is triggered. Determining whether the current mold is correctly placed based on the mold outline region includes: calculating the intersection-union ratio (IUGR) of the mold outline region and the polygonal region of the corresponding mold type; if the IUGR is greater than a first threshold, the mold is indicated as correctly placed; otherwise, an abnormal mold placement is indicated until the mold is correctly placed.

[0095] According to some embodiments, in actual working scenarios, after the mold to be processed needs to be manually or automatically placed in a specific position on the corresponding processing lathe or equipment, it is also necessary to verify and judge whether the placement area of ​​the mold to be processed is correct, so as to ensure that the subsequent processing operation position meets the drawings and process requirements.

[0096] According to some embodiments, edge detection algorithms (such as Canny edge detection) are used to find the edges of the mold, and contour detection algorithms (such as the findContours function in OpenCV) are used to extract the contours from the edges. After obtaining the mold contours, the intersection-over-union (IoU) ratio of the mold contour region and the polygon region of the corresponding type of mold is calculated. If the IoU is greater than a first threshold, the mold is considered to be placed correctly; otherwise, the mold is considered to be placed incorrectly, and an alarm is issued.

[0097] According to some embodiments, after the current mold is correctly placed, the bounding rectangle of the mold outline area is generated, and occlusion is classified, including: occlusion classification includes occlusion and non-occlusion. If the occlusion type is occlusion, then an alarm is triggered.

[0098] According to some embodiments, the current operation data is matched with the operation template. If there is a deviation, an alarm is generated. This includes: matching the current operation data with the operation template to obtain the drilling coordinate deviation and verifying the drilling sequence; if the drilling coordinate deviation is greater than the tolerance requirement or the drilling sequence is inconsistent, an alarm is generated.

[0099] According to some embodiments, the drilling coordinate deviation is obtained as follows: the drilling coordinate deviation is the Euclidean distance between the current drilling center coordinate point and the corresponding drilling center coordinate point in the operation template. The current operation data and the operation template data are read, and for each drilling hole in the current operation data, the deviation between its current coordinates and the coordinates in the operation template is calculated. The drilling coordinate deviation is the Euclidean distance between the current drilling center coordinate point and the corresponding drilling center coordinate point in the operation template, and the deviation value for each drilling hole is recorded. Based on the drawing and process requirements in the actual scenario, a tolerance value is set as the maximum allowable deviation range. For each drilling hole, its deviation is checked to see if it exceeds the tolerance value. If the coordinate deviation of any drilling hole exceeds the tolerance requirement, an alarm is triggered. This verification method ensures that the position of each drilling hole conforms to the drawing and process requirements, reduces the quality inspection pressure, and decreases the occurrence of dimensional mismatches during the installation and use of various types of molds.

[0100] According to some embodiments, verifying the drilling sequence includes: after the current drilling operation is completed, calculating the Euclidean distance between the current drilling hole and any previous drilling center coordinate point minus the absolute value of the Euclidean distance between the corresponding two drilling center coordinate points in the corresponding type of operation template, dividing by the Euclidean distance between the corresponding two drilling center coordinate points in the corresponding type of operation template to obtain the sequence verification value; if the sequence verification value is greater than a second threshold, then it is determined that the drilling sequence is inconsistent.

[0101] According to some embodiments, current operation data and operation template data are read. For the current drill hole (assuming it is the nth drill hole), the Euclidean distance between the current drill hole and any previous drill hole is calculated based on the current operation data. Similarly, the Euclidean distance between the corresponding two drill holes in the operation template is calculated to obtain the sequence verification value. A second threshold is set according to the actual process parameter requirements and machining accuracy requirements as a standard for judging sequence consistency. If the sequence verification value is greater than the second threshold, the drilling sequence is considered inconsistent.

[0102] Figure 4 A schematic diagram illustrating a method for generating a polygonal region of a mold placement block area according to an example embodiment.

[0103] See Figure 4 According to the example embodiment, different types of mold placement block areas are set, and a polygonal region is generated for each mold placement block area, including:

[0104] In S401, a frame image from the camera device is acquired.

[0105] According to some embodiments, a real-time video stream is acquired and decoded to obtain color image data, and the frame images therein are acquired for the setting of the subsequent mold placement block area. The frame images are preprocessed, such as noise reduction and contrast enhancement, to improve image quality.

[0106] In S403, different types of mold placement block areas are set in the frame image, and the key point coordinates on the boundary of each mold placement block area are recorded.

[0107] According to some embodiments, the frame image acquired from the camera device is loaded, the mold placement area is manually labeled and the key point coordinates are recorded. Alternatively, computer vision technology can be used to automatically detect the mold placement area and record the key point coordinates. For example, edge detection methods can be applied to highlight the mold outline, and image segmentation techniques (such as semantic segmentation and instance segmentation) can be used to identify the position and shape of the mold. The segmented image will generate a mask for each mold. The key point coordinates of the mold are extracted from the mask. In some special scenarios, training samples can also be created by manual labeling to train the model to recognize these key points.

[0108] In S405, the key point coordinates are used to construct the polygonal region of the mold placement block area.

[0109] According to some embodiments, finally, based on the key point coordinates extracted above, a polygonal region surrounding the mold is generated, and the polygonal region is numbered for easy subsequent identification.

[0110] Figure 5 This diagram illustrates a method for generating an operation template according to an example embodiment.

[0111] According to some embodiments, see Figure 5 The process involves placing different types of molds in the polygonal regions of the corresponding mold types, performing mold calibration, and obtaining an operation template for each type of mold. This includes: acquiring frame images of different types of molds placed in the polygonal regions of the corresponding mold types as mold calibration images; using the OpenCV library to obtain the pixel coordinates of the drilling center point and the drilling sequence in each mold calibration image, and generating an operation template for the corresponding mold type.

[0112] According to some embodiments, frame images of different types of molds placed within their corresponding polygonal regions are acquired and used as mold calibration maps. The acquired images are preprocessed, such as by grayscale conversion, binarization, and noise removal.

[0113] According to some embodiments, feature detection algorithms in OpenCV (such as Hough transform) are used to detect the drilling positions on the mold and determine the pixel coordinates of the center point of each drilling hole. Based on the drilling positions and relevant process requirements, the drilling sequence is determined. The pixel coordinates of all drilling center points for each mold type, along with their order, are recorded to form an operation template.

[0114] According to some embodiments, in actual production, when the same type of mold is put into the machine, the previously generated operation template can be matched with the current operation data to detect whether the current drilling operation meets the drawings and process requirements, thereby realizing automated production supervision, reducing the cost of manual supervision and the generation of a large number of defective products caused by inadequate supervision.

[0115] According to some embodiments, the design of the present invention can detect and identify the drilling position and drilling sequence through a single camera device, detect and identify drilling position errors and drilling sequence errors, and issue alarms. This improves the accuracy of supervision while saving the high cost of manual supervision. Moreover, the solution itself has low equipment modification costs and can achieve automated production line supervision with low-cost investment.

[0116] According to some embodiments, the design scheme of this invention obtains the mold type and mold outline area through mold identification and segmentation, and then performs placement position correctness judgment and occlusion judgment to ensure that the relative position of the mold is correct. Subsequently, during the drilling operation, the position and sequence of each new hole are judged, enabling timely stopping of the operation when drilling deviation occurs, preventing a large number of defective products due to equipment problems. The design scheme of this invention improves the accuracy of supervision by utilizing multiple judgments and greatly reduces supervision and equipment costs, making this method suitable for more small and medium-sized enterprises and better promoting the automation supervision process in the manufacturing industry.

[0117] Figure 6 A block diagram of a computing device according to an exemplary embodiment of the present invention is shown.

[0118] like Figure 6 As shown, the computing device 30 includes a processor 12 and a memory 14. The computing device 30 may also include a bus 22, a network interface 16, and an I / O interface 18. The processor 12, memory 14, network interface 16, and I / O interface 18 can communicate with each other via the bus 22.

[0119] Processor 12 may include one or more general-purpose CPUs (Central Processing Units), microprocessors, or application-specific integrated circuits, for executing relevant program instructions. According to some embodiments, computing device 30 may also include a high-performance display adapter (GPU) 20 for accelerating processor 12.

[0120] Memory 14 may include a machine-readable medium in the form of volatile memory, such as random access memory (RAM), read-only memory (ROM), and / or cache memory. Memory 14 is used to store one or more programs containing instructions, as well as data. Processor 12 may read the instructions stored in memory 14 to perform the methods described above according to embodiments of the present invention.

[0121] The computing device 30 can also communicate with one or more networks via the network interface 16. The network interface 16 can be a wireless network interface.

[0122] Bus 22 can include address bus, data bus, control bus, etc. Bus 22 provides a path for exchanging information between components.

[0123] It should be noted that, in specific implementations, the computing device 30 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the device described above may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0124] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of medium or device suitable for storing instructions and / or data.

[0125] This invention also provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0126] Those skilled in the art will clearly understand that the technical solutions of the present invention can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit, etc.

[0127] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0129] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0134] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended provisions.

Claims

1. A method for detecting and identifying the location and sequence of boreholes, characterized in that, include: Acquire and decode the multimedia stream from the camera device to obtain the current color image data; The current color image data is segmented for mold recognition to obtain the current mold type and mold outline region; Determine whether the current mold is placed correctly based on the mold outline area; The current color image data is segmented by drilling to obtain the current operation data, which includes the center point coordinates of each drill hole and the drilling sequence during the current operation. Based on the current operation data, determine whether the drilling position and drilling sequence are correct, including: The current operation data is matched with the operation template to obtain the drilling coordinate deviation and verify the drilling sequence; If the borehole coordinate deviation exceeds the tolerance requirement or the borehole sequence is inconsistent, an alarm will be generated. The borehole coordinate deviation is the Euclidean distance between the current borehole center coordinate point and the corresponding borehole center coordinate point in the operation template. After the current drilling operation is completed, calculate the Euclidean distance between the current drilling hole and the center coordinates of any previous drilling hole, subtract the absolute value of the Euclidean distance between the corresponding two drilling center coordinates in the corresponding operation template, divide by the Euclidean distance between the corresponding two drilling center coordinates in the corresponding operation template, and obtain the sequence verification value. If the sequence verification value is greater than the second threshold, then the drilling sequence is determined to be inconsistent.

2. The method according to claim 1, characterized in that, Before acquiring and decoding the multimedia stream from the camera device to obtain the current color image data, the method further includes: Deploy the aforementioned camera device for monitoring the workshop work area; Set up different types of mold placement block areas, and generate a polygonal region for each mold placement block area; Mold calibration is performed on different types of molds to obtain an operation template for each type of mold. The operation template includes the coordinates of all drilling center points and the drilling sequence number for each type of mold.

3. The method according to claim 1, characterized in that, Determining whether the current mold is placed correctly based on the mold outline area includes: Calculate the intersection-union ratio of the mold outline region and the polygonal region of the corresponding type of mold; If the crossover ratio is greater than the first threshold, it indicates that the mold is placed correctly; otherwise, it indicates that the mold is placed abnormally, until the mold is placed correctly.

4. The method according to claim 2, characterized in that, Different types of mold placement block areas are set up, and a polygonal region is generated for each mold placement block area, including: Acquire frame images from the camera device; Different types of mold placement block areas are defined in the frame image, and the key point coordinates on the boundary of each mold placement block area are recorded; The polygonal region of the mold placement block area is constructed using the coordinates of the key points.

5. The method according to claim 2, characterized in that, Mold calibration is performed for different types of molds to obtain the operation template for each type of mold, including: Obtain frame images of different types of molds placed in the polygonal regions of the corresponding mold types, and use them as mold calibration maps; By using an open-source computer vision library, the pixel coordinates of the center point of the drill hole in each mold calibration diagram and the order of drilling requirements are obtained, and operation templates for the corresponding mold types are generated.

6. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

7. A computing device, characterized in that, include: processor; as well as A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-5.

Images