A workpiece positioning device detection system and detection method based on video stream

Abstract

The present invention discloses a workpiece positioning device detection system and method based on video streaming. The detection system includes a control module and a camera, a data processing module, and a detection result output module electrically connected to the control module. The control module is used to control the operation of the camera, the data processing module, and the detection result output module. In the present invention, the first video data is transmitted based on the transmission mode of the video stream, which is conducive to improving the timeliness of the detection process. The first video data is subjected to target detection based on a deep neural network to determine the target image position of the target workpiece and construct a target image position transformation sequence. By determining whether the target image position transformation sequence matches a predetermined target image prior transformation sequence, it is determined whether the detection is passed. This is conducive to improving the automation of the detection process, thereby facilitating the accuracy of workpiece positioning, and further facilitating improving the processing quality of the workpiece.

Description

Technical Field

[0001] The present invention relates to the technical field of numerical control machining, and in particular to a workpiece positioning device detection system and detection method based on video stream. Background Art

[0002] To address the large volume of video data, a specialized fluidization technology, known as video streaming, can be used to extract files during video transmission. Video data transmission based on video streaming meets the basic requirement of processing received information before receiving the complete data, making it suitable for scenarios with high timeliness requirements.

[0003] CNC machining centers are highly efficient automated machine tools composed of mechanical equipment and CNC systems, suitable for machining complex parts. Five-axis machining centers are a typical example. These centers utilize five axes: x, y, z, a, and c. The x, y, z, and ac axes enable simultaneous machining, excelling at machining curved surfaces, special-shaped parts, hollowing, drilling, beveled holes, and bevel cuts. During operation, a five-axis machining center can complete complex machining operations with a single workpiece setup.

[0004] Practice has shown that for CNC machining centers, especially five-axis machining centers, the accuracy of workpiece positioning affects the machining quality of the workpiece. This requires technicians to regularly inspect the workpiece positioning device to ensure its operational reliability, thereby improving the accuracy of workpiece positioning and, in turn, the machining quality of the workpiece.

[0005] It can be seen that how to design a detection system for workpiece positioning devices in order to improve the degree of automation of its detection, thereby facilitating the accuracy of workpiece positioning and further improving the processing quality of the workpiece is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an intelligent device for capturing and identifying hidden dangers based on video streams.

[0007] The present invention discloses a workpiece positioning device detection system based on video stream, wherein the workpiece positioning device is provided in a five-axis machining center and is used to determine the position of a workpiece during machining. The detection system includes a control module and a camera, a data processing module, and a detection result output module electrically connected to the control module. The control module is used to control the operation of the camera, the data processing module, and the detection result output module. The control steps performed by the control module include:

[0008] The control module controls the camera to transmit the captured first video data about the workpiece to the data processing module based on a video stream transmission method;

[0009] The control module controls the data processing module to perform a target detection operation on the first video data based on a deep neural network to determine a target image position of a target workpiece in each frame image of the first video data;

[0010] The control module controls the data processing module to sort the target image positions in each frame of the first video data according to the chronological order, and construct a target image position transformation sequence;

[0011] The control module controls the data processing module to determine whether the target image position transformation sequence matches the predetermined target image position prior transformation sequence. If so, the control module controls the detection result output module to output first result information indicating that the detection has passed. If not, the control module controls the detection result output module to output second result information indicating that the detection has failed.

[0012] It can be seen that in the workpiece positioning device detection system based on video stream disclosed in the first aspect of the present invention, the first video data is transmitted based on the video stream transmission mode, which is conducive to improving the timeliness of the detection process, and the target detection operation is performed on the first video data based on the deep neural network to determine the target image position of the target workpiece, and construct a target image position transformation sequence. By judging whether the target image position transformation sequence matches the predetermined target image prior transformation sequence, it is determined whether the detection is passed. This is conducive to improving the automation of the detection process, thereby being conducive to the accuracy of workpiece positioning, and further conducive to improving the processing quality of the workpiece.

[0013] As an optional embodiment, in the present invention, the data processing module performs a target detection operation on the first video data based on a deep neural network to determine the target image position of the target workpiece in each frame of the first video data, including the following steps:

[0014] The data processing module performs a frame-by-frame image extraction operation on the first video data to obtain a plurality of images to be detected;

[0015] The data processing module performs gridding processing on the image to be detected, so that each image area in the image to be detected is divided into grids;

[0016] The data processing module performs target detection operations on the images in each grid in the image to be detected in sequence based on a deep neural network to determine whether the image in each grid contains a target workpiece;

[0017] The data processing module determines the adjacent grids containing the target workpiece as the target workpiece image area;

[0018] The data processing module determines a target image position of the target workpiece according to a plurality of pixel anchor points in the workpiece image area.

[0019] As an optional embodiment, in the present invention, the control module is further electrically connected to a workpiece positioning device, and the front of the camera faces the workpiece positioning device;

[0020] Before the control module controls the camera to transmit the captured first video data to the data processing module based on a video stream transmission method, the control step further includes:

[0021] The control module controls the second video data captured by the camera at this stage, which is about the workpiece positioning device being located on the first plane, to be transmitted to the data processing module based on a video stream transmission method;

[0022] The control module controls the data processing module to determine a reference target object image according to the second video data;

[0023] The control module controls the workpiece positioning device to translate in the first plane so that the reference target object image in the third video data of the workpiece positioning device captured by the camera in the next stage is located at the center of a single frame image of the third video data;

[0024] The control module obtains the translation amount of the workpiece positioning device in the first plane, and marks the translation amount as the origin position plane calibration amount of the workpiece positioning device.

[0025] As an optional embodiment, in the present invention, after the control module controls the data processing module to determine the reference target object image according to the second video data, the control step further includes:

[0026] The control module controls the data processing module to determine the distance between the reference target object and the camera according to the second video data;

[0027] The control module marks the distance between the reference target object and the camera as the origin position diameter-depth of the workpiece positioning device.

[0028] As an optional embodiment, in the present invention, after the control module obtains the translation amount of the workpiece positioning device in the first plane and marks the translation amount as the origin position calibration amount of the workpiece positioning device, the control step further includes:

[0029] The control module determines whether the origin calibration amount is greater than or equal to a predetermined calibration amount threshold. If so, the control module controls the detection result output module to output third result information indicating that the position of the workpiece positioning device is abnormal.

[0030] A second aspect of the present invention discloses a method for detecting a workpiece positioning device based on video streams. The workpiece positioning device is provided in a five-axis machining center and is used to determine the position of a workpiece during machining. The detection method is applied to the detection system described in the first aspect of the present invention, wherein the detection method comprises:

[0031] The control module controls the camera to transmit the captured first video data about the workpiece to the data processing module based on a video stream transmission method;

[0032] The control module controls the data processing module to perform a target detection operation on the first video data based on a deep neural network to determine a target image position of a target workpiece in each frame of the first video data;

[0033] The control module controls the data processing module to sort the target image positions in each frame of the first video data according to the chronological order, and construct a target image position transformation sequence;

[0034] The control module controls the data processing module to determine whether the target image position transformation sequence matches the predetermined target image position prior transformation sequence. If so, the control module controls the detection result output module to output first result information indicating that the detection has passed. If not, the control module controls the detection result output module to output second result information indicating that the detection has failed.

[0035] It can be seen that in the workpiece positioning device detection method based on video stream disclosed in the second aspect of the present invention, the first video data is transmitted based on the video stream transmission mode, which is conducive to improving the timeliness of the detection process, and the target detection operation is performed on the first video data based on the deep neural network to determine the target image position of the target workpiece, and construct a target image position transformation sequence. By judging whether the target image position transformation sequence matches the predetermined target image prior transformation sequence, it is determined whether the detection is passed. This is conducive to improving the automation of the detection process, thereby being conducive to the accuracy of workpiece positioning, and further conducive to improving the processing quality of the workpiece. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 1 is a schematic structural diagram of a workpiece positioning device detection system based on video stream according to an embodiment of the present invention;

[0038] Figure 2 is a flowchart of a portion of control steps executed by a control module according to an embodiment of the present invention;

[0039] Figure 3 yes Figure 2 The schematic diagram of the sub-step flow of step S102 is shown;

[0040] Figure 4 is a flowchart of another part of the control steps executed by the control module of an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of the origin position diameter-depth measurement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] like Figure 1As shown, the first aspect of the present invention discloses a workpiece positioning device detection system based on video stream, the workpiece positioning device is set in a five-axis machining center, and is used to determine the position of the workpiece during the machining process. The detection system includes a control module and a camera, a data processing module and a detection result output module electrically connected to the control module respectively. The control module is used to control the operation of the camera, the data processing module and the detection result output module. Figure 2 As shown, the control steps performed by the control module include:

[0045] S101: The control module controls the camera to transmit the captured first video data about the workpiece to the data processing module based on the video stream transmission method.

[0046] The video stream-based transmission method is designed to achieve "transmitting data while processing data", which is conducive to the rational use of time resources and thus to the timeliness of the detection process.

[0047] S102: The control module controls the data processing module to perform a target detection operation on the first video data based on a deep neural network, and determines the target image position of the target workpiece in each frame image in the first video data.

[0048] The deep neural network used in step S102 is a trained neural network model, which includes a network structure and a weight function of the deep neural network fixed after training.

[0049] Optionally, the network structure of the deep neural network can refer to the network structure of Yolo or Fast R-CNN. Further optionally, the deep neural network can be trained based on an image set made from workpiece position images pre-captured by a camera, specifically, the image set includes workpiece position images and relevant annotations of the workpiece positions.

[0050] S103: The control module controls the data processing module to sort the target image positions in each frame image of the first video data according to the chronological order, and construct a target image position transformation sequence.

[0051] The target position image transformation sequence can reflect the dynamic process of the target image position change in the first video data.

[0052] S104: The control module controls the data processing module to determine whether the target image position transformation sequence matches the predetermined target image position a priori transformation sequence. If so, execute step S105a; if not, execute step S105b.

[0053] The predetermined target image position a priori transformation sequence may be predetermined based on a workpiece position transformation process determined by a workpiece machining NC program. Optionally, the workpiece machining NC program may be a NC program actually to be executed or a test program for detecting a workpiece positioning device.

[0054] S105a: The control module controls the detection result output module to output first result information indicating that the detection is passed.

[0055] S105b: The control module controls the detection result output module to output second result information indicating that the detection has failed.

[0056] Optionally, the test result output module may be provided with a port for communicating with an external device, through which the test results are output to the outside; or it may be provided with a display, a voice broadcast device, an audio-visual indication device (such as a buzzer, an LED light), etc. for directly outputting the test results.

[0057] It can be seen that in the workpiece positioning device detection system based on video stream disclosed in the first aspect of the present invention, the first video data is transmitted based on the video stream transmission mode, which is conducive to improving the timeliness of the detection process, and the target detection operation is performed on the first video data based on the deep neural network to determine the target image position of the target workpiece, and construct a target image position transformation sequence. By judging whether the target image position transformation sequence matches the predetermined target image prior transformation sequence, it is determined whether the detection is passed. This is conducive to improving the automation of the detection process, thereby being conducive to the accuracy of workpiece positioning, and further conducive to improving the processing quality of the workpiece.

[0058] Optional, such as Figure 3 As shown, step S102 may specifically include the following steps:

[0059] S1021: The data processing module performs a frame-by-frame image extraction operation on the first video data to obtain a plurality of images to be detected.

[0060] The first video data is split into a plurality of images to be detected, so that the subsequent steps can process the individual images to be detected one by one.

[0061] S1022: The data processing module performs gridding processing on the image to be detected, so that each image area in the image to be detected is divided into grids.

[0062] Through grid division, each image area in the image to be detected can be marked in an orderly manner, which is conducive to the efficiency of target detection.

[0063] S1023. The data processing module performs target detection operations on the images in each grid in the image to be detected based on the deep neural network to determine whether the image in each grid contains the target workpiece.

[0064] As previously mentioned, a trained deep neural network model can be used to sequentially perform target detection on each image within each grid in the image to be detected. Specifically, each image within a grid can serve as input to the deep neural network, which extracts target image features (e.g., target artifact image features) from the image. Based on the extracted target features, the deep neural network determines whether the image contains the target artifact. The output of the deep neural network is the determination result of whether the image within the grid contains the target artifact.

[0065] S1024: The data processing module determines the adjacent grids containing the target workpiece as the target workpiece image area.

[0066] Combine the individual grid images into a complete image area of ​​the target workpiece.

[0067] S1025. The data processing module determines the target image position of the target workpiece according to a plurality of pixel anchor points in the workpiece image area.

[0068] Optionally, three pixel anchor points may be set, two of which may be two pixel points on the diagonal line of the target workpiece image area, and another pixel anchor point may be two pixel points at the center position of the target workpiece image area.

[0069] In steps S1021 to S1025, the first video stream data is first disassembled into single images to be detected, and then the images to be detected are divided into grids. The target detection operation is performed on the single grid images, and the single grids containing the target workpiece images are combined to form a whole target workpiece image area. The target image position of the target workpiece is represented (determined) by a number of pixel anchor points. The target image position in each frame of the first video data is carefully and orderly detected, which is conducive to improving the accuracy of identifying the target image position.

[0070] Optionally, the control module is also electrically connected to the workpiece positioning device, and the front of the camera faces the workpiece positioning device. Figure 4 As shown, before the control module controls the camera to transmit the captured first video data to the data processing module based on the video stream transmission mode, the control step further includes the following steps:

[0071] S201. The control module controls the camera to transmit the second video data of the workpiece positioning device located on the first plane captured in this stage to the data processing module based on the video stream transmission method.

[0072] S202: The control module controls the data processing module to determine a reference target object image according to the second video data.

[0073] Optionally, the process of the reference target object image recognition in step S202 may adopt a contour recognition-based approach to distinguish the target workpiece clamped on the workpiece positioning device, and use the image of the target workpiece as the reference target object image.

[0074] S203 , the control module controls the workpiece positioning device to translate in the first plane so that the reference target object image in the third video data of the workpiece positioning device captured by the camera in the next stage is located in the center of the single frame image of the third video data.

[0075] If the reference target object image itself is located at the center of a single frame image of the third video data, the workpiece positioning device may directly execute step S204 without translation.

[0076] S204: The control module obtains the translation amount of the workpiece positioning device in the first plane, and marks the translation amount as the origin position plane calibration amount of the workpiece positioning device.

[0077] As the reference target object image itself is located at the center of a single frame image of the third video data, the workpiece positioning device may not need to be translated. It can be understood that the translation amount is 0 in this case.

[0078] It can be seen that in steps S201 to 204, before the workpiece positioning device is inspected, the initial position of the target workpiece can be calibrated based on the first plane, which is equivalent to initializing the workpiece positioning device, which is beneficial to improving the standardization of the inspection and thus helping to ensure the credibility of the inspection results.

[0079] Further optionally, after step S202, the control step further includes the following operations:

[0080] S2031: The control module controls the data processing module to determine the distance between the reference target object and the camera according to the second video data.

[0081] Optionally, in step S2031 , the distance between the reference target object and the camera may be determined based on binocular camera ranging or monocular image depth estimation.

[0082] S2032: The control module marks the distance between the reference target object and the camera as the origin position diameter-depth of the workpiece positioning device.

[0083] like Figure 5As shown, the camera is fixed and the camera can be used as a reference point to determine the distance between the first plane and the camera, and the distance is used as the origin position diameter depth measurement, that is, the coordinate value of the target workpiece on the normal coordinate axis of the first plane. Then, the first plane and the normal coordinate can determine the original position of the target workpiece in the space, that is, to realize the spatial position initialization of the workpiece positioning device before detection, which is conducive to further improving the standardization of detection, thereby helping to ensure the credibility of the detection results.

[0084] Further optional, such as Figure 4 As shown, after step S204, the control step further includes the following operations:

[0085] S205: The control module determines whether the origin calibration amount is greater than or equal to a predetermined calibration amount threshold. If so, step S206 is executed.

[0086] S206 , the control module controls the detection result output module to output third result information indicating that the position of the workpiece positioning device is abnormal.

[0087] The predetermined calibration threshold can be determined based on the positioning accuracy requirements of the workpiece positioning device. For example, if the plane positioning accuracy of the workpiece positioning device is ±0.5mm, then the predetermined calibration threshold can be determined by ±0.5mm and the camera intrinsic parameter matrix and the camera extrinsic parameter matrix (conversion is required between the spatial coordinates of the real world and the pixel plane coordinates in the image, and for details, please refer to Zhang's calibration method).

[0088] It can be seen that step S205 and step S206 are equivalent to a preliminary inspection of the workpiece positioning device, which is beneficial to further improve the standardization of the inspection, thereby helping to ensure the credibility of the inspection results.

[0089] A second aspect of the present invention discloses a method for detecting a workpiece positioning device based on video stream, wherein the workpiece positioning device is provided in a five-axis machining center and is used to determine the position of the workpiece during machining. The detection method is applied to the detection system described in the first aspect of the present invention, wherein the detection method includes:

[0090] The control module controls the camera to transmit the captured first video data about the workpiece to the data processing module based on the video stream transmission mode;

[0091] The control module controls the data processing module to perform a target detection operation on the first video data based on the deep neural network, and determines a target image position of the target workpiece in each frame of the first video data;

[0092] The control module controls the data processing module to sort the target image positions in each frame of the first video data according to the time sequence, and construct a target image position transformation sequence;

[0093] The control module controls the data processing module to determine whether the target image position transformation sequence matches the predetermined target image position prior transformation sequence. If so, the control module controls the detection result output module to output first result information indicating that the detection has passed. If not, the control module controls the detection result output module to output second result information indicating that the detection has failed.

[0094] It can be seen that in the workpiece positioning device detection method based on video stream disclosed in the second aspect of the present invention, the first video data is transmitted based on the video stream transmission mode, which is conducive to improving the timeliness of the detection process, and the target detection operation is performed on the first video data based on the deep neural network to determine the target image position of the target workpiece, and construct a target image position transformation sequence. By judging whether the target image position transformation sequence matches the predetermined target image prior transformation sequence, it is determined whether the detection is passed. This is conducive to improving the automation of the detection process, thereby being conducive to the accuracy of workpiece positioning, and further conducive to improving the processing quality of the workpiece.

[0095] The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0096] Finally, it should be noted that the video stream-based workpiece positioning device detection system and method disclosed in the embodiment of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A workpiece positioning device detection system based on video stream, characterized in that: The workpiece positioning device is provided in a five-axis machining center and is used to determine the position of the workpiece during machining. The detection system includes a control module and a camera, a data processing module, and a detection result output module electrically connected to the control module. The control module is used to control the operation of the camera, the data processing module, and the detection result output module. The control steps performed by the control module include: The control module controls the camera to transmit the captured first video data about the workpiece to the data processing module based on a video stream transmission method; The control module controls the data processing module to perform a target detection operation on the first video data based on a deep neural network to determine a target image position of a target workpiece in each frame image of the first video data; The control module controls the data processing module to sort the target image positions in each frame of the first video data according to the chronological order, and construct a target image position transformation sequence; The control module controls the data processing module to determine whether the target image position transformation sequence matches the predetermined target image position prior transformation sequence. If so, the control module controls the detection result output module to output first result information indicating that the detection has passed. If not, the control module controls the detection result output module to output second result information indicating that the detection has failed.

2. The detection system for a workpiece positioning device based on video stream according to claim 1, characterized in that: The data processing module performs a target detection operation on the first video data based on a deep neural network to determine a target image position of a target workpiece in each frame of the first video data, including the following steps: The data processing module performs a frame-by-frame image extraction operation on the first video data to obtain a plurality of images to be detected; The data processing module performs gridding processing on the image to be detected, so that each image area in the image to be detected is divided into grids; The data processing module performs target detection operations on the images in each grid in the image to be detected in sequence based on a deep neural network to determine whether the image in each grid contains a target workpiece; The data processing module determines the adjacent grids containing the target workpiece as the target workpiece image area; The data processing module determines a target image position of the target workpiece according to a plurality of pixel anchor points in the workpiece image area.

3. The detection system for a workpiece positioning device based on video stream according to claim 1, characterized in that: The control module is also electrically connected to the workpiece positioning device, and the front of the camera faces the workpiece positioning device; Before the control module controls the camera to transmit the captured first video data to the data processing module based on a video stream transmission method, the control step further includes: The control module controls the second video data captured by the camera at this stage, which is about the workpiece positioning device being located on the first plane, to be transmitted to the data processing module based on a video stream transmission method; The control module controls the data processing module to determine a reference target object image according to the second video data; The control module controls the workpiece positioning device to translate in the first plane so that the reference target object image in the third video data of the workpiece positioning device captured by the camera in the next stage is located at the center of a single frame image of the third video data; The control module obtains the translation amount of the workpiece positioning device in the first plane, and marks the translation amount as the origin position plane calibration amount of the workpiece positioning device.

4. The detection system for a workpiece positioning device based on video stream according to claim 3, characterized in that: After the control module controls the data processing module to determine the reference target object image according to the second video data, the control step further includes: The control module controls the data processing module to determine the distance between the reference target object and the camera according to the second video data; The control module marks the distance between the reference target object and the camera as the origin position diameter-depth of the workpiece positioning device.

5. The detection system for a workpiece positioning device based on video stream according to claim 3, characterized in that: After the control module obtains the translation amount of the workpiece positioning device in the first plane and marks the translation amount as the origin position calibration amount of the workpiece positioning device, the control step further includes: The control module determines whether the origin calibration amount is greater than or equal to a predetermined calibration amount threshold. If so, the control module controls the detection result output module to output third result information indicating that the position of the workpiece positioning device is abnormal.

6. A method for detecting a workpiece positioning device based on video stream, characterized in that: The workpiece positioning device is provided in a five-axis machining center for determining the position of a workpiece during machining. The detection method is applied to the detection system according to any one of claims 1 to 5, wherein the detection method comprises: The control module controls the camera to transmit the captured first video data about the workpiece to the data processing module based on a video stream transmission method; The control module controls the data processing module to perform a target detection operation on the first video data based on a deep neural network to determine a target image position of a target workpiece in each frame of the first video data; The control module controls the data processing module to sort the target image positions in each frame of the first video data according to the chronological order, and construct a target image position transformation sequence; The control module controls the data processing module to determine whether the target image position transformation sequence matches the predetermined target image position prior transformation sequence. If so, the control module controls the detection result output module to output first result information indicating that the detection has passed. If not, the control module controls the detection result output module to output second result information indicating that the detection has failed.

Images