Robot workstation visual inspection method and system based on artificial intelligence

Abstract

The invention provides a robot workstation visual detection method and system based on artificial intelligence, and the method comprises the steps that an acquisition module receives video data from hardware equipment in a video streaming manner, records and stores a plurality of video frames, and updates training data images; performing semi-automatic interactive labeling program labeling on the video frames, and implanting the labeled video frames into the basic algorithm model for loop training to obtain an optimal model; accelerating and filtering the optimal model, performing real-time video stream detection by using the optimal model, calculating a position relationship of target features in a plurality of video frames, outputting a target result, and inputting the target result into the acquisition module when the target result is an updated training data image; and presetting an alarm rule, accumulating an alarm value when the video frame where the target feature is located is in a preset alarm state, and sending an alarm signal when a preset continuous frame number threshold value is reached. According to the invention, low-cost, high-precision and equipment-integrated visual inspection of the robot workstation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a robot workstation visual inspection method and system based on artificial intelligence. Background Art

[0002] In modern production activities, most factories use a large number of robot workstations to achieve automated production. These workstations usually use fixed program design, and the working path of the machine is pre-set and cannot be temporarily modified, nor can it stop automatically. This design has significant safety hazards. If a person mistakenly enters the working range of the robot, an accident is very likely to occur, resulting in injury. Although some robot workstations have added sensors to detect whether people have entered dangerous areas, such as using infrared sensors to detect temperature to determine whether people have passed by, similar methods have obvious shortcomings. First, accurate sensors are often accompanied by higher prices, which will limit the practicality and economy of the overall system. Secondly, the sensors lack intelligence and are prone to misidentify objects other than people, resulting in unexpected stops of the workstation, which not only affects work efficiency, but also increases production costs.

[0003] In recent years, although some workstations have applied artificial intelligence technology to assist production, most of them require additional computer hardware systems for artificial intelligence software calculations. These hardware are independent of video acquisition equipment, and there are problems such as equipment storage protection, data transmission, and data security. In addition, such systems require professional personnel to debug, and a lot of time is spent on data collection, labeling, and model training, which is relatively costly. Therefore, a more efficient solution is needed to address the above problems. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention proposes a robot workstation visual inspection method and system based on artificial intelligence.

[0005] The first aspect of the present invention discloses a robot workstation visual inspection method based on artificial intelligence, comprising:

[0006] S1: The acquisition module receives video data from the hardware device through video streaming, records and saves multiple video frames and updates the training data image;

[0007] S2: labeling the video frames with a semi-automatic interactive labeling program, and implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model;

[0008] S3: Accelerate and filter the optimal model, use the optimal model to perform real-time video stream detection, calculate the position relationship of target features in multiple video frames, output the target result, and input it into the acquisition module when the target result is an updated training data image;

[0009] S4: Preset alarm rules. When the video frame where the target feature is located is in a preset alarm state, the accumulated alarm value reaches a preset continuous frame number threshold and an alarm signal is issued.

[0010] In an optional embodiment, the performing of semi-automatic interactive labeling program labeling on the video frame comprises:

[0011] S201: The video frame is pre-identified by a basic algorithm model to obtain target features;

[0012] S202: predicting the target ID of the target feature by using the DeepSORT algorithm, outputting a high-confidence target video frame, obtaining a correct target ID, and performing operations on it according to the time sequence;

[0013] S203: Rapidly label multiple video frames using the SAM algorithm and the video segmentation algorithm to obtain labeled video frames.

[0014] In an optional embodiment, the step of implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model includes:

[0015] S204: Calculate the similarity of the plurality of marked video frames by using a perceptual hash algorithm, and filter out redundant marked video frames that are identical or highly similar;

[0016] S205: extracting a background image without target features in the marked video frame, pasting a person image in the material library into the background image without target features, and obtaining a composite image;

[0017] S206: performing data enhancement processing on the synthetic image for training of a target detection algorithm model;

[0018] S207: loop steps S204-S206 until the optimal model is obtained.

[0019] In an optional embodiment, the target detection algorithm model is a YOLO model or a DETR model.

[0020] In an optional embodiment, in S3, the positional relationship of the target features in the multiple video frames is calculated by the DeepSORT algorithm, and the target result is output. The target result includes assigning and outputting a target ID to the tracked target features, and recording the target features that have not been successfully tracked through the acquisition module.

[0021] In an optional embodiment, the positional relationship of the target features in the plurality of video frames is calculated by the DeepSORT algorithm, and a target result is output, wherein the target result includes assigning and outputting a target ID to the tracked target features, and recording the target features that have not been successfully tracked by the acquisition module, including:

[0022] S301: extracting target features of the target to be detected through the Re ID network;

[0023] S302: using Kalman filtering to predict the position of the target feature in the next video frame;

[0024] S303: using the Hungarian algorithm and cascade matching to calculate the matching degree between the target features in the two previous and next video frames, obtaining the corresponding relationship between the same target features in each video frame, and assigning a unique target ID to the tracked target features;

[0025] S304: Filtering target features that have not been successfully tracked, setting the target features that have not been successfully tracked as abnormal target video frames, and recording the abnormal target video frames through a collection module.

[0026] In an optional embodiment, in S1, the method for acquiring the updated training data image includes:

[0027] S101: setting a video frame with a confidence lower than 0.3 as a low-confidence target video frame, and recording the low-confidence target video frame;

[0028] S102: Acquire abnormal target video frames through the DeepSORT algorithm.

[0029] In an optional embodiment, the alarm signal is output to an alarm device to sound an alarm, or connected to a working robot for joint work control, or returned to the acquisition module to be recorded in the form of a video stream.

[0030] The second aspect of the present invention discloses a robot workstation visual inspection system based on artificial intelligence, comprising:

[0031] The acquisition module is used to receive video data from the hardware device in the form of video stream, record and save multiple video frames and update the training data image;

[0032] A training module, used for labeling the video frames with a semi-automatic interactive labeling program, and implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model;

[0033] An inference module is used to accelerate and filter the optimal model, and use the optimal model to perform real-time video stream detection, calculate the positional relationship of target features in multiple video frames, and output the target result. When the target result is an updated training data image, it is input into the acquisition module;

[0034] The alarm module is used to preset alarm rules. When the video frame where the target feature is located is in a preset alarm state, the accumulated alarm value reaches a preset continuous frame number threshold and sends out an alarm signal.

[0035] In an optional embodiment, the updated training data image includes a low-confidence target video frame and an abnormal target video frame, the confidence of the low-confidence target video frame is lower than 0.3, and the abnormal target video frame is obtained by a DeepSORT algorithm.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] In the present invention, the acquisition module, integration module, reasoning module and alarm module are integrated and deployed in an integrated edge device, so that the system and solution are easy to use and debug, and at the same time have a high-precision target feature detection algorithm. Therefore, the system can accurately identify the coordinate information of the target feature in real time, and output the corresponding alarm signal to the robot workstation, which can effectively solve the safety hazard problem of personnel entering and exiting the robot workstation during the production process, and solve the pain points of high cost and complex system in the past, and realize low-cost, high-precision and integrated equipment visual inspection of robot workstations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 It is a flow chart of the visual inspection method of the robot workstation based on artificial intelligence of the present invention;

[0040] Figure 2 A schematic diagram of a robot workstation visual inspection system based on artificial intelligence of the present invention;

[0041] Figure 3 It is a schematic diagram of a robot workstation visual inspection device based on artificial intelligence of the present invention;

[0042] Figure 4 It is a schematic diagram based on a computer-readable storage medium in the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0044] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0045] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0046] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] See also Figure 1 , the embodiment of the present invention discloses a robot workstation visual inspection method based on artificial intelligence, including: S1: an acquisition module receives video data from a hardware device in the form of a video stream, records and saves a plurality of video frames and updates a training data image;

[0048] S2: labeling the video frames with a semi-automatic interactive labeling program, and implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model;

[0049] S3: Accelerate and filter the optimal model, use the optimal model to perform real-time video stream detection, calculate the position relationship of target features in multiple video frames, output the target result, and input it into the acquisition module when the target result is an updated training data image;

[0050] S4: Preset alarm rules. When the video frame where the target feature is located is in a preset alarm state, the accumulated alarm value reaches a preset continuous frame number threshold and an alarm signal is issued.

[0051] Specifically, S1 is mainly completed by the acquisition module, S2 is mainly completed by the training module to complete the training of the model, S3 is mainly completed by the inference module to complete the continuous operation of the edge device and the export of the target result. When the target result is to update the training data image, this part is input into the acquisition module for screening again. S4 is mainly completed by the alarm module, which outputs the alarm signal to the robot workstation to solve the potential safety hazards of personnel entering and exiting the robot workstation during the production process.

[0052] The specific implementation method of the scheme of the present invention is that the acquisition module inputs the video frames contained in the collected video data into the training module, and after the marking of the semi-automatic interactive annotation program, it can be supplemented with manual correction to obtain a high-quality video frame image set, and the image set is implanted into the basic algorithm model for cyclic training, so as to continuously improve the accuracy of the model for video frame detection and recognition until the optimal model is obtained, and then, in the reasoning module, the optimal model is first accelerated and filtered, and then the processed optimal model is used to detect the real-time video stream, and the position relationship of the target features in multiple video frames is calculated. Specifically, the reasoning module calculates the position of the target feature in each frame, and selects the position coordinates of the target feature, outputs the target result, and then processes the output target result; finally, the alarm module filters the aforementioned target result according to the preset alarm rule, and when the video frame where the target feature is located is in the preset alarm state, the alarm value is accumulated, and an alarm signal is issued when the preset continuous frame number threshold is reached. Therefore, this scheme can automatically realize the safety detection work in the robot workstation under simple manual intervention.

[0053] In an optional embodiment, in S2, the step of labeling the video frame using a semi-automatic interactive labeling program includes:

[0054] S201: The video frame is pre-identified by a basic algorithm model to obtain target features;

[0055] S202: predicting the target ID of the target feature by using the DeepSORT algorithm, outputting a high-confidence target video frame, obtaining a correct target ID, and performing operations on it according to the time sequence;

[0056] S203: Rapidly label multiple video frames using the SAM algorithm and the video segmentation algorithm to obtain labeled video frames.

[0057] This step is the image marking stage. The semi-automatic interactive marking program is mainly an interactive time-series marking method based on image segmentation. It combines image segmentation technology and user interaction to perform time-series marking. First, the data video (i.e., video frame) is pre-identified by the basic model, that is, the pre-trained high-generalization model in the training module. At the same time, the DeepSORT algorithm is used to predict the target ID of the target feature, and the video frames containing target features with higher confidence are output as candidate targets and provided to the user for selection. After the user determines the target ID of the correct target feature, the video frames are arranged in chronological order and the user can modify, add, delete, etc. At the same time, the video frames are processed with the help of the Segment anythin algorithm (hereinafter referred to as the SAM algorithm) and the video segmentation algorithm, so that the user only needs to perform simple point-selection operations, such as adding target segmentation excitation points and inhibition points, to quickly mark the entire video stream.

[0058] After completing the video frame labeling work, the training module enters the model training stage.

[0059] Furthermore, in S2, the step of implanting the marked video frames into the basic algorithm model for cyclic training to obtain the optimal model includes:

[0060] S204: Calculate the similarity of the plurality of marked video frames by using a perceptual hash algorithm, and filter out redundant marked video frames that are identical or highly similar;

[0061] S205: extracting a background image without target features in the marked video frame, pasting a person image in the material library into the background image without target features, and obtaining a composite image;

[0062] S206: performing data enhancement processing on the synthetic image for training of a target detection algorithm model;

[0063] S207: loop steps S204-S206 until the optimal model is obtained.

[0064] Specifically, the training module uses the perceptual hashing algorithm to calculate the similarity of the labeled video frame images, filters out the same or highly similar redundant labeled video frame images, obtains a refined background image without target features, copies the background image without target features, and randomly pastes the personnel cutouts in the material library into the background image without target features to obtain a synthetic image, thereby achieving the effect of enhancing the diversity of image data and suppressing the influence of background elements; then, the synthetic image is subjected to data enhancement processing, including random scaling, random flipping, random HSV color enhancement, random affine transformation, PCA color enhancement, random occlusion, noise injection, hybrid technology, etc., and is used for training the target detection algorithm model. By marking the video frames in this way, the marking efficiency can be greatly improved and the labor cost can be reduced. Among them, the automatic segmentation and selection method greatly reduces the user's operating pressure compared to the human eye to identify the target frame. At the same time, the video tracking segmentation allows the user to mark the frame by frame and turns the work of confirming the output of the correction model into the result, which can greatly improve the efficiency. In the case of simple targets, the time spent on manual operation can be reduced from the original 3.43 hours to 23 minutes for testing the marking of 1000 frames of the same video after sampling.

[0065] Specifically, the target detection algorithm model is a YOLO model or a DETR model.

[0066] In the scheme of the present invention, the target detection algorithm model used is the YOLOv5 model, including network structure processing and data enhancement processing, wherein the network structure includes a backbone network for extracting image features, using a series of convolutional layers and pooling layers, gradually reducing the resolution of the image, while increasing the number of channels of the feature map, and extracting higher-level semantic features; a neck network for connecting the backbone network and the detection head, further fusing and processing features, and fusing features of different levels by using structures such as feature pyramid networks to improve the detection capability of targets of different scales; a detection head for predicting feature maps and outputting the category probability and bounding box coordinates of the target. Each output unit corresponds to a preset anchor box, which is used to predict whether there is a target in the anchor box and the category and position of the target. The data enhancement processing includes Mosaic data enhancement, which uses 4 pictures to be spliced ​​by random scaling, random cropping and random arrangement to enrich the detection data set, increase the number of small targets, and make the network more robust. In addition, when using Mosaic enhanced training, the data of 4 pictures can be calculated at a time to improve training efficiency; Mixup is used in the field of image classification. In target detection, the pixel values ​​of two images are linearly fused according to a certain coefficient. The label box is directly processed by splicing to increase the diversity of data and improve the generalization ability of the model; Copy-Paste is used to copy and paste the target in one image to another image, increase the number and diversity of targets, and improve the model's detection ability for targets in different scenarios. In the scheme of the present invention, the specific training scheme is: set the training cycle to 300, the learning rate to 0.01, and the learning rate in the model will linearly decrease to 0.001 as the training progresses; after each cycle of training is completed, the model is automatically evaluated, and the accuracy, recall rate, average accuracy of multiple categories within the range of 50% IoU threshold, and average accuracy of multiple classification targets within the range of 50% to 95% IoU threshold of the model on the verification data set are calculated. The four values ​​are weighted and summed, and the cycle model with the minimum value is finally taken as the optimal model, which is output by the training system. At the same time, an automatic cycle training program is built into the training module, and the basic model is automatically updated at intervals of a certain period, and the latest image data and historical image data are added to the training together, and the generalization ability of the basic model is continuously enhanced, so that when it is needed to be used in a specific scene, only a small amount of data on the scene is needed in the video frame marking stage, and a specific model adapted to the scene can be completed by transfer learning with a powerful basic model. Therefore, when it is necessary to deploy its application in a new scene, only a short-term transfer learning training based on the basic model is required to obtain a new model for a specific scene, which is much faster than training a new model from scratch every time.

[0067] In an optional embodiment, in S3, the positional relationship of the target features in the multiple video frames is calculated by the DeepSORT algorithm, and the target result is output. The target result includes assigning and outputting a target ID to the tracked target features, and recording the target features that have not been successfully tracked through the acquisition module.

[0068] Specifically, the inference module is the part of the edge device that runs continuously. In this step, the optimal model output from the training module is accelerated by means of quantization and other means. When the device type is a JETSON series device, the optimal model can be accelerated by the tensorRT acceleration model; at the same time, the inference module will also filter the optimal model. Specifically, the positional relationship of the target features in the multiple video frames is calculated by the DeepSORT algorithm, and the target result is output. The target result includes assigning and outputting the target ID to the tracked target features, and recording the target features that have not been successfully tracked through the acquisition module, including:

[0069] S301: extracting target features of the target to be detected through the Re ID network;

[0070] S302: using Kalman filtering to predict the position of the target feature in the next video frame;

[0071] S303: Use the Hungarian algorithm and cascade matching to calculate the matching degree between the target features in the two previous and next video frames, obtain the corresponding relationship between the same target features in each video frame, and assign a unique target ID to the tracked target feature; specifically, the target features that appear continuously in consecutive video frames are regarded as the tracked target features.

[0072] S304: Filtering target features that have not been successfully tracked, wherein the target features that have not been successfully tracked are set as abnormal target video frames, and the abnormal target video frames are recorded by a collection module. Specifically, target features that do not appear continuously in continuous video frames are regarded as untracked target features.

[0073] Preferably, the inference module is also provided with a front-end setting UI interface for users to view the real-time effect, and set the inference parameters according to actual needs, such as confidence threshold, recognition category, etc.; and the user can also draw a box on the page to select the area of ​​interest, and then the system automatically filters the recognition targets outside the area of ​​interest. By setting the UI visual interface, the operation is simple and easy to understand, the number of parameters that need to be set is small, and they all have clear and easy-to-understand functions. For example, the higher the confidence threshold in the inference module, the stricter the model, the lower the false detection rate, and the output result is easy to understand. Therefore, compared with the existing manual reasoning system solution, this solution is more efficient and convenient to maintain.

[0074] In an optional embodiment, in S1, the method for acquiring the updated training data image includes:

[0075] S101: setting a video frame with a confidence lower than 0.3 as a low-confidence target video frame, and recording the low-confidence target video frame;

[0076] S102: Acquire abnormal target video frames through the DeepSORT algorithm.

[0077] Specifically, the acquisition module receives image data, i.e., video frames, from the hardware device in the form of a video stream, including three data recording methods. The first is manual intervention, i.e., the user determines the start time and end time of the recorded video capture, or records the video duration, and actively stores the video frames contained in the video data; the other two are judged by the results output by the reasoning module. One is to set the video frames with a confidence level lower than 0.3 as low-confidence target video frames, record the low-confidence target video frames determined by the optimal model according to the conditions, and send them to the acquisition module as updated training data images for secondary processing; the second is to record the abnormal target video frames calculated by the DeepSORT algorithm. Through these two methods, the video frames with poor recognition effect and incorrect recognition results are saved, and then implanted into the training module to train the recognition ability of the model again, thereby improving the recognition processing ability of the model.

[0078] In an optional embodiment, the alarm signal is output to an alarm device to sound an alarm, or connected to a working robot for joint work control, or returned to the acquisition module to be recorded in the form of a video stream.

[0079] Specifically, the alarm module is mainly connected to the signal output device. When the target result output by the reasoning module is in the preset alarm state, such as a specific target category appears, or a target enters a specific area, or a specific target enters a specific area, etc., the alarm value will be accumulated. When the alarm value is greater than the preset value within the preset number of consecutive frames, an alarm signal is issued. Among them, the preset value for the alarm in the scheme of the present invention is 5 frames 50%, that is, the alarm signal is output only when there are at least three frames of alarm in five consecutive frames; and the alarm signal in the present invention can be output to the external device in the form of high and low levels, a high level indicates an alarm, and a low level indicates normal; it can also be input into the acquisition module, and the acquisition module records the abnormal time in the form of a video stream; it can also be connected to the working robot for joint control of work.

[0080] In the present invention, by integrating the video stream and the intelligent reasoning process into an integrated device for training and calculation, compared with the existing solution that requires additional computer equipment for reasoning and calculation, it does not need to occupy additional workstation space, reducing possible equipment safety issues and data transmission security issues; secondly, the present invention uses an automatic segmentation and selection method, which can greatly reduce the user's operating pressure compared to the human eye's identification of the target border. At the same time, the video is tracked and segmented, and the user's frame-by-frame annotation work becomes the result of confirming the output of the correction model, which greatly improves the efficiency. When the target is simple, the time spent on manual operation of testing and annotating the same video of 1000 frames after sampling and taking frames is reduced from the original 3.43 hours to 23 minutes. Therefore, the present invention can greatly improve the efficiency of the data annotation stage and reduce labor costs; for For the part that requires manual operation, the present invention can also set a UI visualization interface in a targeted manner, wherein the operation of the UI visualization interface is simple and easy to understand, the number of parameters that need to be set is small, and they all have clear and easy-to-understand functions, such as the confidence threshold in the reasoning module. The higher the confidence threshold, the stricter the model, and the result will have a lower false detection rate, which is easy for users to understand. Therefore, the present invention is easier to deploy and maintain; furthermore, the training module of the present invention is more efficient when training new models. In the presence of a strongly generalized basic model, each time the application is deployed in a new scene, only a short-term transfer learning training based on the basic model is required to obtain a new model for a specific scene, which is much faster than training a new model from scratch each time; finally, the present invention sets double-layer filtering in the reasoning module and the alarm module, which can achieve a lower false alarm rate, and the test in the verification set can filter out more than 85% of the misrecognition of the small model of the yo l ov5 target detection algorithm, greatly improving the accuracy of visual detection.

[0081] See also Figure 2 The second aspect of the present invention discloses a robot workstation visual inspection system based on artificial intelligence, comprising:

[0082] The acquisition module is used to receive video data from the hardware device in the form of video stream, record and save multiple video frames and update the training data image;

[0083] A training module, used for labeling the video frames with a semi-automatic interactive labeling program, and implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model;

[0084] An inference module is used to accelerate and filter the optimal model, and use the optimal model to perform real-time video stream detection, calculate the positional relationship of target features in multiple video frames, and output the target result. When the target result is an updated training data image, it is input into the acquisition module;

[0085] The alarm module is used to preset alarm rules. When the video frame where the target feature is located is in a preset alarm state, the accumulated alarm value reaches a preset continuous frame number threshold and sends out an alarm signal.

[0086] In an optional embodiment, the updated training data image includes a low-confidence target video frame and an abnormal target video frame, the confidence of the low-confidence target video frame is lower than 0.3, and the abnormal target video frame is obtained by a DeepSORT algorithm.

[0087] It should be noted that the target features described in the present invention are mostly the staff in the robot workstation.

[0088] In this system, the acquisition module, integration module, reasoning module and alarm module are integrated and deployed in an integrated edge device, which is easy to use and debug, and has a high-precision target feature detection algorithm. Therefore, this system can accurately identify the coordinate information of the target features in real time and output the corresponding alarm signal to the robot workstation, which can effectively solve the safety hazard problem of personnel entering and exiting the robot workstation during the production process, and solve the pain points of high cost and complex system in the past, and realize low-cost, high-precision and integrated equipment visual inspection of robot workstations.

[0089] See also Figure 3 The third aspect of the present invention discloses a robot workstation visual inspection device based on artificial intelligence, comprising:

[0090] at least one processor, and

[0091] a memory communicatively connected to the at least one processor; wherein,

[0092] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the artificial intelligence-based robot workstation visual inspection method as described in any one of the first aspects of the present invention.

[0093] The computer device may be a terminal, and the computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robot workstation visual detection method based on artificial intelligence is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0094] See also Figure 4 The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the artificial intelligence-based robot workstation visual inspection method as described in any one of the first aspects of the present invention.

[0095] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned robot workstation visual detection method based on artificial intelligence. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0096] Alternatively, if the above modules of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot workstation visual inspection method based on artificial intelligence, characterized in that: The method comprises: S1: The acquisition module receives video data from the hardware device through video streaming, records and saves multiple video frames and updates the training data image; S2: labeling the video frames with a semi-automatic interactive labeling program, and implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model; S3: Accelerate and filter the optimal model, use the optimal model to perform real-time video stream detection, calculate the position relationship of target features in multiple video frames, output the target result, and input it into the acquisition module when the target result is an updated training data image; S4: Preset alarm rules. When the video frame where the target feature is located is in a preset alarm state, the accumulated alarm value reaches a preset continuous frame number threshold and an alarm signal is issued.

2. The artificial intelligence-based robot workstation visual inspection method according to claim 1 is characterized in that: The step of marking the video frames with a semi-automatic interactive annotation program includes: S201: The video frame is pre-identified by a basic algorithm model to obtain target features; S202: predicting the target ID of the target feature by using the DeepSORT algorithm, outputting a high-confidence target video frame, obtaining a correct target ID, and performing operations on it according to the time sequence; S203: Rapidly label multiple video frames using the SAM algorithm and the video segmentation algorithm to obtain labeled video frames.

3. The artificial intelligence-based robot workstation visual inspection method according to claim 2 is characterized in that: The step of implanting the labeled video frames into the basic algorithm model for cyclic training to obtain the optimal model includes: S204: Calculate the similarity of the plurality of marked video frames by using a perceptual hash algorithm, and filter out redundant marked video frames that are identical or highly similar; S205: extracting a background image without target features in the marked video frame, pasting a person image in the material library into the background image without target features, and obtaining a composite image; S206: performing data enhancement processing on the synthetic image for training of a target detection algorithm model; S207: loop steps S204-S206 until the optimal model is obtained.

4. The artificial intelligence-based robot workstation visual inspection method according to claim 3 is characterized in that: The target detection algorithm model is a YOLO model or a DETR model.

5. The artificial intelligence-based robot workstation visual inspection method according to claim 4 is characterized in that: In S3, the positional relationship of the target features in the multiple video frames is calculated by the DeepSORT algorithm, and the target result is output. The target result includes assigning and outputting a target ID to the tracked target features, and recording the target features that have not been successfully tracked through the acquisition module.

6. The artificial intelligence-based robot workstation visual inspection method according to claim 5 is characterized in that: The DeepSORT algorithm is used to calculate the positional relationship of the target features in the plurality of video frames, and the target result is output, wherein the target result includes assigning and outputting a target ID to the tracked target features, and recording the target features that have not been successfully tracked through the acquisition module, including: S301: extracting target features of the target to be detected through the ReID network; S302: using Kalman filtering to predict the position of the target feature in the next video frame; S303: using the Hungarian algorithm and cascade matching to calculate the matching degree between the target features in the two previous and next video frames, obtaining the corresponding relationship between the same target features in each video frame, and assigning a unique target ID to the tracked target feature; S304: Filtering target features that have not been successfully tracked, setting the target features that have not been successfully tracked as abnormal target video frames, and recording the abnormal target video frames through a collection module.

7. The artificial intelligence-based robot workstation visual inspection method according to claim 6 is characterized in that: In S1, the method of acquiring the updated training data image includes: S101: setting a video frame with a confidence lower than 0.3 as a low-confidence target video frame, and recording the low-confidence target video frame; S102: Acquire abnormal target video frames through the DeepSORT algorithm.

8. The artificial intelligence-based robot workstation visual inspection method according to claim 7, characterized in that: The alarm signal is output to the alarm device to sound an alarm, or connected to the working robot for joint work control, or returned to the acquisition module to be recorded in the form of a video stream.

9. A robot workstation visual inspection system based on artificial intelligence, comprising: The acquisition module is used to receive video data from the hardware device in the form of video stream, record and save multiple video frames and update the training data image; A training module, used for labeling the video frames with a semi-automatic interactive labeling program, and implanting the labeled video frames into a basic algorithm model for cyclic training to obtain an optimal model; An inference module is used to accelerate and filter the optimal model, and use the optimal model to perform real-time video stream detection, calculate the positional relationship of target features in multiple video frames, and output the target result. When the target result is an updated training data image, it is input into the acquisition module; The alarm module is used to preset alarm rules. When the video frame where the target feature is located is in a preset alarm state, the accumulated alarm value reaches a preset continuous frame number threshold and sends out an alarm signal.

10. The artificial intelligence-based robot workstation visual inspection system according to claim 9, characterized in that: The updated training data images include low-confidence target video frames and abnormal target video frames, the confidence of the low-confidence target video frames is lower than 0.3, and the abnormal target video frames are obtained through the DeepSORT algorithm.

Images