A vision-based production line control method, device, system, and electronic equipment

Abstract

The present invention discloses a vision-based production line control method and device, system, and electronic device, and relates to the field of machine vision technology. In industrial production workshops, some important production line processes require strict process control, and it is necessary to identify illegal removal and insertion of items. At the same time, some dangerous actions of relevant personnel around the production line also need to be identified in a timely manner to reduce the risk of personal injury and equipment damage. The present invention uses an event camera to obtain images around the production line and arranges network cameras in the open area of ​​the assembly line. Based on image processing related algorithms and deep learning related algorithms, a processor is used for processing and analysis. When the event camera outputs a large number of events in a short period of time, illegal access identification and dangerous action identification are triggered respectively. The processor records and saves various results and transmits them to the mobile terminal of the manager, thereby efficiently realizing multi-faceted control of the production line.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a vision-based production line control method and device, system, and electronic equipment. Background Art

[0002] In industrial production workshops, some important production line processes require strict process control. Generally, we do not want products to be taken away or put in midway on certain key production lines, as this will bring hidden dangers to subsequent production processes and may affect product quality. Therefore, it is necessary to identify the illegal removal and placement of products on the production line. At the same time, some dangerous actions of relevant personnel around the production line also need to be identified in a timely manner to avoid various hidden dangers and reduce the risk of personal injury and equipment damage.

[0003] In order to identify illegal products, traditional solutions generally include hardware control, that is, adding hardware facilities to the assembly line to restrict manual access, and issuing corresponding alarm prompts. However, with the complexity of the production line and the different process requirements of different products, it is no longer suitable to add corresponding hardware restriction equipment at will. In addition, hardware restriction equipment is not easy for management personnel to trace and confirm.

[0004] On the other hand, in order to identify dangerous actions of relevant personnel around the production line, surveillance network cameras are generally used for image monitoring, and judgment is made with the help of action recognition algorithms. Traditional network cameras are not easy to obtain high-quality images in some workshops with a large dynamic range, and some special fast-moving abnormal actions are prone to motion blur, which is not conducive to the calculation and processing of action recognition algorithms. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a vision-based production line control method and device, system, and electronic device to solve the technical problems existing in the related art that it is inconvenient to add hardware restriction devices to the production line, it is inconvenient to trace and confirm illegal pick-and-place situations, and the traditional network camera motion recognition algorithm in workshops with a large dynamic range is prone to errors.

[0006] According to a first aspect of an embodiment of the present application, a vision-based production line control method is provided, comprising:

[0007] Acquire a first real-time video stream collected by a network camera, wherein there is at least one network camera, and the field of view of each network camera only includes an open area on the production line, and the open areas do not intersect with each other;

[0008] In the first real-time video stream, a classification prediction is performed on the current key frame I0 every T frames by using the trained pick-and-place recognition neural network;

[0009] If the neural network classification output is a normal working class, then proceed to the next prediction;

[0010] If the neural network classification output is abnormal pick-and-place, continue to collect the next key frame I1, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F;

[0011] If the proportion of the interval [θ1,θ2] consisting of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the directional distribution of F is greater than the take-out recognition threshold G, then this violation is the take-out product;

[0012] If the proportion of the interval [θ3,θ4] consisting of the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 in the directional distribution of F is greater than the insertion recognition threshold P, then this violation is the insertion of the product;

[0013] Acquire a first real-time event stream collected by an event camera, wherein the field of view of the event camera includes an active area near the production line, accumulate each E events in the first real-time event stream into an event frame image, and store the event frame image in a time window queue;

[0014] The time window queue is predicted by a trained action recognition neural network to identify whether a dangerous action occurs.

[0015] Furthermore, the training process of the trained pick-and-place recognition neural network includes the following steps:

[0016] Extract frame images from multiple videos of the production line to create a data set;

[0017] Extracting frame images of normal working of the production line as the normal working class of the data set;

[0018] Extracting the frame image picture at the moment when abnormal pick-and-place occurs on the production line as the abnormal pick-and-place class of the data set;

[0019] The classification neural network is trained using the data set to achieve the required corresponding accuracy, thereby obtaining the trained pick-and-place recognition neural network.

[0020] Furthermore, the dense optical flow F is calculated through I0 and I1, and the direction distribution of F is counted, including the following steps:

[0021] Through the optical flow calculation method, a dense optical flow F is calculated from two frames of images I0 and I1 with similar pictures and motion changes, the optical flow direction parameter value of each pixel point of the dense optical flow F is calculated, and the proportion of the optical flow direction parameter values ​​of all pixels in different angle ranges is counted.

[0022] Furthermore, storing the event frame image in the time window queue includes the following steps:

[0023] Set a time window queue with a length of LE, and set all initial values ​​to 0;

[0024] In the first real-time event stream, an event frame image is obtained every time E events are accumulated and stored in the time window queue, so that a time window queue full of event frame images is obtained in real time.

[0025] Furthermore, the training process of the trained action recognition neural network includes the following steps:

[0026] The event flow information of normal actions and special dangerous actions in the movable area near the production line acquired by the event camera are stored separately;

[0027] In each event stream, every E events are accumulated into a frame image to generate a video containing action information, and a data set is made. The data set contains normal action event frame videos and various abnormal action event frame videos;

[0028] The action recognition neural network is trained using the data set to achieve the required corresponding accuracy, thereby obtaining the trained action recognition neural network.

[0029] Furthermore, it also includes:

[0030] When the event camera continues to work, when the number of events output by the event camera within the time interval Δt is greater than the trigger threshold E t When the trained pick-and-place recognition neural network and the trained action recognition neural network are triggered to work;

[0031] Within the delay time interval ΔT, if the number of events output by the event camera within the time interval Δt is less than the trigger threshold E t , the trained pick-and-place recognition neural network and the trained action recognition neural network will stop working and wait for the next trigger to start.

[0032] According to a second aspect of an embodiment of the present application, a vision-based production line control device is provided, comprising:

[0033] An acquisition module, used for acquiring a first real-time video stream collected by a network camera, wherein there is at least one network camera, and the field of view of each network camera only includes an open area on the production line, and the open areas do not intersect with each other;

[0034] A first prediction module, configured to perform a classification prediction on the current key frame I0 every T frames by using a trained pick-and-place recognition neural network in the first real-time video stream;

[0035] The first judgment module is used to continue the next prediction if the neural network classification output is a normal pick-and-place class;

[0036] The second judgment module is used to continue collecting the next key frame I1 if the neural network classification output is an abnormal pick-and-place class, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F;

[0037] The third judgment module is used to determine that if the proportion of the interval [θ1, θ2] consisting of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the directional distribution of F is greater than the take-out recognition threshold G, then the violation is the take-out product;

[0038] The fourth judgment module is used to determine that if the proportion of the angle distribution of F falling within the interval [θ3, θ4] formed by the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 is greater than the insertion recognition threshold P, then this violation is the insertion of the product;

[0039] an acquisition and storage module, configured to acquire a first real-time event stream collected by an event camera, wherein the field of view of the event camera includes an active area near the production line, accumulate each E events in the first real-time event stream into an event frame image, and store the event frame image in a time window queue;

[0040] The second prediction module is used to predict the time window queue through a trained action recognition neural network to identify whether a dangerous action occurs.

[0041] According to a third aspect of an embodiment of the present application, a vision-based production line control system is provided, comprising:

[0042] At least one network camera, used for collecting the first real-time video stream, wherein the field of view of each network camera only includes the open areas on the production line, and the open areas do not intersect each other;

[0043] An event camera, used to collect a first real-time event stream, wherein a field of view of the event camera includes an active area near the production line;

[0044] A switch, configured to transmit the first real-time video stream to a processor;

[0045] The processor is used to obtain a first real-time video stream collected by a network camera. In the first real-time video stream, a classification prediction is performed on the current key frame I0 every T frames through a trained pick-and-place recognition neural network; if the neural network classification output is a normal working class, the next prediction is continued; if the neural network classification output is an abnormal pick-and-place class, the next key frame I1 is continued to be collected, and a dense optical flow F is calculated through I0 and I1, and the direction distribution of F is statistically analyzed; if the proportion of the interval [θ1, θ2] composed of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the direction distribution of F is greater than the take-out recognition threshold G, then the violation anomaly is the take-out product; if the proportion of the interval [θ3, θ4] composed of the lower bound θ3 of the put-in angle and the upper bound θ4 of the put-in angle in the direction distribution of F is greater than the put-in recognition threshold P, then the violation anomaly is the put-in product; obtain the first real-time event stream collected by the event camera, accumulate each E event in the first real-time event stream into an event frame image, and store the event frame image in a time window queue; predict the time window queue through a trained action recognition neural network to identify whether a dangerous action occurs.

[0046] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0047] one or more processors;

[0048] A memory for storing one or more programs;

[0049] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0050] According to a fifth aspect of an embodiment of the present application, a processor-readable storage medium is provided, on which processor instructions are stored. When the instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0051] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0052] According to the above technical solution, this application uses a network camera to obtain images of open areas on the production line to identify and alarm illegal pick-and-place, thereby avoiding the addition of hardware restriction devices on complex production lines;

[0053] The pick-and-place recognition neural network is used for image judgment, so the recognition effect is stable, and a high recognition accuracy rate can be achieved for illegal pick-and-place of products in multiple open areas on the production line;

[0054] The event camera is used to identify dangerous actions in the workshop, which overcomes the problems of poor image quality and unstable action recognition algorithm obtained by traditional network cameras in some workshops with large dynamic ranges. It can better adapt to the monitoring of workshop scenes with various dynamic ranges and effectively perform action recognition.

[0055] The event camera is used as a trigger for illegal pick-and-place identification and dangerous action identification, overcoming the problem that some surveillance cameras or other restricted equipment can only work around the clock and are very inefficient when there is no one at the production line. In this way, processing and identification are only performed when potential abnormalities occur, making the management and control method highly resource-efficient;

[0056] Using network cameras and event cameras as the core sensors for management and control can record images of illegal picking and placing and dangerous actions, and then interactively control the status of the production line for easy traceability and confirmation.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 The figure is a flow chart of a vision-based production line control method according to an exemplary embodiment.

[0060] Figure 2 It is a schematic diagram of a vision-based production line control system according to an exemplary embodiment.

[0061] Figure 3 It is a block diagram of a vision-based production line control device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0064] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0065] Figure 1 is a flowchart of a vision-based production line control method according to an exemplary embodiment. Figure 1 , the method may include:

[0066] Step S11, obtaining a first real-time video stream collected by a network camera, wherein there is at least one network camera, and the field of view of each network camera only includes an open area on the production line, and each open area does not intersect with each other;

[0067] Step S12, in the first real-time video stream, performing classification prediction on the current key frame I0 once every T frames through the trained pick-and-place recognition neural network;

[0068] Step S13, if the neural network classification output is a normal working class, proceed to the next prediction;

[0069] Step S14, if the neural network classification output is abnormal pick-and-place, continue to collect the next key frame I1, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F;

[0070] Step S15, if the proportion of the directional distribution of F that falls within the interval [θ1, θ2] formed by the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle is greater than the take-out recognition threshold G, then this violation is a product being taken out;

[0071] Step S16: If the proportion of the angles in the direction distribution of F that fall within the interval [θ3, θ4] formed by the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 is greater than the insertion recognition threshold P, then this violation is caused by the insertion of the product;

[0072] Step S17, obtaining a first real-time event stream collected by an event camera, wherein the field of view of the event camera includes an active area near the production line, accumulating every E events in the first real-time event stream into an event frame image, and storing the event frame image in a time window queue;

[0073] Step S18, predicting the time window queue through the trained action recognition neural network to identify whether a dangerous action occurs.

[0074] It can be seen from the above embodiments that the present application uses a network camera to obtain images of open areas on the production line for illegal picking and placing identification and alarm, thereby avoiding the addition of hardware restriction devices on complex production lines; a pick-and-place recognition neural network is used for image judgment, so that the recognition effect is stable, and a high recognition accuracy rate can be achieved for illegal picking and placing of products in multiple open areas on the production line; an event camera is used to identify dangerous actions in the workshop, which overcomes the problems of poor image quality and unstable action recognition algorithm obtained by traditional network cameras in some workshops with a large dynamic range, and can better adapt to the monitoring of workshop scene images with a variety of dynamic ranges, and can effectively perform action recognition; network cameras and event cameras are used as core sensors for management and control, which can record the images when illegal picking and placing and dangerous actions occur, and then the production line status can be interactively controlled for easy traceability and confirmation.

[0075] In the specific implementation of step S11, a first real-time video stream captured by a network camera is obtained. There is at least one network camera, and the field of view of each network camera only includes open areas on the production line, and the open areas do not intersect with each other.

[0076] Specifically, refer to Figure 2 In the open areas of production line 1 (two of them are shown in the figure): the first open area 2 and the second open area 3, a first network camera 4 and a second network camera 5 are arranged, and the position and angle of the first network camera 4 are adjusted so that its field of view is almost limited to the first open area 2, and the position and angle of the second network camera 5 are adjusted so that its field of view is almost limited to the second open area 3.

[0077] In the specific implementation of step S12, in the first real-time video stream, a classification prediction is performed on the current key frame I0 every T frames through the trained pick-and-place recognition neural network.

[0078] Specifically, in a real-time video stream with a frame rate of FPS, the current key frame I0 is input into the trained pick-and-place recognition neural network every T frames. The output result of the trained pick-and-place recognition neural network is a normal working class or an abnormal pick-and-place class. Not every frame is processed and predicted. This can omit redundant information of high frame rate while meeting real-time requirements.

[0079] In this embodiment, the training process of the trained pick-and-place recognition neural network includes the following steps:

[0080] A1, extract frame images from multiple videos of the production line to create a data set;

[0081] A2, extracting frame images of normal working of the production line as the normal working class of the data set;

[0082] A3, extracting the frame image picture at the moment when abnormal pick-and-place occurs on the production line as the abnormal pick-and-place class of the data set;

[0083] A4, using the data set to train a classification neural network to achieve the required corresponding accuracy, and obtain the trained pick-and-place recognition neural network.

[0084] Using a trained pick-and-place recognition neural network for image judgment can stabilize the recognition effect, improve universality for multiple open areas on the production line, and achieve a higher recognition accuracy.

[0085] In the specific implementation of step S13, if the neural network classification output is a normal working class, then proceed to the next prediction.

[0086] Specifically, if the neural network classification output is a normal working class, it continues to run in the current state and continues to acquire and predict the next key frame, indicating that no abnormality has occurred at present.

[0087] In the specific implementation of step S14, if the neural network classification output is an abnormal pick-and-place class, then continue to collect the next key frame I1, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F.

[0088] Specifically, if the neural network classification output is an abnormal pick-and-place class, the optical flow is calculated with the help of the next acquired key frame I1 and the current key frame I0 to distinguish between abnormal pick-and-place and abnormal placement. Optical flow calculation is only performed when the neural network classification output is an abnormal pick-and-place class, and no optical flow calculation is performed when the output is a normal working class. This can reduce the system's computing consumption as much as possible, thereby meeting real-time detection and alarm requirements.

[0089] Furthermore, the dense optical flow F is calculated through I0 and I1, and the direction distribution of F is counted, including the following steps:

[0090] Through the optical flow calculation method, a dense optical flow F is calculated from two frames of images I0 and I1 with similar pictures and motion changes, the optical flow direction parameter value of each pixel point of the dense optical flow F is calculated, and the proportion of the optical flow direction parameter values ​​of all pixels in different angle ranges is counted.

[0091] The direction of the optical flow indicates the direction of movement of the object in the picture. The proportion of different angle intervals can reflect the movement trend of the product. Taking out and putting in are two abnormal categories with obvious differences in movement directions. The calculation and statistics of dense optical flow can effectively distinguish whether the product is taken out or put in.

[0092] In the specific implementation of step S15, if the proportion of the interval [θ1, θ2] consisting of the lower limit θ1 of the take-out angle and the upper limit θ2 of the take-out angle in the directional distribution of F is greater than the take-out recognition threshold G, then the violation is the take-out product.

[0093] Specifically, the lower bound θ1 and the upper bound θ2 of the take-out angle are set, and the take-out recognition threshold G is set. If the proportion of F's directional distribution falling in the interval [θ1, θ2] is greater than the take-out recognition threshold G, it means that the anomaly meets the optical flow result when taking out the product, and it can be considered that the violation is the take-out of the product.

[0094] In the specific implementation of step S16, if the proportion of the directional distribution of F falling within the interval [θ3, θ4] formed by the lower limit θ3 of the insertion angle and the upper limit θ4 of the insertion angle is greater than the insertion recognition threshold P, then the violation anomaly is the insertion of the product.

[0095] Specifically, the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 are set, and the insertion recognition threshold P is set. If the proportion of the direction distribution of F falling in the interval [θ3,θ4] is greater than the insertion recognition threshold P, it means that the anomaly meets the optical flow result when the product is placed, and it can be considered that the illegal anomaly is the placement of the product.

[0096] In the specific implementation of step S17, a first real-time event stream captured by an event camera is obtained, the field of view of the event camera includes an active area near the production line, every E events in the first real-time event stream are accumulated into an event frame image, and the event frame image is stored in a time window queue.

[0097] Specifically, refer to Figure 2 The event camera 7 can capture the motion information of the movable area near the production line 1, and output a certain number of events corresponding to the current human activities in the movable area near the production line 1, and potential abnormalities may occur, and perform identification and judgment.

[0098] Furthermore, storing the event frame image in the time window queue includes the following steps:

[0099] Set a time window queue with a length of LE, and set all initial values ​​to 0;

[0100] In the first real-time event stream, an event frame image is obtained every time E events are accumulated and stored in the time window queue, so that a time window queue full of event frame images is obtained in real time.

[0101] The time window queue saves the timing information of the current action in real time, and the event frame image saves the spatial information of the current action. The complete time window queue can be effectively processed and predicted by the trained action recognition neural network to obtain the action recognition result. The use of the time window queue can meet the real-time recognition of dangerous actions.

[0102] In the specific implementation of step S18, the time window queue is predicted by a trained action recognition neural network to identify whether a dangerous action occurs.

[0103] Specifically, the action recognition neural network can predict the time window queue that meets the conditions to obtain the action recognition result, and input the time window queue full of event frame images into the trained action recognition neural network. The trained action recognition neural network will output the action recognition result, so as to determine whether any dangerous action to be identified occurs.

[0104] Furthermore, the training process of the trained action recognition neural network includes the following steps:

[0105] The event flow information of normal actions and special dangerous actions in the movable area near the production line acquired by the event camera are stored separately;

[0106] In each event stream, every E events are accumulated into a frame image to generate a video containing action information, and a data set is made. The data set contains normal action event frame videos and various abnormal action event frame videos;

[0107] The action recognition neural network is trained using the data set to achieve the required corresponding accuracy, thereby obtaining the trained action recognition neural network.

[0108] Event cameras have the characteristics of low latency and high dynamic range, and can perform action recognition well in some workshop scenes with a large dynamic range and when fast actions that are prone to motion blur occur.

[0109] Furthermore, a vision-based production line control method provided by an embodiment of the present invention may also include:

[0110] Step S19: when the event camera continues to work, when the number of events output by the event camera within the time interval Δt is greater than the trigger threshold E t When the trained pick-and-place recognition neural network and the trained action recognition neural network are triggered to work;

[0111] Specifically, refer to Figure 2, event camera 7 can capture the motion information of the movable area near production line 1. The more events it outputs, the more people are currently active. Potential anomalies may occur, and algorithms need to be run for identification and judgment. The number of events output by event camera 7 is used as the trigger condition of the system, and the time interval Δt and trigger threshold E are set. t , when the number of events output by the event camera within the time interval Δt is greater than the trigger threshold E t When the trained pick-and-place recognition neural network and the trained action recognition neural network start to work and make corresponding predictions and judgments.

[0112] Step S20: within the delay time interval ΔT, if the number of events output by the event camera within the time interval Δt is less than the trigger threshold E t , the trained pick-and-place recognition neural network and the trained action recognition neural network will stop working and wait for the next trigger to start.

[0113] Specifically, a delay time interval ΔT is set. If the number of events output by the event camera within the delay time interval ΔT is kept smaller than the trigger threshold E t , indicating that there has been no human activity in the current area for a period of time, and the trained pick-and-place recognition neural network and the trained action recognition neural network have stopped working, waiting for the next trigger to start. Therefore, the system does not need to work around the clock, and only processes and identifies when potential anomalies occur, thereby making the management and control method highly efficient in resource utilization.

[0114] The embodiment of the present invention also provides a production line control system based on vision, referring to Figure 2 , the system may include:

[0115] At least one network camera, used for collecting the first real-time video stream, wherein the field of view of each network camera only includes the open areas on the production line, and the open areas do not intersect each other;

[0116] An event camera, used to collect a first real-time event stream, wherein a field of view of the event camera includes an active area near the production line;

[0117] A switch, configured to transmit the first real-time video stream to a processor;

[0118] The processor is used to obtain a first real-time video stream collected by a network camera. In the first real-time video stream, a classification prediction is performed on the current key frame I0 every T frames through a trained pick-and-place recognition neural network; if the neural network classification output is a normal working class, the next prediction is continued; if the neural network classification output is an abnormal pick-and-place class, the next key frame I1 is continued to be collected, and a dense optical flow F is calculated through I0 and I1, and the direction distribution of F is statistically analyzed; if the proportion of the interval [θ1, θ2] composed of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the direction distribution of F is greater than the take-out recognition threshold G, then the violation anomaly is the take-out product; if the proportion of the interval [θ3, θ4] composed of the lower bound θ3 of the put-in angle and the upper bound θ4 of the put-in angle in the direction distribution of F is greater than the put-in recognition threshold P, then the violation anomaly is the put-in product; obtain the first real-time event stream collected by the event camera, accumulate each E event in the first real-time event stream into an event frame image, and store the event frame image in a time window queue; predict the time window queue through a trained action recognition neural network to identify whether a dangerous action occurs.

[0119] Specifically, refer to Figure 2 , taking two network cameras as an example, the first network camera 4 and the second network camera 5 are connected to the network switch 6, and the network switch 6 is connected to the processor 8;

[0120] Arrange an event camera 7 near the production line 1, and adjust the position and angle of the event camera 7 so that its field of view includes the movable area near the production line 1. Connect the event camera 7 to the processor 8;

[0121] The processor 8 records and stores various processing and analysis results, and may also further send them to the mobile terminal 9 .

[0122] Taking the first open area 2 and the first network camera 4 as an example (the second open area 3 and the second network camera 5 have the same algorithm), the specific content of identifying illegal access is as follows:

[0123] The processor 8 obtains the real-time video stream of the first open area 2 captured by the first network camera 4, and in the real-time video stream with a frame rate of FPS, performs a classification prediction on the current key frame I0 every T frames through the trained pick-and-place recognition neural network;

[0124] If the neural network classification output is a normal working class, then proceed to the next prediction;

[0125] If the neural network classification output is abnormal pick-and-place, continue to collect the next key frame I1, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F;

[0126] If the proportion of the interval [θ1, θ2] consisting of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the directional distribution of F is greater than the take-out identification threshold G, then the violation is a product taken out, and the processor 8 records and saves the processing and analysis results, and can also further send them to the mobile terminal 9;

[0127] If the proportion of the interval [θ3, θ4] formed by the lower limit θ3 of the insertion angle and the upper limit θ4 of the insertion angle in the directional distribution of F is greater than the insertion recognition threshold P, then the violation is the insertion of the product, and the processor 8 records and saves the processing and analysis results, and can also further send them to the mobile terminal 9.

[0128] As can be seen from the above embodiments, the first network camera 4 can obtain the image of the first open area 2 on the production line 1 to identify and alarm illegal pick-up and put, and the processor 8 can record and save the abnormal image, and can also further send it to the mobile terminal 9, so that the management personnel can view the relevant image and other information for easy tracing and confirmation. The pick-up and put recognition neural network is used to judge the image, and when the abnormal result is judged, the next key frame is obtained, the dense optical flow F is calculated, the direction distribution of F is counted, and the two states of taking out and putting in are distinguished.

[0129] Specifically, in a real-time video stream with a frame rate of FPS, a neural network classification prediction is performed every T frames, which can omit redundant information of high frame rate. At the same time, optical flow calculation is only performed when it is predicted to be an abnormal class, which can minimize the computing consumption of the system and thus meet the needs of real-time detection and alarm.

[0130] Taking event camera 7 and production line 1 as an example, the specific content of identifying dangerous actions is:

[0131] The processor 8 obtains the real-time event stream of the movable area near the production line 1 collected by the event camera 7, accumulates each E event in the real-time event stream into an event frame image, and stores the event frame image in the time window queue;

[0132] The time window queue is predicted by a trained action recognition neural network to identify whether a dangerous action occurs.

[0133] It can be seen from the above embodiments that the present application adopts event cameras to identify dangerous actions in the workshop, which overcomes the problems of poor image quality and unstable action recognition algorithm obtained by traditional network cameras in some workshops with a large dynamic range. It can better adapt to the monitoring of workshop scene images with a variety of dynamic ranges, effectively perform action recognition, and use time window queues to meet the real-time recognition of dangerous actions.

[0134] Corresponding to the aforementioned embodiment of the vision-based production line control method, the present application also provides an embodiment of a vision-based production line control device.

[0135] Figure 3 It is a block diagram of a vision-based production line control device according to an exemplary embodiment.

[0136] Reference Figure 3 , the device comprises:

[0137] An acquisition module 21 is used to acquire a first real-time video stream collected by a network camera, where there is at least one network camera, and the field of view of each network camera only includes an open area on the production line, and the open areas do not intersect with each other;

[0138] A first prediction module 22 is used to perform a classification prediction on the current key frame I0 every T frames through a trained pick-and-place recognition neural network in the first real-time video stream;

[0139] The first judgment module 23 is used to continue the next prediction if the neural network classification output is a normal pick-and-place class;

[0140] The second judgment module 24 is used to continue to collect the next key frame I1 if the neural network classification output is an abnormal pick-and-place class, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F;

[0141] The third judgment module 25 is used to determine that if the proportion of the interval [θ1, θ2] consisting of the lower limit θ1 of the take-out angle and the upper limit θ2 of the take-out angle in the directional distribution of F is greater than the take-out recognition threshold G, then the violation is the take-out product;

[0142] The fourth judgment module 26 is used to determine that if the proportion of the angle distribution F falling within the interval [θ3, θ4] formed by the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 is greater than the insertion recognition threshold P, then the violation is the product insertion;

[0143] An acquisition and storage module 27 is used to acquire a first real-time event stream collected by an event camera, wherein the field of view of the event camera includes an active area near the production line, accumulate each E events in the first real-time event stream into an event frame image, and store the event frame image in a time window queue;

[0144] The second prediction module 28 is used to predict the time window queue through a trained action recognition neural network to identify whether a dangerous action occurs.

[0145] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0147] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a vision-based production line control method as described above.

[0148] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, a vision-based production line control method as described above is implemented.

[0149] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0150] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A vision-based production line control method, characterized in that: include: Acquire a first real-time video stream collected by a network camera, wherein there is at least one network camera, and the field of view of each network camera only includes an open area on the production line, and the open areas do not intersect with each other; In the first real-time video stream, a classification prediction is performed on the current key frame I0 every T frames by using the trained pick-and-place recognition neural network; If the neural network classification output is a normal working class, then proceed to the next prediction; If the neural network classification output is abnormal pick-and-place, continue to collect the next key frame I1, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F; If the proportion of the interval [θ1,θ2] consisting of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the directional distribution of F is greater than the take-out recognition threshold G, then this violation is the take-out product; If the proportion of the interval [θ3,θ4] consisting of the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 in the directional distribution of F is greater than the insertion recognition threshold P, then this violation is the insertion of the product; Acquire a first real-time event stream collected by an event camera, wherein the field of view of the event camera includes an active area near the production line, accumulate each E events in the first real-time event stream into an event frame image, and store the event frame image in a time window queue; Predicting the time window queue by using a trained action recognition neural network to identify whether a dangerous action occurs; The method further includes: When the event camera continues to work, when the number of events output by the event camera within the time interval Δt is greater than the trigger threshold E t When the trained pick-and-place recognition neural network and the trained action recognition neural network are triggered to work; Within the delay time interval ΔT, if the number of events output by the event camera within the time interval Δt is less than the trigger threshold E t , the trained pick-and-place recognition neural network and the trained action recognition neural network will stop working and wait for the next trigger to start.

2. A vision-based production line control method according to claim 1, characterized in that: The training process of the trained pick and place recognition neural network includes the following steps: Extract frame images from multiple videos of the production line to create a data set; Extracting frame images of normal working of the production line as the normal working class of the data set; Extracting the frame image picture at the moment when abnormal pick-and-place occurs on the production line as the abnormal pick-and-place class of the data set; The classification neural network is trained using the data set to achieve the required corresponding accuracy, thereby obtaining the trained pick-and-place recognition neural network.

3. The vision-based production line control method according to claim 1, characterized in that: The dense optical flow F is calculated through I0 and I1, and the direction distribution of F is counted, including the following steps: Through the optical flow calculation method, a dense optical flow F is calculated from two frames of images I0 and I1 with similar pictures and motion changes, the optical flow direction parameter value of each pixel point of the dense optical flow F is calculated, and the proportion of the optical flow direction parameter values ​​of all pixel points in different angle ranges is counted.

4. The vision-based production line control method according to claim 1, characterized in that: Storing the event frame image in the time window queue includes the following steps: Set a time window queue with a length of LE, and set all initial values ​​to 0; In the first real-time event stream, an event frame image is obtained every time E events are accumulated and stored in the time window queue, so that a time window queue full of event frame images is obtained in real time.

5. The vision-based production line control method according to claim 1, characterized in that: The training process of the trained action recognition neural network includes the following steps: The event flow information of normal actions and special dangerous actions in the movable area near the production line acquired by the event camera are stored separately; In each event stream, each E event is accumulated into a frame image to generate a video containing action information, and a data set is made. The data set contains normal action event frame videos and various abnormal action event frame videos; The action recognition neural network is trained using the data set to achieve the required corresponding accuracy, thereby obtaining the trained action recognition neural network.

6. A device for the vision-based production line control method according to claim 1, characterized in that: include: An acquisition module, used for acquiring a first real-time video stream collected by a network camera, wherein there is at least one network camera, and the field of view of each network camera only includes an open area on the production line, and the open areas do not intersect with each other; A first prediction module, configured to perform a classification prediction on the current key frame I0 every T frames by using a trained pick-and-place recognition neural network in the first real-time video stream; The first judgment module is used to continue the next prediction if the neural network classification output is a normal pick-and-place class; The second judgment module is used to continue collecting the next key frame I1 if the neural network classification output is an abnormal pick-and-place class, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F; The third judgment module is used to determine that if the proportion of the interval [θ1, θ2] consisting of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the directional distribution of F is greater than the take-out recognition threshold G, then the violation is the take-out product; The fourth judgment module is used to determine that if the proportion of the angle distribution of F falling within the interval [θ3, θ4] formed by the lower limit of the insertion angle θ3 and the upper limit of the insertion angle θ4 is greater than the insertion recognition threshold P, then this violation is the insertion of the product; an acquisition and storage module, configured to acquire a first real-time event stream collected by an event camera, wherein the field of view of the event camera includes an active area near the production line, accumulate each E events in the first real-time event stream into an event frame image, and store the event frame image in a time window queue; The second prediction module is used to predict the time window queue through a trained action recognition neural network to identify whether a dangerous action occurs.

7. A system for the vision-based production line control method according to claim 1, characterized in that: include: At least one network camera, used for collecting the first real-time video stream, wherein the field of view of each network camera only includes the open areas on the production line, and the open areas do not intersect each other; An event camera, used to collect a first real-time event stream, wherein a field of view of the event camera includes an active area near the production line; A switch, configured to transmit the first real-time video stream to a processor; A processor is used to obtain a first real-time video stream captured by a network camera, and in the first real-time video stream, a classification prediction is performed on the current key frame I0 every T frames through a trained pick-and-place recognition neural network; if the classification output of the neural network is a normal working class, the next prediction is continued; If the neural network classification output is abnormal pick-and-place class, continue to collect the next key frame I1, calculate the dense optical flow F through I0 and I1, and count the direction distribution of F; if the proportion of the interval [θ1,θ2] composed of the lower bound θ1 of the take-out angle and the upper bound θ2 of the take-out angle in the direction distribution of F is greater than the take-out recognition threshold G, then this violation anomaly is the take-out product; if the proportion of the interval [θ3,θ4] composed of the lower bound θ3 of the put-in angle and the upper bound θ4 of the put-in angle in the direction distribution of F is greater than the put-in recognition threshold P, then this violation anomaly is the put-in product; obtain the first real-time event stream collected by the event camera, accumulate each E event in the first real-time event stream into an event frame image, and store the event frame image in the time window queue; The time window queue is predicted by a trained action recognition neural network to identify whether a dangerous action occurs.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A processor-readable storage medium having processor instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images