Foreign matter deviation detection method, detection device, and foreign matter deviation detection system

Abstract

The application provides a foreign matter deviation detection method, a detection device and a foreign matter deviation detection system. The method comprises the following steps: acquiring a plurality of original images; training the plurality of original images by using a deep convolutional neural network to obtain an image detection model; acquiring a current image of a conveying belt surface in real time and inputting the current image into the image detection model to output target detection frame information corresponding to an article in the current image; detecting a target position of the article according to the target detection frame information; and in the case that the distance between the target position and the center line of the conveying belt is greater than a predetermined distance, controlling an alarm information to be sent. The method solves the problem that in the prior art, the conveying machine is affected by the working environment, it is difficult for an inspection personnel to comprehensively and quickly detect the deviation phenomenon, and the foreign matter deviation leads to falling accidents or damage to the equipment.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection technology, and more specifically, to a method, device, computer-readable storage medium, and system for detecting foreign object deviation. Background Technology

[0002] Belt conveyors, as the main transportation method in underground and open-pit coal mines, are the preferred equipment for the continuous transport of bulk materials and are currently developing towards larger scale, higher intelligence, and greater energy efficiency. Among them, the conveyor belt, as one of the most important components of a belt conveyor, plays a crucial role in carrying and traction of materials.

[0003] However, when conveying materials on a belt conveyor, the conveyor belt pulls the material being conveyed on the upper surface. During the pulling process, the material is affected by the vibration frequency of the conveyor belt and sways and deviates. This can easily cause foreign objects in the material to deviate or even fall off during the conveying process. Due to the influence of the working environment, it is difficult for inspection personnel to detect deviations comprehensively and quickly, which can lead to foreign objects falling off or damaging the equipment. Summary of the Invention

[0004] The main objective of this application is to provide a method, device, computer-readable storage medium, and system for detecting foreign object deviation, so as to at least solve the problem in the prior art where the conveyor is subject to the influence of the working environment, making it difficult for inspection personnel to fully and quickly detect deviation, which leads to foreign objects falling or damaging the equipment.

[0005] To achieve the above objectives, according to one aspect of this application, a method for detecting foreign object deviation is provided. The method includes: acquiring multiple original images, wherein the original images are images of the surface of a conveyor belt; training the multiple original images using a deep convolutional neural network to obtain an image detection model, wherein the training data of the image detection model consists of the multiple original images and corresponding detection box information, wherein the detection box information includes the position coordinates of the detection box, the width of the detection box, and the height of the detection box; acquiring a current image of the surface of the conveyor belt in real time and inputting the current image into the image detection model, and outputting target detection box information corresponding to an object in the current image; detecting the target position of the object based on the target detection box information; and, if the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, controlling the issuance of an alarm message, wherein the alarm message indicates that the object has deviated from the conveyor belt.

[0006] Optionally, an image detection model is established by training multiple original images using a deep convolutional neural network, including: obtaining the convolution function, activation function, pooling function, and fully connected function of the deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, an activation layer, a pooling layer, a fully connected layer, and an output layer, and the convolution function, the pooling function, and the fully connected function are functions for calculating the outputs of the convolutional layer, the activation layer, the pooling layer, and the fully connected layer, respectively; determining a target function based on the convolution function, the pooling function, and the fully connected function, wherein the target function is the detection box information corresponding to the original image calculated based on the original image; using batch stochastic gradient descent to substitute multiple sets of training data into the target function for iterative training until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value, thereby obtaining optimized parameters, wherein the optimized parameters are the parameters obtained after training the parameters of the target function; and determining the image detection model using the deep convolutional neural network with the optimized parameters.

[0007] Optionally, the training data further includes a probability array, which represents the probability that an item in the current image belongs to each category and the probability that the current image does not contain the item. Before iteratively training by substituting multiple sets of training data into the objective function using batch stochastic gradient descent until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value and the optimized parameters are obtained, the method further includes: calculating the true probability corresponding to the true classification of the item in the image through the fully connected layer, where the true classification is the actual category to which the item belongs; determining the loss function of the probability array based on the true classification and the true probability of the item to obtain a first loss function; determining the loss function of the detection box based on the theoretical detection box information and the actual detection box information corresponding to the true classification to obtain a second loss function, where the theoretical detection box information is the detection box information corresponding to the item obtained during training, and the actual detection box information is the actual detection box information corresponding to the item; and summing the first loss function and the second loss function to determine the final loss function.

[0008] Optionally, the first loss function and the second loss function are summed to determine the loss function, including: when the current image does not contain the item, the loss calculation of the detection box is not performed, and the first loss function is determined as the loss function.

[0009] Optionally, before detecting the target location of the item based on the target detection box information, the method further includes: acquiring a current image of the surface of the conveyor belt and inputting the current image into the image detection model, outputting the target detection box information corresponding to the item in the current image and the probability array; acquiring multiple detection box data of the previous n frames relative to the current image, the detection box data including the probability array and the detection box information corresponding to the previous n frames, each frame of the image corresponding to the probability array and the detection box information; projecting the detection box data corresponding to the previous n frames of the image onto the current image; if the projection error is less than a set error threshold, determining that the current image is associated with the previous n frames of the image, and obtaining a tracking sequence of the item in the current image, the tracking sequence including multiple detection box data; and using the argmax function to calculate the category of the item in the current image.

[0010] Optionally, before issuing an alarm and performing automatic correction when the distance between the target location and the centerline of the conveyor belt is greater than a predetermined distance, the method further includes: calculating and processing multiple straight lines of the conveyor belt using the Hough transform method based on the image of the item, wherein the direction of the straight lines is along the rolling direction of the conveyor belt; processing the multiple straight lines by straight line filtering to obtain two edge lines of the conveyor belt, wherein the distance between the two edge lines is the width of the conveyor belt; and calculating the centerline of the conveyor belt based on the two edge lines.

[0011] Optionally, after training an image detection model on multiple original images using a deep convolutional neural network, the method further includes: acquiring a current image of the surface of the conveyor belt in real time and inputting the current image into the image detection model, outputting the target detection box information corresponding to the item in the current image and the probability array; monitoring the movement trajectory of the item in real time based on the target detection box information and the probability array; calculating the distance difference with the center line in real time based on the movement trajectory; and determining that the item is shifting when the distance difference is greater than a distance threshold, and controlling the issuance of a warning message, the warning message being used to indicate that the item is shifting and has a risk of falling.

[0012] According to another aspect of this application, a foreign object deviation detection device is provided. The device includes: a first acquisition unit for acquiring multiple original images, wherein the original images are images of the surface of a conveyor belt; a training unit for training an image detection model using a deep convolutional neural network on the multiple original images, wherein the training data of the image detection model consists of the multiple original images and corresponding detection box information, wherein the detection box information includes the position coordinates of the detection box, the width of the detection box, and the height of the detection box; a first output unit for acquiring a current image of the surface of the conveyor belt in real time and inputting the current image into the image detection model, and outputting target detection box information corresponding to an object in the current image; a detection unit for detecting the target position of the object based on the target detection box information; and a first control unit for controlling the issuance of an alarm message when the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, wherein the alarm message indicates that the object has deviated from the conveyor belt.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0014] According to another aspect of this application, a foreign object deviation detection system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing a method.

[0015] Applying the technical solution of this application, in the foreign object deviation detection method, the method includes: First, acquiring multiple original images, wherein the original images are images of the surface of a conveyor belt; then, using a deep convolutional neural network to train the multiple original images to obtain an image detection model, wherein the training data of the image detection model consists of the multiple original images and corresponding detection box information, wherein the detection box information includes the position coordinates of the detection box, the width and height of the detection box; then, acquiring the current image of the surface of the conveyor belt in real time and inputting the current image into the image detection model, outputting the target detection box information corresponding to the object in the current image; then, detecting the target position of the object based on the target detection box information; finally, if the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, controlling the issuance of an alarm message, wherein the alarm message indicates that the object has deviated from the conveyor belt. This application involves acquiring image data from the upper surface of the conveyor belt during the production process. Object information is established through object classification and detection using a deep learning algorithm. Item tracking and positioning are then performed based on this information, along with linear motion monitoring and analysis. This includes determining the item's trajectory and distance from the conveyor belt's centerline. If the item exceeds a predetermined distance, an alarm is triggered, and corrective action is taken. This application solves the problem in existing technologies where conveyor belts are limited by the working environment, making it difficult for inspection personnel to comprehensively and quickly detect deviations, leading to foreign object misalignment, accidental falling, or equipment damage. Attached Figure Description

[0016] Figure 1 A hardware structure block diagram of a mobile terminal performing a foreign object deviation detection method according to an embodiment of this application is shown.

[0017] Figure 2 A flowchart illustrating a foreign object deviation detection method according to an embodiment of this application is shown.

[0018] Figure 3 A schematic diagram of an identification, positioning, and tracking architecture provided in an embodiment of this application is shown;

[0019] Figure 4 A schematic diagram of the installation structure of a foreign object deviation detection device provided in the embodiments of this application is shown;

[0020] Figure 5 A schematic diagram illustrating the image detection model processing image data according to an embodiment of this application is shown;

[0021] Figure 6 A structural block diagram of a foreign object deviation detection device provided according to an embodiment of this application is shown.

[0022] The above figures include the following reference numerals:

[0023] 102. Processor; 104. Memory; 106. Transmission equipment; 108. Input / output equipment; 1. Image acquisition equipment; 2. Idler frame; 3. Conveyor belt. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0028] Hough Transform: A feature extraction technique widely used in image analysis, computer vision, and digital image processing to identify features in objects, such as lines.

[0029] As described in the background section, in the prior art, the conveyor belt pulls the material conveyed on the upper surface. During the pulling process, the material is affected by the vibration frequency of the conveyor belt and sways and deviates, which can easily cause foreign objects in the material to deviate or even fall off during the conveying process. In order to solve the problem that the conveyor is constrained by the working environment and it is difficult for inspection personnel to detect deviation phenomenon comprehensively and quickly, resulting in foreign objects deviating and falling off or damaging the equipment, the embodiments of this application provide a foreign object deviation detection method, detection device, computer-readable storage medium and foreign object deviation detection system.

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0031] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a foreign object deviation detection method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] This embodiment provides a method for detecting foreign object deviation that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 2 This is a flowchart of a foreign object deviation detection method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0035] Step S201: Acquire multiple raw images, which are images of the surface of the conveyor belt.

[0036] Specifically, multiple image acquisition devices can be installed above the conveyor belt to acquire multiple raw images, and then the image acquisition devices send the multiple raw images to the processing device.

[0037] Step S202: A deep convolutional neural network is used to train multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box.

[0038] Specifically, a deep convolutional neural network model is constructed using the aforementioned multiple original images. The deep convolutional neural network model is then iteratively trained using the aforementioned multiple original images to obtain an image detection model, which outputs the corresponding detection box information in the image.

[0039] Step S203: Acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image.

[0040] Specifically, the image of the surface of the conveyor belt is acquired in real time, i.e., the current image, and the current image is input into the image detection model to output the target detection box information corresponding to the items in the current image.

[0041] Step S204: Detect the target location of the above-mentioned item based on the target detection box information;

[0042] Specifically, the target position of the object can be obtained based on the position coordinate information in the target detection box information, and the width and height of the detection box are the size of the object in the image.

[0043] Step S205: If the distance between the target position and the center line of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the item has deviated from the conveyor belt.

[0044] Specifically, taking half the width of the aforementioned conveyor belt as the baseline, i.e. the center line of the aforementioned conveyor belt, if the distance between the target position and the center line is greater than a predetermined distance, it indicates that the aforementioned item has deviated from the safe area of ​​the aforementioned conveyor belt. At this time, an alarm message is issued to remind the operator, and the intelligent analysis device sends a current signal or a switch signal, which is transmitted through the field network to the field PLC system and the intelligent analysis device in the central control room. The PLC system controls the automatic deviation correction action to achieve automatic deviation correction and avoid safety production accidents caused by foreign objects deviating from the correct path.

[0045] In the above embodiments, firstly, multiple original images are acquired, which are images of the surface of the conveyor belt. Then, a deep convolutional neural network is used to train the multiple original images to obtain an image detection model. The training data of the image detection model consists of the multiple original images and corresponding detection box information. The detection box information includes the position coordinates, width, and height of the detection box. Next, the current image of the conveyor belt surface is acquired in real time and input into the image detection model, outputting the target detection box information corresponding to the item in the current image. Then, the target position of the item is detected based on the target detection box information. Finally, if the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, an alarm is issued, indicating that the item has deviated from the conveyor belt. This application, through image data acquisition of the upper surface of the conveyor belt during the production process, establishes item information based on object classification detection using a deep learning algorithm. At this time, item tracking and positioning are performed based on the item information, while simultaneously performing straight-line monitoring analysis, including determining the item's trajectory and distance from the centerline of the conveyor belt. When the distance exceeds a predetermined distance, an alarm is issued and correction is performed. This application solves the problem in the prior art where the conveyor is constrained by the working environment, making it difficult for inspection personnel to comprehensively and quickly detect deviation, which can lead to foreign objects falling or damaging the equipment.

[0046] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the foreign object deviation detection method of this application will be described in detail below with reference to specific embodiments.

[0047] To improve the accuracy and reliability of image detection, in an optional implementation, step S202 includes:

[0048] Step S2021: Obtain the convolution function, activation function, pooling function, and fully connected function of the deep convolutional neural network. The deep convolutional data network includes an input layer, a convolutional layer, an activation layer, a pooling layer, a fully connected layer, and an output layer. The convolution function, pooling function, and fully connected function are functions for calculating the outputs of the convolutional layer, activation layer, pooling layer, and fully connected layer, respectively.

[0049] Specifically, the corresponding functions are obtained based on the neural network's convolutional layers, activation layers, pooling layers, and fully connected layers.

[0050] Step S2022: Determine the target function based on the above convolution function, the above pooling function and the above fully connected function. The target function is the detection box information corresponding to the above original image calculated based on the above original image.

[0051] Specifically, multiple sets of the above training data are input into the above input layer, and then processed sequentially through the above convolution function, the above pooling function, and the above fully connected function to calculate the target function, which is finally obtained through the above output layer.

[0052] Step S2023: The batch stochastic gradient descent method is used to substitute multiple sets of training data into the above objective function for iterative training until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value, and the optimized parameters are obtained. The optimized parameters are the parameters obtained after training the parameters of the above objective function.

[0053] Specifically, the original image is substituted into the above objective function using batch stochastic gradient descent for iterative training. In each iteration, the loss function value of the deep convolutional neural network model is calculated, which is the difference between the output of the objective function and the output data corresponding to the original image. During the iterative training process, the parameters of the deep convolutional neural network model are continuously adjusted using batch stochastic gradient descent to gradually reduce the loss function value, thereby helping to optimize the efficiency and results of parameter training. Finally, the optimized parameters, i.e. the parameters of the objective function, are obtained after training.

[0054] Step S2024: The deep convolutional neural network with the above-mentioned optimized parameters is used to determine the above-mentioned image detection model.

[0055] Specifically, after multiple iterations of training, optimized parameters were obtained. These optimized parameters were then substituted into the function of the deep convolutional neural network to obtain an optimized deep convolutional neural network model, which was then identified as the aforementioned image detection model.

[0056] To improve the accuracy and robustness of the model, in an optional implementation, before step S2023, the method further includes:

[0057] Step S301: The true probability corresponding to the true classification of the items in the image is calculated through the fully connected layer mentioned above. The true classification is the category to which the items actually belong.

[0058] Specifically, the probability of an item in an image belonging to a category is calculated through a fully connected layer. Among the multiple category probabilities, the highest probability value is the true probability p corresponding to the true category u.

[0059] Step S302: Determine the loss function of the probability array based on the true classification and true probability of the items to obtain the first loss function.

[0060] Specifically, the loss function for the probability array is determined based on the true classification u and the true probability p of the aforementioned items, for example: L cls (p,u)=-logp u This refers to the first loss function mentioned above. Of course, those skilled in the art can choose other functional expressions to represent the loss function of the probability array.

[0061] Step S303: Determine the loss function of the detection box based on the theoretical detection box information and the real detection box information corresponding to the real classification, and obtain the second loss function. The theoretical detection box information is the detection box information obtained during the training process corresponding to the item, and the real detection box information is the actual detection box information corresponding to the item.

[0062] Specifically, the bounding box regression loss function needs to compare the theoretical bounding box information corresponding to the true classification. and the actual detection box information v = (v x ,v y ,v w ,v h ), where x is the horizontal coordinate of the detection box, y is the vertical coordinate of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0063] Step S304: Sum the first loss function and the second loss function to determine the loss function.

[0064] Specifically, the final loss function is a weighted sum of the two loss functions, that is, the sum of the first loss function and the second loss function mentioned above, as shown in the example: Here, α is a parameter used to control the balance between the probability array loss and the detection box regression loss.

[0065] To save computing resources, in an optional implementation, step S304 includes:

[0066] Step S3041: If the current image does not contain the aforementioned items, the loss calculation of the detection box is not performed, and the first loss function is determined as the aforementioned loss function.

[0067] Specifically, u = 0 is defined as the background class (negative label), which typically refers to background regions in the image. Since these regions are not the target of interest, there is no need to perform bounding box regression operations to adjust their position and size. Therefore, background candidate regions are not included in the regression loss to save computational resources and avoid unnecessary operations on irrelevant regions.

[0068] To improve the safety and stability of item transport on the production line, in one optional embodiment, after step S202, the method further includes:

[0069] Step S401: Acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image and the probability array.

[0070] Specifically, in order to calculate the motion trajectory of each item, the current image of the surface of the conveyor belt is acquired in real time and input into the image detection model. The output is the target detection box information and the probability array corresponding to the item in the current image.

[0071] Step S402: Based on the target detection box information and the probability array, the motion trajectory of the object is obtained in real time.

[0072] Specifically, the category of an item is determined based on its probability array, and its position is determined based on the object detection box information. The movement trajectory of the item is obtained in real time through both methods.

[0073] Step S403: Calculate the distance difference between the above-mentioned motion trajectory and the above-mentioned center line in real time.

[0074] Specifically, the distance difference between the device and the center line of the transport belt is calculated in real time based on the movement trajectory.

[0075] Step S404: If the distance difference is greater than the distance threshold, it is determined that the item is shifting, and a warning message is issued to indicate that the item is shifting and there is a risk of it falling.

[0076] Specifically, if the distance difference exceeds the distance threshold, it indicates that the item is shifting and at risk of falling. In this case, the control system will issue a warning message to alert the operator.

[0077] To reduce false positives and improve detection accuracy, in one optional implementation, such asFigure 3 As shown, before step S204 above, the method further includes:

[0078] Step S501: Obtain the current image of the surface of the transport belt and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image and the probability array.

[0079] Specifically, since the same material will appear in the camera's field of view multiple times, and the detection algorithm used processes each frame of image data, resulting in multiple object detection results for the same material, the position and category of the current image are first obtained. The current image of the surface of the conveyor belt is obtained in real time and the current image is input into the image detection model. The output is the target detection box information corresponding to the item in the current image and the probability array.

[0080] Step S502: Obtain multiple detection box data for the previous n frames relative to the current image. The detection box data includes the probability array and detection box information corresponding to the previous n frames. Each frame of the image corresponds one-to-one with the probability array and the detection box information.

[0081] Specifically, in order to obtain accurate and robust material category and location of the current image, the detection box data of the previous n frames of images is obtained by combining camera calibration parameters and conveyor belt speed. The detection box data of the previous n frames of images will be stored in the camera device memory or can be obtained directly.

[0082] Step S403: Project the detection box data corresponding to the previous n frames of the above images onto the current image.

[0083] Specifically, using the position information of the detection boxes in the previous n frames of images, the motion trajectory of the detection boxes is calculated by motion estimation algorithms (such as optical flow, Kalman filtering, etc.). Based on the calculated motion trajectory, the detection boxes in the previous n frames of images are projected along their motion direction to calculate the position of the detection boxes in the current image. The projected detection box data is then superimposed on the current image to display the motion trajectory of the detection boxes in the previous n frames of images. Image processing software or programming languages ​​can be used to implement the projection and superposition of the detection box data.

[0084] Step S504: If the projection error is less than a set error threshold, determine that the current image is associated with the previous n frames of the image, and obtain the tracking sequence of the item in the current image, wherein the tracking sequence includes multiple detection box data.

[0085] Specifically, by tracking the movement trajectory of materials, the number of times the same material is detected multiple times can be reduced, thereby reducing the possibility of false detections. When the projection error is less than a set error threshold, it can be determined that the current image is correlated with the previous n frames. This allows for the acquisition of tracking sequences for different materials. Each tracking sequence contains multiple detection boxes, and each detection box data consists of 5 data points (p, t). x ,t y ,t w ,t h ).

[0086] Step S505: The category of the item in the current image is calculated using the argmax function on the above tracking sequence.

[0087] Specifically, the category at the current moment is obtained by voting on the tracking sequence. in Let T represent the probability that the detection box belongs to category j at time i, T represent the time of the current image, and i represent the time of the previous n frames.

[0088] To more accurately control and monitor the movement of items on the conveyor belt surface, in an optional embodiment, prior to step S205, the method further includes:

[0089] Step S601: Based on the image of the above-mentioned items, the Hough transform method is used to calculate and process the data to obtain multiple straight lines of the above-mentioned transport belts, the direction of the above-mentioned straight lines being along the rolling direction of the above-mentioned transport belts.

[0090] Specifically, the image containing the aforementioned items is processed using the Hough transform method to obtain all the straight lines in the image, retaining a portion of the straight lines whose direction is along the rolling direction of the aforementioned conveyor belt.

[0091] Step S602: The multiple straight lines are processed by straight line filtering to obtain two edge lines of the conveyor belt, and the distance between the two edge lines is the width of the conveyor belt.

[0092] Specifically, by using methods such as linear filtering, the two edge lines of the conveyor belt can be accurately obtained, thereby enabling precise control and monitoring of surface items within the conveyor belt area on automated production lines.

[0093] Step S603: Calculate the center line of the transport belt based on the two edge lines.

[0094] Specifically, the central axis of the two symmetrical edge lines is the center line of the aforementioned conveyor belt.

[0095] This embodiment relates to a specific method for detecting foreign object deviation, such as...Figure 4 and Figure 5 As shown, it includes the following steps:

[0096] Step S1: Multiple image acquisition devices 1 acquire image data of the upper surface of the conveyor belt 3;

[0097] Step S2: Classify and detect objects using a deep learning-based object detection system to establish item information;

[0098] Step S3: After the original image data passes through the convolutional layer, a feature map is obtained. For each data unit on the feature map, the pooling layer generates different candidate detection boxes with a fixed template, i.e., candidate region generation. The output is usually a 7x7 or 9x9 feature map. After passing through the fully connected layer, the classification score (the above probability array) and detection box regression (detection box information) are obtained.

[0099] Step S4: Track the item based on the item information established by multiple image acquisition devices 1 within a certain time period, and simultaneously monitor and analyze the item's location during the tracking process;

[0100] Step S5: Based on the position of the item and the distance between it and the roller frames 2 on both sides, a judgment is made. When the predetermined threshold is exceeded, it is shown that the foreign object has deviated. At the same time, the early warning unit sends an alarm signal to remind the on-duty personnel and transmits it to the on-site PLC system and the intelligent analysis equipment in the central control room through the on-site network. The PLC system controls the automatic error correction device to perform automatic error correction processing.

[0101] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0102] This application also provides a foreign object deviation detection device. It should be noted that the foreign object deviation detection device of this application can be used to execute the foreign object deviation detection method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0103] The following describes the foreign object deviation detection device provided in the embodiments of this application.

[0104] Figure 6 This is a structural block diagram of a foreign object deviation detection device according to an embodiment of this application. Figure 6As shown, the device includes:

[0105] The first acquisition unit 10 is used to acquire multiple original images, which are images of the surface of the conveyor belt.

[0106] Specifically, multiple image acquisition devices can be installed above the conveyor belt to acquire multiple raw images, and then the image acquisition devices send the multiple raw images to the processing device.

[0107] The training unit 20 is used to train a deep convolutional neural network on multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box.

[0108] Specifically, a deep convolutional neural network model is constructed using the aforementioned multiple original images. The deep convolutional neural network model is then iteratively trained using the aforementioned multiple original images to obtain an image detection model, which outputs the corresponding detection box information in the image.

[0109] The first output unit 30 is used to acquire the current image of the surface of the transport belt in real time, input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image.

[0110] Specifically, the image of the surface of the conveyor belt is acquired in real time, i.e., the current image, and the current image is input into the image detection model to output the target detection box information corresponding to the items in the current image.

[0111] Detection unit 40 is used to detect the target position of the above-mentioned item based on the target detection box information;

[0112] Specifically, the target position of the object can be obtained based on the position coordinate information in the target detection box information, and the width and height of the detection box are the size of the object in the image.

[0113] The first control unit 50 is configured to issue an alarm message when the distance between the target position and the center line of the transport belt is greater than a predetermined distance. The alarm message indicates that the item has deviated from the transport belt.

[0114] Specifically, taking half the width of the aforementioned conveyor belt as the baseline, i.e. the center line of the aforementioned conveyor belt, if the distance between the target position and the center line is greater than a predetermined distance, it indicates that the aforementioned item has deviated from the safe area of ​​the aforementioned conveyor belt. At this time, an alarm message is issued to remind the operator, and the intelligent analysis device sends a current signal or a switch signal, which is transmitted through the field network to the field PLC system and the intelligent analysis device in the central control room. The PLC system controls the automatic deviation correction action to achieve automatic deviation correction and avoid safety production accidents caused by foreign objects deviating from the correct path.

[0115] In this embodiment, a first acquisition unit is used to acquire multiple original images, which are images of the surface of the conveyor belt; a training unit is used to train an image detection model using a deep convolutional neural network on the multiple original images, the training data of the image detection model being the multiple original images and corresponding detection box information, the detection box information including the position coordinates of the detection box, the width and height of the detection box; a first output unit is used to acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, outputting the target detection box information corresponding to the item in the current image; a detection unit is used to detect the target position of the item based on the target detection box information; a first control unit is used to control the issuance of an alarm message when the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, the alarm message indicating that the item has deviated from the conveyor belt. This application, through image data acquisition of the upper surface of the conveyor belt during the production process, establishes item information based on object classification detection using a deep learning algorithm, tracks and locates the item based on the item information, and simultaneously performs straight-line monitoring analysis, including determining the item's trajectory and distance from the centerline of the conveyor belt. When the distance exceeds a predetermined distance, an alarm message is issued and correction is performed. This application solves the problem in the prior art where the conveyor is constrained by the working environment, making it difficult for inspection personnel to comprehensively and quickly detect deviation, which can lead to foreign objects falling or damaging the equipment.

[0116] To improve the accuracy and reliability of image object detection, in one optional implementation, the training unit includes:

[0117] The acquisition module acquires the convolution function, activation function, pooling function, and fully connected function of the aforementioned deep convolutional neural network. The aforementioned deep convolutional data network includes an input layer, a convolutional layer, an activation layer, a pooling layer, a fully connected layer, and an output layer. The aforementioned convolutional function, pooling function, and fully connected function are functions for calculating the outputs of the aforementioned convolutional layer, activation layer, pooling layer, and fully connected layer, respectively.

[0118] Specifically, the corresponding functions are obtained based on the neural network's convolutional layers, activation layers, pooling layers, and fully connected layers.

[0119] The first determining module determines the target function based on the above-mentioned convolution function, pooling function, and fully connected function. The target function is the detection box information corresponding to the above-mentioned original image calculated based on the above-mentioned original image.

[0120] Specifically, multiple sets of the above training data are input into the above input layer, and then processed sequentially through the above convolution function, the above pooling function, and the above fully connected function to calculate the target function, which is finally obtained through the above output layer.

[0121] The training module uses batch stochastic gradient descent to substitute multiple sets of training data into the above objective function for iterative training until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value, thereby obtaining optimized parameters. The optimized parameters are the parameters obtained after training the parameters of the above objective function.

[0122] Specifically, the original image is substituted into the above objective function using batch stochastic gradient descent for iterative training. In each iteration, the loss function value of the deep convolutional neural network model is calculated, which is the difference between the output of the objective function and the output data corresponding to the original image. During the iterative training process, the parameters of the deep convolutional neural network model are continuously adjusted using batch stochastic gradient descent to gradually reduce the loss function value, thereby helping to optimize the efficiency and results of parameter training. Finally, the optimized parameters, i.e. the parameters of the objective function, are obtained after training.

[0123] The second determining module determines the image detection model using the deep convolutional neural network with the above-mentioned optimized parameters.

[0124] Specifically, after multiple iterations of training, optimized parameters were obtained. These optimized parameters were then substituted into the function of the deep convolutional neural network to obtain an optimized deep convolutional neural network model, which was then identified as the aforementioned image detection model.

[0125] To improve the accuracy and robustness of the model, in one optional implementation, the above-mentioned apparatus further includes:

[0126] The first computing unit, after iteratively training multiple sets of training data into the above objective function using the batch stochastic gradient descent method until the loss function value of the above deep convolutional neural network is less than or equal to a predetermined value and before obtaining the optimized parameters, calculates the true probability corresponding to the true classification of the items in the image through the above fully connected layer. The above true classification is the category to which the above items actually belong.

[0127] Specifically, the probability of an item in an image belonging to a category is calculated through a fully connected layer. Among the multiple category probabilities, the highest probability value is the true probability p corresponding to the true category u.

[0128] The first determining unit determines the loss function of the probability array based on the true classification and true probability of the items, thus obtaining the first loss function.

[0129] Specifically, the loss function for the probability array is determined based on the true classification u and the true probability p of the aforementioned items, for example: L cls (p,u)=-logp u This refers to the first loss function mentioned above. Of course, those skilled in the art can choose other functional expressions to represent the loss function of the probability array.

[0130] The second determining unit determines the loss function of the detection box based on the theoretical detection box information and the real detection box information corresponding to the real classification, and obtains the second loss function. The theoretical detection box information is the detection box information obtained during the training process corresponding to the item, and the real detection box information is the actual detection box information corresponding to the item.

[0131] Specifically, the bounding box regression loss function needs to compare the theoretical bounding box information corresponding to the true classification. and the actual detection box information v = (v x ,v y ,v w ,v h ), where x is the horizontal coordinate of the detection box, y is the vertical coordinate of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0132] The third determining unit sums the first loss function and the second loss function to determine the loss function.

[0133] Specifically, the final loss function is a weighted sum of the two loss functions, that is, the sum of the first loss function and the second loss function mentioned above, as shown in the example: Here, α is a parameter used to control the balance between the probability array loss and the detection box regression loss.

[0134] To save computing resources, in one optional implementation, the third determining unit includes:

[0135] The third determining module, in the case that the current image does not contain the aforementioned items, does not perform the loss calculation of the aforementioned detection box, and determines the aforementioned first loss function as the aforementioned loss function.

[0136] Specifically, u = 0 is defined as the background class (negative label), which typically refers to background regions in the image. Since these regions are not the target of interest, there is no need to perform bounding box regression operations to adjust their position and size. Therefore, background candidate regions are not included in the regression loss to save computational resources and avoid unnecessary operations on irrelevant regions.

[0137] To improve the safety and stability of material transport on the production line, in one optional embodiment, the above-mentioned device further includes:

[0138] The second output unit, after training an image detection model using a deep convolutional neural network on multiple original images, wherein the training data of the image detection model consists of multiple original images and corresponding detection box information, acquires the current image of the surface of the conveyor belt in real time and inputs the current image into the image detection model, and outputs the target detection box information corresponding to the items in the current image and the probability array.

[0139] Specifically, in order to calculate the motion trajectory of each item, the current image of the surface of the conveyor belt is acquired in real time and input into the image detection model. The output is the target detection box information and the probability array corresponding to the item in the current image.

[0140] The monitoring unit is used to monitor and obtain the movement trajectory of the above-mentioned items in real time based on the above-mentioned target detection box information and the above-mentioned probability array.

[0141] Specifically, the category of an item is determined based on its probability array, and its position is determined based on the object detection box information. The movement trajectory of the item is obtained in real time through both methods.

[0142] The second calculation unit is used to calculate the distance difference between the above-mentioned motion trajectory and the above-mentioned center line in real time.

[0143] Specifically, the distance difference between the device and the center line of the transport belt is calculated in real time based on the movement trajectory.

[0144] The second control unit is used to determine that the item is shifting when the distance difference is greater than the distance threshold, and to issue a warning message that the item is shifting and there is a risk of it falling.

[0145] Specifically, if the distance difference exceeds the distance threshold, it indicates that the item is shifting and at risk of falling. In this case, the control system will issue a warning message to alert the operator.

[0146] To reduce false detections and improve detection accuracy, in one optional embodiment, the above-mentioned device further includes:

[0147] The second output unit is used to acquire the current image of the surface of the conveyor belt and input the current image into the image detection model before detecting the target position of the item based on the target detection box information, and output the target detection box information and probability array corresponding to the item in the current image.

[0148] Specifically, since the same material will appear in the camera's field of view multiple times, and the detection algorithm used processes each frame of image data, resulting in multiple object detection results for the same material, the position and category of the current image are first obtained. The current image of the surface of the conveyor belt is obtained in real time and the current image is input into the image detection model. The output is the target detection box information corresponding to the item in the current image and the probability array.

[0149] The first acquisition unit is used to acquire multiple detection box data of the previous n frames relative to the current image. The detection box data includes the probability array and the detection box information corresponding to the previous n frames. Each frame of the image corresponds one-to-one with the probability array and the detection box information.

[0150] Specifically, in order to obtain accurate and robust material category and location of the current image, the detection box data of the previous n frames of images is obtained by combining camera calibration parameters and conveyor belt speed. The detection box data of the previous n frames of images will be stored in the camera device memory or can be obtained directly.

[0151] The projection unit is used to project the detection box data corresponding to the previous n frames of the above images onto the current image.

[0152] Specifically, using the position information of the detection boxes in the previous n frames of images, the motion trajectory of the detection boxes is calculated by motion estimation algorithms (such as optical flow, Kalman filtering, etc.). Based on the calculated motion trajectory, the detection boxes in the previous n frames of images are projected along their motion direction to calculate the position of the detection boxes in the current image. The projected detection box data is then superimposed on the current image to display the motion trajectory of the detection boxes in the previous n frames of images. Image processing software or programming languages ​​can be used to implement the projection and superposition of the detection box data.

[0153] The fourth determining unit is used to determine that the current image is associated with the previous n frames of the image when the projection error is less than a set error threshold, and to obtain the tracking sequence of the item in the current image, wherein the tracking sequence includes multiple detection box data.

[0154] Specifically, by tracking the movement trajectory of materials, the number of times the same material is detected multiple times can be reduced, thereby reducing the possibility of false detections. When the projection error is less than a set error threshold, it can be determined that the current image is correlated with the previous n frames. This allows for the acquisition of tracking sequences for different materials. Each tracking sequence contains multiple detection boxes, and each detection box data consists of 5 data points (p, t). x ,t y ,t w ,t h ).

[0155] The third calculation unit is used to calculate the category of the item in the current image by using the argmax function on the above tracking sequence.

[0156] Specifically, the category at the current moment is obtained by voting on the tracking sequence. in Let T represent the probability that the detection box belongs to category j at time i, T represent the time of the current image, and i represent the time of the previous n frames.

[0157] To more accurately control and monitor the movement of items on the conveyor belt surface, in one optional embodiment, the above-mentioned device further includes:

[0158] The first processing unit is configured to, before issuing an alarm message when the distance between the target location and the centerline of the conveyor belt is greater than a predetermined distance, calculate and process multiple straight lines of the conveyor belt based on the image of the item being located using the Hough transform method, wherein the direction of the straight lines is along the rolling direction of the conveyor belt.

[0159] Specifically, the image containing the aforementioned items is processed using the Hough transform method to obtain all the straight lines in the image, retaining a portion of the straight lines whose direction is along the rolling direction of the aforementioned conveyor belt.

[0160] The second processing unit processes the multiple straight lines by linear filtering to obtain two edge lines of the conveyor belt, and the distance between the two edge lines is the width of the conveyor belt.

[0161] Specifically, by using methods such as linear filtering, the two edge lines of the conveyor belt can be accurately obtained, thereby enabling precise control and monitoring of surface items within the conveyor belt area on automated production lines.

[0162] The fourth calculation unit calculates the center line of the transport belt based on the two edge lines mentioned above.

[0163] Specifically, the central axis of the two symmetrical edge lines is the center line of the aforementioned conveyor belt.

[0164] The aforementioned foreign object deviation detection device includes a processor and a memory. The first acquisition unit, training unit, etc., are all stored as program units in the memory, and the processor executes the program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the various modules may be located in different processors in any combination.

[0165] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the problems in existing technologies regarding conveyor systems being affected by the working environment, making it difficult for inspection personnel to comprehensively and quickly detect deviations, leading to foreign objects falling or damaging the equipment.

[0166] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0167] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the foreign object deviation detection method.

[0168] Specifically, methods for detecting foreign object deviation include:

[0169] Step S201: Acquire multiple raw images, wherein the raw images are images of the surface of the conveyor belt;

[0170] Step S202: A deep convolutional neural network is used to train multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box.

[0171] Step S203: Acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image;

[0172] Step S204: Detect the target location of the above-mentioned item based on the target detection box information;

[0173] Step S205: If the distance between the target position and the center line of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the item has deviated from the conveyor belt.

[0174] This invention provides a processor for running a program, wherein the program executes the foreign object deviation detection method.

[0175] Specifically, methods for detecting foreign object deviation include:

[0176] Step S201: Acquire multiple raw images, wherein the raw images are images of the surface of the conveyor belt;

[0177] Step S202: A deep convolutional neural network is used to train multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box.

[0178] Step S203: Acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image;

[0179] Step S204: Detect the target location of the above-mentioned item based on the target detection box information;

[0180] Step S205: If the distance between the target location and the centerline of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the item has deviated from the conveyor belt. This embodiment of the invention provides a foreign object deviation detection system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: The foreign object deviation detection system described herein can be a server, PC, PAD, mobile phone, etc.

[0181] Step S201: Acquire multiple raw images, wherein the raw images are images of the surface of the conveyor belt;

[0182] Step S202: A deep convolutional neural network is used to train multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box.

[0183] Step S203: Acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image;

[0184] Step S204: Detect the target location of the above-mentioned item based on the target detection box information;

[0185] Step S205: If the distance between the target position and the center line of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the item has deviated from the conveyor belt.

[0186] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0187] Step S201: Acquire multiple raw images, wherein the raw images are images of the surface of the conveyor belt;

[0188] Step S202: A deep convolutional neural network is used to train multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box.

[0189] Step S203: Acquire the current image of the surface of the conveyor belt in real time and input the current image into the image detection model, and output the target detection box information corresponding to the items in the current image;

[0190] Step S204: Detect the target location of the above-mentioned item based on the target detection box information;

[0191] Step S205: If the distance between the target position and the center line of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the item has deviated from the conveyor belt.

[0192] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0193] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0194] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0195] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0196] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0197] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0198] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0199] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0200] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0201] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0202] 1) The foreign object deviation detection method of this application first acquires multiple original images, which are images of the surface of a conveyor belt; then, a deep convolutional neural network is used to train the multiple original images to obtain an image detection model. The training data of the image detection model consists of the multiple original images and corresponding detection box information, including the position coordinates, width, and height of the detection box; then, the current image of the conveyor belt surface is acquired in real time and input into the image detection model, outputting the target detection box information corresponding to the object in the current image; then, the target position of the object is detected based on the target detection box information; finally, if the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the object has deviated from the conveyor belt. This application acquires image data of the upper surface of the conveyor belt during the production process, establishes object information based on object classification detection using a deep learning algorithm, tracks and locates the object based on the object information, and performs straight-line monitoring analysis, including determining the object's trajectory and distance from the centerline of the conveyor belt. When the distance exceeds a predetermined distance, an alarm message is issued and deviation correction is performed. This application solves the problem in the prior art where the conveyor is constrained by the working environment, making it difficult for inspection personnel to comprehensively and quickly detect deviation, which can lead to foreign objects falling or damaging the equipment.

[0203] 2) The foreign object deviation detection device of this application includes a first acquisition unit for acquiring multiple original images, wherein the original images are images of the surface of a conveyor belt; a training unit for training an image detection model using a deep convolutional neural network on the multiple original images, wherein the training data of the image detection model consists of the multiple original images and corresponding detection box information, wherein the detection box information includes the position coordinates of the detection box, the width and height of the detection box; a first output unit for acquiring a current image of the surface of the conveyor belt in real time and inputting the current image into the image detection model, and outputting target detection box information corresponding to the object in the current image; a detection unit for detecting the target position of the object based on the target detection box information; and a first control unit for controlling the issuance of an alarm message when the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance, wherein the alarm message indicates that the object has deviated from the conveyor belt. This application involves acquiring image data from the upper surface of the conveyor belt during the production process. Object information is established through object classification and detection using a deep learning algorithm. Item tracking and positioning are then performed based on this information, along with linear motion monitoring and analysis. This includes determining the item's trajectory and distance from the conveyor belt's centerline. If the item exceeds a predetermined distance, an alarm is triggered, and corrective action is taken. This application solves the problem in existing technologies where conveyor belts are limited by the working environment, making it difficult for inspection personnel to comprehensively and quickly detect deviations, leading to foreign object misalignment, accidental falling, or equipment damage.

[0204] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for detecting foreign object deviation, characterized in that, The method includes: Acquire multiple raw images, wherein the raw images are images of the surface of the conveyor belt; An image detection model is obtained by training multiple original images using a deep convolutional neural network. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box. The current image of the surface of the conveyor belt is acquired in real time and input into the image detection model, and the target detection box information corresponding to the items in the current image is output. The target location of the item is detected based on the target detection box information; If the distance between the target location and the centerline of the conveyor belt is greater than a predetermined distance, an alarm message is issued, indicating that the item has deviated from the conveyor belt. An image detection model is established by training multiple original images using a deep convolutional neural network (DCNN), including: obtaining the convolution function, activation function, pooling function, and fully connected function of the DCNN, wherein the DCNN includes an input layer, a convolutional layer, an activation layer, a pooling layer, a fully connected layer, and an output layer, and the convolution function, pooling function, and fully connected function are functions for calculating the outputs of the convolutional layer, activation layer, pooling layer, and fully connected layer, respectively; determining a target function based on the convolution function, pooling function, and fully connected function, wherein the target function is the detection box information corresponding to the original image calculated based on the original image; iteratively training the target function using batch stochastic gradient descent by substituting multiple sets of training data into the target function until the loss function value of the DCNN is less than or equal to a predetermined value, thereby obtaining optimized parameters, wherein the optimized parameters are the parameters obtained after training the parameters of the target function; and determining the image detection model using the DCNN with the optimized parameters. The training data also includes a probability array, which represents the probability that an item in the current image belongs to each category and the probability that the current image does not contain the item. Before iteratively training by substituting multiple sets of training data into the objective function using batch stochastic gradient descent until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value and the optimized parameters are obtained, the method further includes: calculating the true probability corresponding to the true category of the item in the image through the fully connected layer, where the true category is the actual category to which the item belongs; determining the loss function of the probability array based on the true category and the true probability of the item to obtain a first loss function; determining the loss function of the detection box based on the theoretical detection box information and the actual detection box information corresponding to the true category to obtain a second loss function, where the theoretical detection box information is the detection box information corresponding to the item obtained during training, and the actual detection box information is the actual detection box information corresponding to the item; and summing the first loss function and the second loss function to determine the final loss function. Before detecting the target location of the item based on the target detection box information, the method further includes: acquiring a current image of the surface of the conveyor belt and inputting the current image into the image detection model, outputting the target detection box information corresponding to the item in the current image and the probability array; acquiring multiple detection box data of the previous n frames relative to the current image, the detection box data including the probability array and the detection box information corresponding to the previous n frames, each frame of the image corresponding to the probability array and the detection box information; projecting the detection box data corresponding to the previous n frames of the image onto the current image; if the projection error is less than a set error threshold, determining that the current image and the previous n frames of the image are associated, and obtaining a tracking sequence of the item in the current image, the tracking sequence including multiple detection box data; and using the argmax function to calculate the category of the item in the current image.

2. The method according to claim 1, characterized in that, The first loss function and the second loss function are summed to determine the loss function, which includes: If the current image does not contain the item, the loss calculation for the detection box is not performed, and the first loss function is determined as the loss function.

3. The method according to claim 1, characterized in that, Before issuing an alarm message when the distance between the target location and the centerline of the conveyor belt is greater than a predetermined distance, the method further includes: The image of the item is processed using the Hough transform method to obtain multiple straight lines of the conveyor belt, the direction of which is along the rolling direction of the conveyor belt; Multiple straight lines are processed by straight line filtering to obtain two edge lines of the conveyor belt, and the distance between the two edge lines is the width of the conveyor belt; The centerline of the transport belt is calculated based on the two edge lines.

4. The method according to claim 1, characterized in that, After training an image detection model on multiple original images using a deep convolutional neural network, the method further includes: The current image of the surface of the conveyor belt is acquired in real time and input into the image detection model. The output is the target detection box information corresponding to the items in the current image and the probability array. The motion trajectory of the object is obtained in real time based on the target detection box information and the probability array; The distance difference between the trajectory and the center line is calculated in real time. If the distance difference is greater than a distance threshold, it is determined that the item is shifting, and a warning message is issued to indicate that the item is shifting and there is a risk of it falling.

5. A foreign object deviation detection device, characterized in that, The device includes: The first acquisition unit is used to acquire multiple original images, wherein the original images are images of the surface of the conveyor belt; The training unit is used to train a deep convolutional neural network on multiple original images to obtain an image detection model. The training data of the image detection model consists of multiple original images and corresponding detection box information. The detection box information includes the position coordinates of the detection box, the width and height of the detection box. The first output unit is used to acquire the current image of the surface of the conveyor belt in real time, input the current image into the image detection model, and output the target detection box information corresponding to the item in the current image. The detection unit is used to detect the target position of the item based on the target detection box information; A first control unit is configured to issue an alarm message when the distance between the target position and the centerline of the conveyor belt is greater than a predetermined distance. The alarm message indicates that the item has deviated from the conveyor belt. The training unit includes: an acquisition module for acquiring the convolution function, activation function, pooling function, and fully connected function of the deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, an activation layer, a pooling layer, a fully connected layer, and an output layer, and the convolution function, the pooling function, and the fully connected function are functions for calculating the outputs of the convolutional layer, the activation layer, the pooling layer, and the fully connected layer, respectively; a first determination module for determining a target function based on the convolution function, the pooling function, and the fully connected function, wherein the target function is the detection box information corresponding to the original image calculated based on the original image; a training module for iteratively training the target function using batch stochastic gradient descent by substituting multiple sets of training data into the target function until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value, thereby obtaining optimized parameters, wherein the optimized parameters are the parameters obtained after training the parameters of the target function; and a second determination module for determining the image detection model using the deep convolutional neural network with the optimized parameters. The training data further includes a probability array, which represents the probability that an item in the current image belongs to each category and the probability that the current image does not contain the item. The device further includes: a first calculation unit, used to calculate the true probability corresponding to the true category of the item in the image through the fully connected layer before iteratively training by substituting multiple sets of training data into the objective function using batch stochastic gradient descent until the loss function value of the deep convolutional neural network is less than or equal to a predetermined value and before obtaining the optimization parameters; a first determination unit, used to determine the loss function of the probability array based on the true category and the true probability of the item, to obtain a first loss function; a second determination unit, used to determine the loss function of the detection box based on the theoretical detection box information and the actual detection box information corresponding to the true category, to obtain a second loss function; the theoretical detection box information is the detection box information corresponding to the item obtained during the training process, and the actual detection box information is the actual detection box information corresponding to the item; and a third determination unit, used to sum the first loss function and the second loss function to determine the loss function. The device further includes: a second output unit, configured to acquire a current image of the surface of the conveyor belt and input the current image into the image detection model before detecting the target position of the item based on the target detection box information, and output the target detection box information corresponding to the item in the current image and the probability array; a first acquisition unit, configured to acquire multiple detection box data of the previous n frames of images relative to the current image, the detection box data including the probability array and the detection box information corresponding to the previous n frames of images, each frame of the image corresponding to the probability array and the detection box information; a projection unit, configured to project the detection box data corresponding to the previous n frames of images onto the current image; a fourth determination unit, configured to determine that the current image is associated with the previous n frames of images when the projection error is less than a set error threshold, and obtain a tracking sequence of the item in the current image, the tracking sequence including multiple detection box data; and a third calculation unit, configured to calculate the category of the item in the current image using the argmax function on the tracking sequence.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 4.

7. A foreign object deviation detection system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 4.

Images