Methods, apparatus, electronic devices and computer-readable media for generating abnormal information
By extracting image sequences and target images from target videos, extracting edge information, and generating statistical graphs, the problem of high computational complexity and limited generalization ability of machine learning algorithms in conveyor belt anomaly detection is solved, and fast and effective anomaly information generation is achieved.
Patent Information
- Application Number
- CN202110414369.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-04-16
AI Technical Summary
Existing machine vision machine learning algorithms require extensive labeling and training to detect and identify anomalies in conveyor belt transportation processes. This results in high computational complexity and limited generalization ability, leading to inefficient and slow generation of anomaly information.
Image sequences and target images are extracted from the target video. Edge information is extracted to generate statistical charts representing abnormal conditions of the conveyor belt. Abnormal information is generated in a pre-set order. Pooling operations and activation function processing are performed on the edge image sequences and the corresponding edge images of the target images to generate various abnormal information of the conveyor belt.
It enables the rapid and effective identification of abnormal situations during conveyor belt transportation, reduces the need for annotation and training, lowers computational complexity, and improves generalization ability.
Smart Images

Figure CN113139451B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically to methods, apparatus, electronic devices, and computer-readable media for generating abnormal information. Background Technology
[0002] Currently, many industries rely on conveyor belts to transport objects. During this process, various abnormal situations may occur, such as object blockages or conveyor belt breakage. To detect these abnormalities, a common approach is to use machine learning models, such as neural networks, to generate information about these anomalies.
[0003] However, when using the above method to generate the above-mentioned abnormal information, the following technical problems often occur:
[0004] Machine vision-based machine learning algorithms require extensive labeling and training for detection and recognition, resulting in high computational complexity and limited generalization ability. This makes generating abnormal information about conveyor belts inefficient and slow. Summary of the Invention
[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0006] Some embodiments of this disclosure provide methods, apparatus, electronic devices, and computer-readable media for generating abnormal information to address one of the technical problems mentioned in the background section above.
[0007] In a first aspect, some embodiments of this disclosure provide an anomaly information generation method, comprising: extracting an image sequence and a target image from a target video, wherein the target image is an image of a conveyor belt without any items to be transported, and the images in the image sequence are images of items being transported on the conveyor belt; extracting edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image; generating a statistical chart representing the anomaly of the conveyor belt based on the edge image sequence and the edge image corresponding to the target image; and sequentially generating various anomaly information of the conveyor belt according to a pre-set order in the statistical chart.
[0008] Optionally, generating a statistical chart representing the conveyor belt anomaly based on the edge image sequence and the edge image corresponding to the target image includes: performing a first pooling operation on each edge image in the edge image sequence to generate a first pooled image, resulting in a first pooled image sequence; performing a second pooling operation on the edge image corresponding to the target image to generate a second pooled image; and generating the statistical chart based on the first pooled image sequence and the second pooled image.
[0009] Optionally, generating the statistical graph based on the first pooling image sequence and the second pooling image includes: summing the pixel value sets of each row in the second pooling image to obtain a first vector; multiplying the corresponding pixels of each first pooling image in the first pooling image sequence to obtain a multiplied image; performing square root processing on each pixel in the multiplied image according to the number of first pooling images in the first pooling image sequence to obtain a processed image; summing the pixel value sets of each row in the processed image to obtain a second vector, wherein the data dimension of the second vector is the same as that of the first vector; and generating the statistical graph based on the first vector and the second vector.
[0010] Optionally, generating the statistical graph based on the first vector and the second vector includes: subtracting the first vector from the second vector to obtain a third vector; inputting the third vector into a target activation function to obtain an output result; performing a moving average on the output result to obtain a processing result; and visualizing the processing result to obtain the statistical graph.
[0011] Optionally, the above-mentioned abnormal information of the conveyor belt is generated sequentially according to a pre-set order in the above-mentioned statistical chart, including: determining the set of coordinate points whose ordinate values are greater than a first threshold according to the direction of the horizontal axis of the above-mentioned statistical chart; for each coordinate point in the above-mentioned coordinate point set, performing the following processing steps to generate abnormal information corresponding to the above-mentioned coordinate point: determining the set of ordinate values of at least one coordinate point within a predetermined interval centered on the horizontal coordinate of the above-mentioned coordinate point; determining whether the mean value corresponding to the above-mentioned ordinate value set is greater than a second threshold; in response to determining that the mean value is greater than the second threshold value, determining the congestion location of the items on the above-mentioned conveyor belt corresponding to the above-mentioned coordinate point; and generating abnormal information of the above-mentioned conveyor belt according to the congestion location of the items.
[0012] Optionally, the method further includes: in response to the mean being less than or equal to the predetermined threshold, determining the obstruction position of the item on the conveyor belt corresponding to the coordinate point; and generating abnormal information of the conveyor belt based on the obstruction position of the item.
[0013] Optionally, the above-mentioned extraction of edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image includes: performing image correction on each image in the image sequence to generate a first corrected image, thereby obtaining a first corrected image sequence; performing image correction on the target image to generate a second corrected image; performing noise reduction processing on each first corrected image in the first corrected image sequence to generate a first noise-reduced image, thereby obtaining a first noise-reduced image sequence; performing noise reduction processing on the second corrected image to generate a second noise-reduced image; and using an edge detection algorithm and / or a contour detection algorithm to extract edge information from the first noise-reduced image sequence and the second noise-reduced image, thereby obtaining an edge image sequence and an edge image corresponding to the target image.
[0014] Secondly, some embodiments of this disclosure provide an anomaly information generation apparatus, comprising: an extraction unit configured to extract an image sequence and a target image from a target video, wherein the target image is an image of a conveyor belt without any items to be transported, and the images in the image sequence are images of items being transported on the conveyor belt; an extraction unit configured to extract edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image; a first generation unit configured to generate a statistical chart representing the abnormal condition of the conveyor belt based on the edge image sequence and the edge image corresponding to the target image; and a second generation unit configured to sequentially generate various anomaly information of the conveyor belt according to a pre-set order in the statistical chart.
[0015] Optionally, the first generation unit is configured to: perform a first pooling operation on each edge image in the above edge image sequence to generate a first pooled image, thereby obtaining a first pooled image sequence; perform a second pooling operation on the edge image corresponding to the above target image to generate a second pooled image; and generate the above statistical graph based on the above first pooled image sequence and the above second pooled image.
[0016] Optionally, the first generation unit is configured to: add the pixel value sets of each row in the second pooling image to obtain a first vector; multiply the corresponding pixels of each first pooling image in the first pooling image sequence to obtain a multiplied image; perform square root processing on each pixel in the multiplied image according to the number of first pooling images in the first pooling image sequence to obtain a processed image; add the pixel value sets of each row in the processed image to obtain a second vector, wherein the data dimension of the second vector is the same as that of the first vector; and generate the statistical graph based on the first vector and the second vector.
[0017] Optionally, the first generation unit is configured to: subtract the first vector from the second vector to obtain a third vector; input the third vector into the target activation function to obtain an output result; perform a moving average on the output result to obtain a processing result; and visualize the processing result to obtain the statistical chart.
[0018] Optionally, the second generation unit is configured to: sequentially determine a set of coordinate points whose ordinate values are greater than a first threshold based on the direction of the horizontal axis in the above statistical chart; for each coordinate point in the above coordinate point set, perform the following processing steps to generate abnormal information corresponding to the above coordinate point: determine a set of ordinate values of at least one coordinate point within a predetermined interval centered on the horizontal coordinate of the above coordinate point; determine whether the mean value corresponding to the above ordinate value set is greater than a second threshold; in response to determining that the mean value is greater than the second threshold, determine the congestion location of the items on the above conveyor belt corresponding to the above coordinate point; and generate abnormal information of the above conveyor belt based on the congestion location of the items.
[0019] Optionally, the second generation unit is configured to: in response to the mean being less than or equal to the second threshold, determine the position of the object obstructing the coordinate point on the conveyor belt; and generate abnormal information of the conveyor belt based on the object obstruction position.
[0020] Optionally, the extraction unit is configured to: perform image correction on each image in the image sequence to generate a first corrected image, thereby obtaining a first corrected image sequence; perform image correction on the target image to generate a second corrected image; perform noise reduction processing on each first corrected image in the first corrected image sequence to generate a first denoised image, thereby obtaining a first denoised image sequence; perform noise reduction processing on the second corrected image to generate a second denoised image; and extract edge information from the first denoised image sequence and the second denoised image using an edge detection algorithm and / or a contour detection algorithm, thereby obtaining the edge image sequence and the edge image corresponding to the target image.
[0021] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.
[0022] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.
[0023] The above embodiments of this disclosure have the following beneficial effects: the abnormal information generation method of some embodiments of this disclosure can quickly and efficiently determine the abnormal situation of the conveyor belt during the transportation of objects. Specifically, machine vision-based machine learning algorithms for detection and recognition require a large amount of annotation and training, and have high computational complexity and limited generalization ability. This results in the generation of abnormal information of the conveyor belt being inefficient and slow. Based on this, the abnormal information generation method of some embodiments of this disclosure can first extract image sequences and target images from the target video to serve as data support for subsequent abnormal information generation. Then, the edge information of the above image sequences and the above target images is extracted to obtain the edge image sequence corresponding to the above image sequence and the edge image corresponding to the above target image. Here, the edge image sequence corresponding to the above image sequence and the edge image corresponding to the above target image are among the most important image features. The edge images retain a significant portion of the image information in the original image, and by analyzing this image information, various abnormal information of the conveyor belt can be effectively generated. Furthermore, based on the edge image sequence and the edge image corresponding to the above target image, a statistical chart characterizing the abnormal situation of the conveyor belt is generated. The generated statistical chart can more easily and intuitively display the abnormal information of the conveyor belt. Finally, following the pre-defined order in the aforementioned statistical chart, the various anomaly information for the conveyor belt are generated sequentially. In summary, the above anomaly information generation method can quickly and efficiently identify anomalies in the conveyor belt during object transport. Furthermore, it does not require extensive labeling and training, has low computational complexity, and exhibits strong generalization ability. Attached Figure Description
[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0025] Figure 1 This is a schematic diagram of an application scenario of an exception information generation method according to some embodiments of the present disclosure;
[0026] Figure 2 This is a flowchart of some embodiments of the abnormal information generation method according to this disclosure;
[0027] Figure 3 This is a schematic diagram of image correction according to some embodiments of the abnormal information generation method of this disclosure;
[0028] Figure 4 This is a schematic diagram of edge image generation according to some embodiments of the abnormal information generation method of this disclosure;
[0029] Figure 5This is a schematic diagram illustrating the determination of the congestion location of items according to some embodiments of the abnormal information generation method of this disclosure;
[0030] Figure 6 These are flowcharts of other embodiments of the anomaly information generation method according to this disclosure;
[0031] Figure 7 These are schematic diagrams illustrating the structure of some embodiments of the anomaly information generation apparatus according to this disclosure;
[0032] Figure 8 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation
[0033] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0034] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0038] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0039] Figure 1 This is a schematic diagram illustrating an application scenario of an exception information generation method according to some embodiments of the present disclosure.
[0040] exist Figure 1In this application scenario, electronic device 101 can first extract image sequence 102 and target image 103 from the target video. Target image 103 is an image where no items are to be transported on the conveyor belt, while the images in image sequence 102 are images of items being transported on the conveyor belt. In this application scenario, image sequence 102 includes images 1021, 1022, and 1023. In the target video, the time interval between images 1021 and 1022 can be the same as the time interval between images 1022 and 1023. The time corresponding to image 1021 is earlier than that of image 1022. The time corresponding to image 1022 is earlier than that of image 1023. Then, edge information of image sequence 102 and target image 103 is extracted to obtain edge image sequence 104 corresponding to image sequence 102 and edge image 105 corresponding to target image 103. In this application scenario, the aforementioned edge image sequence 104 may include edge image 1041, edge image 1042, and edge image 1043. Furthermore, based on the aforementioned edge image sequence 104 and the edge image 5 corresponding to the target image 103, a statistical graph 106 representing the conveyor belt anomaly is generated. Finally, according to a pre-set order in the statistical graph 106, various abnormal information 107s of the conveyor belt are generated sequentially. In this application scenario, the aforementioned abnormal information 107 may include abnormal information 1071, abnormal information 1072, and abnormal information 1073. As an example, abnormal information 1071 may be a blockage of items. Abnormal information 1072 may be a partial breakage of the conveyor belt. Abnormal information 1073 may be items adhering to the conveyor belt.
[0041] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.
[0042] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.
[0043] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of an anomaly information generation method according to the present disclosure. This anomaly information generation method includes the following steps:
[0044] Step 201: Extract image sequences and target images from the target video.
[0045] In some embodiments, the execution entity of the above-described exception information generation method (e.g.) Figure 1 The electronic device shown can extract an image sequence and a target image from a target video. The target image is an image where no items are to be transported on the conveyor belt, and the images in the image sequence are images of items being transported on the conveyor belt. The target video consists of multiple frames. The target video can be a video monitoring the transport of various items on a conveyor belt. As an example, the image sequence can be obtained by randomly selecting several consecutive frames from the target video. From these multiple frames, one frame showing no items to be transported on the conveyor belt can be randomly selected as the target image.
[0046] Step 202: Extract the edge information of the above image sequence and the above target image to obtain the edge image sequence corresponding to the above image sequence and the edge image corresponding to the above target image.
[0047] In some embodiments, the execution entity can extract edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image. The edge image can be an image obtained after edge extraction of the original image. An edge is the junction between an image region and another attribute region; it is where regional attributes change abruptly, where there is the greatest uncertainty in the image, and where image information is most concentrated; the edges of an image contain rich information.
[0048] As an example, the aforementioned execution entity can use the Laplacian Gaussian operator to extract the edge information of the aforementioned image sequence and the aforementioned target image, thereby obtaining the edge image sequence corresponding to the aforementioned image sequence and the edge image corresponding to the aforementioned target image.
[0049] As an example, the aforementioned execution entity can use the computer vision library (OpenCV) to call relevant interfaces to extract the edge information of the aforementioned image sequence and the aforementioned target image, thereby obtaining the edge image sequence corresponding to the aforementioned image sequence and the edge image corresponding to the aforementioned target image.
[0050] In some optional implementations of certain embodiments, the extraction of edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image may include the following steps:
[0051] The first step is to perform image correction on each image in the above image sequence to generate a first corrected image, thus obtaining a first corrected image sequence.
[0052] The second step is to perform image correction on the target image to generate a second corrected image.
[0053] As an example, the aforementioned implementing entity can, in accordance with Figure 3 To generate the first corrected image sequence and the second corrected image.
[0054] The specific method is as follows: Use the Paint tool in Windows to determine the coordinates of the four edge points of the conveyor belt in the image sequence and the target image. Then, project and transform the quadrilateral area formed by connecting the four points a, b, c, and d into a rectangular area according to the direction of the conveyor belt movement to obtain the first corrected image sequence and the second corrected image.
[0055] The third step is to perform noise reduction processing on each of the first corrected images in the first corrected image sequence to generate a first denoised image, thus obtaining the first denoised image sequence.
[0056] As an example, the aforementioned execution entity can perform Gaussian noise reduction processing on each of the first corrected images in the aforementioned first corrected image sequence to generate a first denoised image, thereby obtaining a first denoised image sequence.
[0057] The fourth step is to perform noise reduction processing on the second corrected image to generate a second noise-reduced image.
[0058] As an example, the aforementioned execution entity can perform Gaussian noise reduction on the aforementioned second corrected image to generate a second denoised image.
[0059] Fifth, using edge detection algorithms and / or contour detection algorithms, extract the edge information of the first denoised image sequence and the second denoised image to obtain the edge image sequence and the edge image corresponding to the target image. The edge detection algorithms may include, but are not limited to, at least one of the following: Sobel edge detection, Prewitt edge detection, and Canny edge detection.
[0060] As an example, the first denoised image in the first denoised image sequence can be as follows: Figure 4 As shown in 401. The first contour detection can be as follows: Figure 4 As shown in 402. The second contour detection can be as follows: Figure 4 As shown in 403, the first edge detection can be as follows: Figure 4 As shown in 404, the standard for the first contour detection can be an adaptive binarization threshold of 20. The standard for the second contour detection can be a global binarization threshold of 127. The first edge detection can be Canny edge detection.
[0061] The second denoised image can be as follows: Figure 4 As shown in 405 of the diagram. The third contour detection can be as follows: Figure 4 As shown in 406. The fourth contour detection can be as follows: Figure 4 As shown in 407. The second edge detection can be as follows: Figure 4 As shown in 408, the first contour detection standard can be an adaptive binarization threshold of 20. The second contour detection standard can be a global binarization threshold of 127. The first edge detection can be Canny edge detection.
[0062] Step 203: Generate a statistical chart representing the abnormal situation of the conveyor belt based on the edge image sequence and the edge image corresponding to the target image.
[0063] In some embodiments, the execution entity may generate a statistical chart representing the abnormal condition of the conveyor belt based on the edge image sequence and the edge image corresponding to the target image.
[0064] As an example, the aforementioned implementing entity can generate a statistical chart using the following steps:
[0065] The first step is to subtract the pixel set of the target image from the pixel set of each edge image in the above edge image sequence to obtain the edge image sequence after subtraction.
[0066] The second step is to multiply the corresponding pixels of each image in the above image sequence after the edges have been subtracted to obtain the multiplied image.
[0067] The third step is to perform a square root operation on each pixel in the multiplied image based on the number of edge image sequences in the above edge image sequence to obtain the processed image.
[0068] The fourth step is to input the processed image into the target activation function to obtain the output result. The target activation function can be a Rectified Linear Unit (ReLU) activation function.
[0069] The fifth step is to visualize the output results to obtain the statistical chart. As an example, the executing entity can use relevant visualization tools to visualize the output results and obtain the statistical chart. These visualization tools can be Python visualization libraries.
[0070] Step 204: Generate each abnormal information of the conveyor belt in sequence according to the pre-set order in the above statistical chart.
[0071] In some embodiments, the executing entity may sequentially generate various abnormal information messages of the conveyor belt according to a pre-set order in the statistical chart. These abnormal information messages may include, but are not limited to: conveyor belt blockage, partial breakage of the conveyor belt, and the presence of adhering substances on the conveyor belt. As an example, the executing entity may sequentially generate these abnormal information messages of the conveyor belt in various ways according to a pre-set order in the statistical chart.
[0072] In some optional implementations of certain embodiments, generating the various abnormal information of the conveyor belt sequentially according to a pre-set order in the statistical chart may include the following steps:
[0073] The first step is to determine the set of coordinate points whose ordinate values are greater than the first threshold, based on the direction of the horizontal axis in the above statistical chart. For example, the first threshold could be "0.2".
[0074] The second step is to perform the following processing steps for each coordinate point in the above coordinate point set to generate the corresponding anomaly information:
[0075] The first sub-step involves determining the set of ordinate values for at least one coordinate point within a predetermined interval, centered on the x-coordinate of the aforementioned coordinate point. For example, the predetermined interval could be a range of 10 to 30 pixels to the left and right of the x-coordinate of the coordinate point.
[0076] The second sub-step involves determining whether the mean value corresponding to the aforementioned set of ordinate values is greater than a second threshold. For example, the second threshold could be half of the first threshold.
[0077] The third sub-step involves determining the congestion location of the items on the conveyor belt corresponding to the coordinate point, in response to the determination that the average value is greater than the second threshold. The congestion location can be pixel coordinates in an image.
[0078] As an example, such as Figure 5 As shown, Figure 5 From a particular image in the image sequence on the left, the statistical graph can be moved clockwise to obtain... Figure 5 The statistical chart on the right shows a threshold of 0.2. By comparing the statistical chart with the image, it can be determined that the dashed line area in the image represents a congested area, indicating potential congestion.
[0079] The fourth sub-step involves generating abnormal information about the conveyor belt based on the location of the congestion. For example, the executing entity can use relevant camera equipment to capture images of the congestion at the location of the congestion to generate the abnormal information for the conveyor belt.
[0080] Optionally, the above steps also include:
[0081] The first step is to determine the position of the object obstructing the conveyor belt corresponding to the coordinate point when the mean value is less than or equal to the second threshold value.
[0082] The specific locations where items are obstructed can be found in the section on determining congested areas, and will not be elaborated upon here.
[0083] The second step is to generate abnormal information about the transmission belt based on the location of the obstruction by the aforementioned items.
[0084] As an example, the aforementioned executing entity can use relevant camera equipment to capture images of the obstruction of objects to generate abnormal information about the conveyor belt.
[0085] The above embodiments of this disclosure have the following beneficial effects: the abnormal information generation method of some embodiments of this disclosure can quickly and efficiently determine the abnormal situation of the conveyor belt during the transportation of objects. Specifically, machine vision-based machine learning algorithms for detection and recognition require a large amount of annotation and training, and have high computational complexity and limited generalization ability. This results in the generation of abnormal information of the conveyor belt being inefficient and slow. Based on this, the abnormal information generation method of some embodiments of this disclosure can first extract image sequences and target images from the target video to serve as data support for subsequent abnormal information generation. Then, the edge information of the above image sequences and the above target images is extracted to obtain the edge image sequence corresponding to the above image sequence and the edge image corresponding to the above target image. Here, the edge image sequence corresponding to the above image sequence and the edge image corresponding to the above target image are among the most important image features. The edge images retain a significant portion of the image information in the original image, and by analyzing this image information, various abnormal information of the conveyor belt can be effectively generated. Furthermore, based on the edge image sequence and the edge image corresponding to the above target image, a statistical chart characterizing the abnormal situation of the conveyor belt is generated. The generated statistical chart can more easily and intuitively display the abnormal information of the conveyor belt. Finally, following the pre-defined order in the aforementioned statistical chart, the various anomaly information for the conveyor belt are generated sequentially. In summary, the above anomaly information generation method can quickly and efficiently identify anomalies in the conveyor belt during object transport. Furthermore, it does not require extensive labeling and training, has low computational complexity, and exhibits strong generalization ability.
[0086] Further reference Figure 6 The diagram illustrates a flow 600 of another embodiment of the anomaly information generation method according to the present disclosure. This anomaly information generation method includes the following steps:
[0087] Step 601: Extract image sequences and target images from the target video.
[0088] Step 602: Extract the edge information of the above image sequence and the above target image to obtain the edge image sequence corresponding to the above image sequence and the edge image corresponding to the above target image.
[0089] In some embodiments, the specific implementation of steps 601-602 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 201-202 in the corresponding embodiments will not be repeated here.
[0090] Step 603: Perform a first pooling operation on each edge image in the above edge image sequence to generate a first pooled image, thus obtaining a first pooled image sequence.
[0091] In some embodiments, the executing entity (e.g. Figure 1 The electronic device shown can perform a first pooling operation on each edge image in the above edge image sequence to generate a first pooled image, thus obtaining a first pooled image sequence. The first pooling operation can be, but is not limited to, one of the following: average pooling or max pooling.
[0092] Step 604: Perform a second pooling operation on the edge image corresponding to the target image to generate a second pooled image.
[0093] In some embodiments, the execution entity may perform a second pooling operation on the edge image corresponding to the target image to generate a second pooled image. The second pooling operation may be, but is not limited to, one of the following: average pooling or max pooling.
[0094] Step 605: Generate the statistical graph based on the first pooling image sequence and the second pooling image.
[0095] In some embodiments, the execution entity may generate the statistical graph in various ways based on the first pooling image sequence and the second pooling image.
[0096] In some optional implementations of certain embodiments, generating the statistical graph based on the first pooling image sequence and the second pooling image may include the following steps:
[0097] The first step is to sum the pixel values of each row in the second pooling image to obtain the first vector.
[0098] The second step is to multiply the corresponding pixels of each first pooling image in the above first pooling image sequence to obtain the multiplied image.
[0099] The third step involves performing a square root operation on each pixel in the multiplied image based on the number of first pooling images in the first pooling image sequence, to obtain the processed image.
[0100] As an example, if the number of first pooling images in the above first pooling image sequence is 3, then each pixel in the multiplied image is subjected to cube root processing to obtain the processed image.
[0101] The fourth step involves summing the pixel values of each row in the processed image to obtain a second vector. The data dimension of this second vector is the same as that of the first vector.
[0102] Fifth step: Generate the statistical graph based on the first vector and the second vector mentioned above.
[0103] As an example, the aforementioned executing entity can generate the aforementioned statistical graph using various methods based on the aforementioned first vector and the aforementioned second vector.
[0104] Optionally, generating the statistical graph based on the first vector and the second vector may include the following steps:
[0105] The first step is to subtract the first vector from the second vector to obtain the third vector.
[0106] It should be noted that subtracting the first vector from the second vector can remove the influence of conveyor belt feature information on determining whether the conveyor belt is abnormal. This makes the subsequently generated abnormal information more accurate.
[0107] The second step is to input the third vector into the target activation function to obtain the output result. The target activation function can be a ReLU activation function.
[0108] The third step is to perform a moving average on the above output results to obtain the processed result.
[0109] The fourth step is to visualize the processing results to obtain the statistical chart. As an example, the executing entity can use relevant visualization tools to visualize the processing results and obtain the statistical chart. These visualization tools can be Python visualization libraries.
[0110] Step 606: Generate each abnormal information of the conveyor belt in sequence according to the pre-set order in the above statistical chart.
[0111] In some embodiments, the specific implementation of step 606 and its resulting technical effects can be found in [reference needed]. Figure 2 Step 204 in the corresponding embodiment will not be repeated here.
[0112] from Figure 6 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 6The flowchart 600 of the anomaly information generation method in some corresponding embodiments further emphasizes the specific steps of pooling the edge image sequence and the corresponding edge image of the target image. Therefore, the schemes described in these embodiments, through pooling the edge image sequence and the corresponding edge image of the target image, can effectively alleviate the problem of unevenness between adjacent pixels caused by extracting edge information from the image sequence and the aforementioned target image, making the subsequently generated statistical graphs and corresponding anomaly information more accurate.
[0113] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an anomaly information generation device, which are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.
[0114] like Figure 7 As shown, an anomaly information generation device 700 includes: an extraction unit 701, an extraction unit 702, a first generation unit 703, and a second generation unit 704. The extraction unit 701 is configured to extract an image sequence and a target image from a target video, wherein the target image is an image of a conveyor belt without any items to be transported, and the images in the image sequence are images of items being transported on the conveyor belt. The extraction unit 702 is configured to extract edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image. The first generation unit 703 is configured to generate a statistical chart representing the anomaly of the conveyor belt based on the edge image sequence and the edge image corresponding to the target image. The second generation unit 704 is configured to sequentially generate various anomaly information of the conveyor belt according to a pre-set order in the statistical chart.
[0115] In some optional implementations of some embodiments, the first generation unit 703 in the above-described apparatus 700 may be further configured to: perform a first pooling operation on each edge image in the above-described edge image sequence to generate a first pooled image, thereby obtaining a first pooled image sequence; perform a second pooling operation on the edge image corresponding to the above-described target image to generate a second pooled image; and generate the above-described statistical graph based on the above-described first pooled image sequence and the above-described second pooled image.
[0116] In some optional implementations of certain embodiments, the first generation unit 703 in the above-described apparatus 700 may be further configured to: add the pixel value sets of each row in the second pooling image to obtain a first vector; multiply the corresponding pixels of each first pooling image in the first pooling image sequence to obtain a multiplied image; perform square root processing on each pixel in the multiplied image according to the number of first pooling images in the first pooling image sequence to obtain a processed image; add the pixel value sets of each row in the processed image to obtain a second vector, wherein the data dimension of the second vector is the same as that of the first vector; and generate the statistical graph based on the first vector and the second vector.
[0117] In some optional implementations of some embodiments, the first generation unit 703 in the above-described apparatus 700 may be further configured to: subtract the first vector from the second vector to obtain a third vector; input the third vector into the target activation function to obtain an output result; perform a moving average processing on the output result to obtain a processing result; and visualize the processing result to obtain the above-described statistical chart.
[0118] In some optional implementations of certain embodiments, the second generation unit 704 in the above-described device 700 may be further configured to: sequentially determine a set of coordinate points whose ordinate values are greater than a first threshold based on the direction of the horizontal axis in the above-described statistical chart; for each coordinate point in the above-described coordinate point set, perform the following processing steps to generate abnormal information corresponding to the above-described coordinate point: determine a set of ordinate values of at least one coordinate point within a predetermined interval centered on the horizontal coordinate of the above-described coordinate point; determine whether the mean value corresponding to the above-described ordinate value set is greater than a second threshold; in response to determining that the mean value is greater than the above-described second threshold, determine the congestion location of the items on the above-described conveyor belt corresponding to the above-described coordinate point; and generate abnormal information of the above-described conveyor belt based on the congestion location of the items.
[0119] In some optional implementations of some embodiments, the second generation unit 704 in the above-described device 700 may be further configured to: determine the position of the object obstructing the conveyor belt corresponding to the coordinate point in response to the mean value being less than or equal to the second threshold; and generate abnormal information of the conveyor belt based on the position of the object obstructing the conveyor belt.
[0120] In some optional implementations of certain embodiments, the extraction unit 701 in the above-described apparatus 700 may be further configured to: perform image correction on each image in the above-described image sequence to generate a first corrected image, thereby obtaining a first corrected image sequence; perform image correction on the above-described target image to generate a second corrected image; perform noise reduction processing on each of the first corrected images in the above-described first corrected image sequence to generate a first noise-reduced image, thereby obtaining a first noise-reduced image sequence; perform noise reduction processing on the above-described second corrected image to generate a second noise-reduced image; and extract edge information of the above-described first noise-reduced image sequence and the above-described second noise-reduced image using an edge detection algorithm and / or a contour detection algorithm, thereby obtaining the edge image sequence and the edge image corresponding to the above-described target image.
[0121] It is understandable that the units described in the device 700 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 700 and the units contained therein, and will not be repeated here.
[0122] The following is for reference. Figure 8 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of the electronic device (800) in the device. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.
[0123] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0124] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 8 Each box shown can represent a device or multiple devices as needed.
[0125] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined in the methods of some embodiments of this disclosure.
[0126] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0127] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0128] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: extract an image sequence and a target image from a target video, wherein the target image is an image of a conveyor belt without any items to be transported, and the images in the image sequence are images of items being transported on the conveyor belt; extract edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image; generate a statistical chart characterizing the abnormal condition of the conveyor belt based on the edge image sequence and the edge image corresponding to the target image; and sequentially generate various abnormal information of the conveyor belt according to a pre-set order in the statistical chart.
[0129] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0131] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an extraction unit, an extraction unit, a first generation unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, the second generation unit may be described as "a unit that sequentially generates various abnormal information messages of the conveyor belt according to a pre-set order in the aforementioned statistical chart."
[0132] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0133] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A method for generating abnormal information, comprising: Extract image sequences and target images from the target video, wherein the target image is an image in which there are no items to be transported on the conveyor belt, and the images in the image sequence are images of items being transported on the conveyor belt; Extract edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image; A first pooling operation is performed on each edge image in the edge image sequence to generate a first pooled image, resulting in a first pooled image sequence; A second pooling operation is performed on the edge image corresponding to the target image to generate a second pooling image; The pixel value sets of each row in the second pooled image are summed to obtain the first vector; The corresponding pixels of each first pooling image in the first pooling image sequence are multiplied together to obtain the multiplied image; Based on the number of first pooling images in the first pooling image sequence, each pixel in the multiplied image is square-rooted to obtain the processed image; The pixel value sets of each row in the processed image are added together to obtain a second vector, wherein the data dimension of the second vector is the same as that of the first vector; Generating a statistical graph based on the first vector and the second vector includes: subtracting the first vector from the second vector to obtain a third vector; inputting the third vector into a target activation function to obtain an output result; performing a moving average on the output result to obtain a processing result; and visualizing the processing result to obtain the statistical graph. According to the pre-set order in the statistical chart, the various abnormal information of the conveyor belt are generated sequentially.
2. The method according to claim 1, wherein, The step of generating various abnormal information of the conveyor belt sequentially according to a pre-set order in the statistical chart includes: Based on the direction of the horizontal axis in the statistical chart, the set of coordinate points whose vertical coordinate values are greater than the first threshold are determined sequentially; For each coordinate point in the set of coordinate points, perform the following processing steps to generate the corresponding anomaly information: Determine the set of ordinate values for at least one coordinate point within a predetermined interval centered on the x-coordinate of the stated coordinate point; Determine whether the mean value corresponding to the set of ordinate values is greater than the second threshold; In response to determining that the mean is greater than the second threshold, the congestion location of the items on the conveyor belt corresponding to the coordinate point is determined; Based on the location of the congestion in the items, generate abnormal information about the transmission belt.
3. The method according to claim 2, wherein, The method further includes: In response to the mean being less than or equal to the second threshold, the obstruction position of the item on the conveyor belt corresponding to the coordinate point is determined; Anomaly information for the transmission belt is generated based on the location of the obstruction by the object.
4. The method according to claim 1, wherein, The step of extracting edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image includes: Each image in the image sequence is subjected to image correction to generate a first corrected image, resulting in a first corrected image sequence; The target image is subjected to image correction to generate a second corrected image; Each first corrected image in the first corrected image sequence is subjected to noise reduction processing to generate a first denoised image, thus obtaining a first denoised image sequence; The second corrected image is subjected to noise reduction processing to generate a second noise-reduced image; Using edge detection algorithms and / or contour detection algorithms, edge information of the first denoised image sequence and the second denoised image is extracted to obtain the edge image sequence and the edge image corresponding to the target image.
5. An anomaly information generation device, comprising: An extraction unit is configured to extract an image sequence and a target image from a target video, wherein the target image is an image in which no item is to be transported on the conveyor belt, and the images in the image sequence are images of items being transported on the conveyor belt. The extraction unit is configured to extract edge information from the image sequence and the target image to obtain an edge image sequence corresponding to the image sequence and an edge image corresponding to the target image; A first generation unit is configured to perform a first pooling operation on each edge image in the edge image sequence to generate a first pooled image, resulting in a first pooled image sequence; perform a second pooling operation on the edge image corresponding to the target image to generate a second pooled image; sum the pixel value sets of each row in the second pooled image to obtain a first vector; multiply the corresponding pixels of each first pooled image in the first pooled image sequence to obtain a multiplied image; perform square root processing on each pixel in the multiplied image according to the number of first pooled images in the first pooled image sequence to obtain a processed image; sum the pixel value sets of each row in the processed image to obtain a second vector, wherein the data dimension of the second vector is the same as that of the first vector; generate a statistical graph based on the first vector and the second vector, including: subtracting the first vector from the second vector to obtain a third vector; inputting the third vector into a target activation function to obtain an output result; performing a moving average processing on the output result to obtain a processing result; and visualizing the processing result to obtain the statistical graph. The second generation unit is configured to generate various abnormal information of the conveyor belt in sequence according to a pre-set order in the statistical chart.
6. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.
7. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.
Citation Information
Patent Citations
Cigarette packet seal defect detecting system based on image identification technology
CN103512888A
A chain plate defect online detection device and method based on machine vision
CN109829876A
